Date and Location: 3/28/2017 Tue. 2 pm, CS Conference Room 206
 
Speaker: Dr. Dong Dai
 
Title: Exploiting Local Resources to Accelerate Scientific Workflows
 
Abstract: Data-driven scientific workflows contain series of standalone tasks that are interconnected through shared data files. Existing HPC storage stack force these shared data files to be written into remote global storage (normally parallel file system) to share data among different tasks and nodes. However, this ends up with non-optimal performance since the shared data file will be moved back and forth from/to remote parallel file system. These data movements are expensive both in time and energy. A commonly used strategy is to stage the shared data files in faster storage to reduce the cost of data movement. For example, the data can be staged in Burst Buffer to achieve better I/O performance. However, this needs special hardware/software to provide such a persistent, fast storage. In this research, instead of reducing the cost of each data movement, we propose to minimize the number of data movements. Specifically, we propose to utilize the local resources of compute nodes (main memory) to cache shared data files in a workflow. Together with locality-aware scheduler, it will reduce the data movement effectively. In this talk, I will summarize the state-of-the-art of relevant research and present our design. Preliminary results from simulation will also be presented.