On the use of burst buffers for accelerating data-intensive scientific workflows

Rafael F. da Silva, Scott Callaghan, & Ewa Deelman

Submitted August 2017, SCEC Contribution #7610

Science applications frequently produce and consume large volumes of data, but delivering this data to and from compute resources can be challenging, as parallel file system performance has not kept up with compute and memory performance. To mitigate this I/O bottleneck, some systems have deployed burst buffers, but their impact on performance for real-world workflow applications is not always clear. In this paper, we examine the impact of burst buffers through the remote-shared, allocatable burst buffers on the Cori system at NERSC. By running a subset of the SCEC CyberShake workflow, a production seismic hazard analysis workflow, we find that using burst buffers offers read and write improvements of about an order of magnitude, and these improvements lead to increased job performance, even for long-running CPU-bound jobs.

Key Words
Scientific Workflows, Burst Buffers, High-Performance Computing, In Transit Processing

Citation
da Silva, R. F., Callaghan, S., & Deelman, E. (2017, 08). On the use of burst buffers for accelerating data-intensive scientific workflows. Oral Presentation at International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2017).

Related Projects & Working Groups
CyberShake