High performance threaded data streaming for large scale simulations

Viraj Bhat, Scott Klasky, Scott Atchley, Micah Beck, Doug McCune, Manish Parashar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

29 Scopus citations

Abstract

We have developed a threaded parallel data streaming approach using Logistical Networking (LN) to transfer multi-terabyte simulation data from computers at NERSC to our local analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and later transferring this data to be post-processed. Our algorithms are network aware, and can stream data at up to 97Mbs on a 100Mbs link from CA to NJ during a live simulation, using less than 5% CPU overhead at NERSC. This method is the first step in setting up a pipeline for simulation workflow and data management.

Original languageEnglish (US)
Title of host publicationProceedings - Fifth IEEE/ACM International Workshop on Grid Computing
EditorsR. Buyya
Pages243-250
Number of pages8
DOIs
StatePublished - 2004
EventProceedings - Fifth IEEE/ACM International Workshop on Grid Computing - Pittsburgh, PA, United States
Duration: Nov 8 2004Nov 8 2004

Publication series

NameProceedings - IEEE/ACM International Workshop on Grid Computing
ISSN (Print)1550-5510

Other

OtherProceedings - Fifth IEEE/ACM International Workshop on Grid Computing
Country/TerritoryUnited States
CityPittsburgh, PA
Period11/8/0411/8/04

All Science Journal Classification (ASJC) codes

  • General Engineering

Fingerprint

Dive into the research topics of 'High performance threaded data streaming for large scale simulations'. Together they form a unique fingerprint.

Cite this