High throughput data movement

Scott Klasky, Hasan Abbasi, Viraj Bhat, Ciprian Docan, Steve Hodson, Chen Jin, Jay Lofstead, Manish Parashar, Karsten Schwan, Matthew Wolf

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

In this chapter, we look at technology changes affecting scientists who run data-intensive simulations, particularly concerning the ways in which these computations are run and how the data they produce is analyzed. As computer systems and technology evolve, and as usage policy of supercomputers often permits very long runs, simulations are starting to run for over 24 hours and produce unprecedented amounts of data. Previously, data produced by supercomputer applications was simply stored as files for subsequent analysis, sometimes days or weeks later. However, as the amount of the data becomes very large and/or the rates at which data is produced or consumed by supercomputers become very high, this approach no longer works, and high-throughput data movement techniques are needed.

Original languageEnglish (US)
Title of host publicationScientific Data Management
Subtitle of host publicationChallenges, Technology, and Deployment
PublisherCRC Press
Pages151-180
Number of pages30
ISBN (Electronic)9781420069815
ISBN (Print)9781420069808
DOIs
StatePublished - Jan 1 2009

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Computer Science(all)
  • Mathematics(all)

Cite this

Klasky, S., Abbasi, H., Bhat, V., Docan, C., Hodson, S., Jin, C., Lofstead, J., Parashar, M., Schwan, K., & Wolf, M. (2009). High throughput data movement. In Scientific Data Management: Challenges, Technology, and Deployment (pp. 151-180). CRC Press. https://doi.org/10.1201/9781420069815