High throughput data movement

Scott Klasky, Hasan Abbasi, Viraj Bhat, Ciprian Docan, Steve Hodson, Chen Jin, Jay Lofstead, Manish Parashar, Karsten Schwan, Matthew Wolf

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

In this chapter, we look at technology changes affecting scientists who run data-intensive simulations, particularly concerning the ways in which these computations are run and how the data they produce is analyzed. As computer systems and technology evolve, and as usage policy of supercomputers often permits very long runs, simulations are starting to run for over 24 hours and produce unprecedented amounts of data. Previously, data produced by supercomputer applications was simply stored as files for subsequent analysis, sometimes days or weeks later. However, as the amount of the data becomes very large and/or the rates at which data is produced or consumed by supercomputers become very high, this approach no longer works, and high-throughput data movement techniques are needed.

Original languageEnglish (US)
Title of host publicationScientific Data Management
Subtitle of host publicationChallenges, Technology, and Deployment
PublisherCRC Press
Pages151-180
Number of pages30
ISBN (Electronic)9781420069815
ISBN (Print)9781420069808
DOIs
StatePublished - Jan 1 2009

Fingerprint

Supercomputers
High Throughput
Throughput
Supercomputer
Computer systems
Movement
Long-run
Simulation

All Science Journal Classification (ASJC) codes

  • Computer Science(all)
  • Mathematics(all)

Cite this

Klasky, S., Abbasi, H., Bhat, V., Docan, C., Hodson, S., Jin, C., ... Wolf, M. (2009). High throughput data movement. In Scientific Data Management: Challenges, Technology, and Deployment (pp. 151-180). CRC Press. https://doi.org/10.1201/9781420069815
Klasky, Scott ; Abbasi, Hasan ; Bhat, Viraj ; Docan, Ciprian ; Hodson, Steve ; Jin, Chen ; Lofstead, Jay ; Parashar, Manish ; Schwan, Karsten ; Wolf, Matthew. / High throughput data movement. Scientific Data Management: Challenges, Technology, and Deployment. CRC Press, 2009. pp. 151-180
@inbook{d0222aa6d15e494aaa1aa1226ed693e8,
title = "High throughput data movement",
abstract = "In this chapter, we look at technology changes affecting scientists who run data-intensive simulations, particularly concerning the ways in which these computations are run and how the data they produce is analyzed. As computer systems and technology evolve, and as usage policy of supercomputers often permits very long runs, simulations are starting to run for over 24 hours and produce unprecedented amounts of data. Previously, data produced by supercomputer applications was simply stored as files for subsequent analysis, sometimes days or weeks later. However, as the amount of the data becomes very large and/or the rates at which data is produced or consumed by supercomputers become very high, this approach no longer works, and high-throughput data movement techniques are needed.",
author = "Scott Klasky and Hasan Abbasi and Viraj Bhat and Ciprian Docan and Steve Hodson and Chen Jin and Jay Lofstead and Manish Parashar and Karsten Schwan and Matthew Wolf",
year = "2009",
month = "1",
day = "1",
doi = "10.1201/9781420069815",
language = "English (US)",
isbn = "9781420069808",
pages = "151--180",
booktitle = "Scientific Data Management",
publisher = "CRC Press",

}

Klasky, S, Abbasi, H, Bhat, V, Docan, C, Hodson, S, Jin, C, Lofstead, J, Parashar, M, Schwan, K & Wolf, M 2009, High throughput data movement. in Scientific Data Management: Challenges, Technology, and Deployment. CRC Press, pp. 151-180. https://doi.org/10.1201/9781420069815

High throughput data movement. / Klasky, Scott; Abbasi, Hasan; Bhat, Viraj; Docan, Ciprian; Hodson, Steve; Jin, Chen; Lofstead, Jay; Parashar, Manish; Schwan, Karsten; Wolf, Matthew.

Scientific Data Management: Challenges, Technology, and Deployment. CRC Press, 2009. p. 151-180.

Research output: Chapter in Book/Report/Conference proceedingChapter

TY - CHAP

T1 - High throughput data movement

AU - Klasky, Scott

AU - Abbasi, Hasan

AU - Bhat, Viraj

AU - Docan, Ciprian

AU - Hodson, Steve

AU - Jin, Chen

AU - Lofstead, Jay

AU - Parashar, Manish

AU - Schwan, Karsten

AU - Wolf, Matthew

PY - 2009/1/1

Y1 - 2009/1/1

N2 - In this chapter, we look at technology changes affecting scientists who run data-intensive simulations, particularly concerning the ways in which these computations are run and how the data they produce is analyzed. As computer systems and technology evolve, and as usage policy of supercomputers often permits very long runs, simulations are starting to run for over 24 hours and produce unprecedented amounts of data. Previously, data produced by supercomputer applications was simply stored as files for subsequent analysis, sometimes days or weeks later. However, as the amount of the data becomes very large and/or the rates at which data is produced or consumed by supercomputers become very high, this approach no longer works, and high-throughput data movement techniques are needed.

AB - In this chapter, we look at technology changes affecting scientists who run data-intensive simulations, particularly concerning the ways in which these computations are run and how the data they produce is analyzed. As computer systems and technology evolve, and as usage policy of supercomputers often permits very long runs, simulations are starting to run for over 24 hours and produce unprecedented amounts of data. Previously, data produced by supercomputer applications was simply stored as files for subsequent analysis, sometimes days or weeks later. However, as the amount of the data becomes very large and/or the rates at which data is produced or consumed by supercomputers become very high, this approach no longer works, and high-throughput data movement techniques are needed.

UR - http://www.scopus.com/inward/record.url?scp=85056539578&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056539578&partnerID=8YFLogxK

U2 - 10.1201/9781420069815

DO - 10.1201/9781420069815

M3 - Chapter

SN - 9781420069808

SP - 151

EP - 180

BT - Scientific Data Management

PB - CRC Press

ER -

Klasky S, Abbasi H, Bhat V, Docan C, Hodson S, Jin C et al. High throughput data movement. In Scientific Data Management: Challenges, Technology, and Deployment. CRC Press. 2009. p. 151-180 https://doi.org/10.1201/9781420069815