Adaptive Placement of Data Analysis Tasks for Staging Based In-Situ Processing

Zhe Wang, Pradeep Subedi, Matthieu Dorier, Philip E. Davis, Manish Parashar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

In-situ processing addresses the gap between speeds of computing and I/O capabilities by processing data close to the data source, i.e., on the same system as the data source (e.g., a simulation). However, the effective implementation of in-situ processing workflows requires the optimization of several design parameters such as where on the system workflow data analysis/visualization (ana/vis) as placed and how execution as well as the interaction and data exchanges between ana/vis are coordinated. For example, in the case of hybrid in-situ processing, interacting ana/vis may be tightly or loosely coupled depending on their placement, and this can lead to very different performance and scalability. A key challenge is deciding the most appropriate ana/vis placement, which depends on dynamic applications, workflow, and system characteristics that might change at runtime. In this paper, we present a framework to support online adaptive data analysis placement during the execution of an in-situ workflow. Specifically, the paper presents a model and architecture, and explores several data analysis placement strategies. Evaluation results show that dynamically choosing appropriate data analysis placement strategies can balance the benefits and overhead of different data analysis placement patterns to reduce in-situ processing time.

Original languageEnglish (US)
Title of host publicationProceedings - 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics, HiPC 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages242-251
Number of pages10
ISBN (Electronic)9781665410168
DOIs
StatePublished - 2021
Externally publishedYes
Event28th IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2021 - Virtual, Bangalore, India
Duration: Dec 17 2021Dec 18 2021

Publication series

NameProceedings - 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics, HiPC 2021

Conference

Conference28th IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2021
Country/TerritoryIndia
CityVirtual, Bangalore
Period12/17/2112/18/21

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Information Systems

Keywords

  • Adaptive workflow
  • Data-driven
  • In-situ
  • In-transit
  • Monitor
  • Near-real-time decision

Fingerprint

Dive into the research topics of 'Adaptive Placement of Data Analysis Tasks for Staging Based In-Situ Processing'. Together they form a unique fingerprint.

Cite this