Runtime Management of Data Quality for Scientific Observatories Using Edge and In-Transit Resources

Ali Reza Zamani, Daniel Balouek-Thomert, J. J. Villalobos, Ivan Rodero, Manish Parashar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Modern Cyberinfrastructures (CIs) operate to bring content produced from remote data sources such as sensors and scientific instruments and deliver it to end users and workflow applications. Maintaining data quality/resolution and on-time data delivery while considering an increasing number of computing, storage and network resources requires a reactive system, able to adapt to changing demands. In this paper, we propose a modelization of such system by expressing the dynamic stage of resources in the context of edge and in-transit computing. By considering resource utilization, approximation techniques and users' constraints, our proposed engine is generating mappings of workflow stages on heterogeneous geo-distributed resources. We specifically propose a runtime management layer that adapts the data resolution being delivered to the users by implementing feedback loops over the resources involved in the delivery and processing of the data streams. We implement our model into a subscription-based data streaming framework which enables integration of large facilities and advanced CIs. Experimental results show that dynamically adapting data resolution can overcome bandwidth limitation in wide area streaming analytics.

Original languageEnglish (US)
Title of host publicationProceedings - 2018 30th International Symposium on Computer Architecture and High Performance Computing, SBAC-PAD 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages274-281
Number of pages8
ISBN (Electronic)9781538677698
DOIs
StatePublished - Feb 20 2019
Event30th International Symposium on Computer Architecture and High Performance Computing, SBAC-PAD 2018 - Lyon, France
Duration: Sep 24 2018Sep 27 2018

Publication series

NameProceedings - 2018 30th International Symposium on Computer Architecture and High Performance Computing, SBAC-PAD 2018

Conference

Conference30th International Symposium on Computer Architecture and High Performance Computing, SBAC-PAD 2018
Country/TerritoryFrance
CityLyon
Period9/24/189/27/18

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture
  • Computer Science (miscellaneous)

Fingerprint

Dive into the research topics of 'Runtime Management of Data Quality for Scientific Observatories Using Edge and In-Transit Resources'. Together they form a unique fingerprint.

Cite this