Supporting Data-Intensive Workflows in Software-Defined Federated Multi-Clouds

Javier Diaz-Montes, Manuel Diaz-Granados, Mengsong Zou, Shu Tao, Manish Parashar

Research output: Contribution to journalArticlepeer-review

8 Scopus citations

Abstract

Cloud computing is emerging as a viable platform for scientific exploration. Elastic and on-demand access to resources (and other services), the abstraction of 'unlimited' resources, and attractive pricing models provide incentives for scientists to move their workflows into clouds. Generalizing these concepts beyond a single virtualized datacenter, it is possible to create federated marketplaces where different types of resources (e.g., clouds, HPC grids, supercomputers) that may be geographically distributed, are collectively exposed as a single elastic infrastructure. This presents opportunities for optimizing the execution of application workflows with heterogeneous and dynamic requirements, and tackling larger scale problems. In this paper, we introduce a framework to manage the end-to-end execution of data-intensive application workflows in dynamic software-defined resource federation. This framework enables the autonomic execution of workflows by elastically provisioning an appropriate set of resources that meet application requirements, and by adapting this set of resources at runtime as the requirements change. It also allows users to customize scheduling policies that drive the way resources federated and used. To demonstrate the benefits of our approach, we study the execution of two different data-intensive scientific workflows in a multi-cloud federation using different policies and objective functions.

Original languageEnglish (US)
Pages (from-to)250-263
Number of pages14
JournalIEEE Transactions on Cloud Computing
Volume6
Issue number1
DOIs
StatePublished - Jan 1 2018

All Science Journal Classification (ASJC) codes

  • Software
  • Information Systems
  • Hardware and Architecture
  • Computer Science Applications
  • Computer Networks and Communications

Keywords

  • Autonomics
  • Cloud computing
  • CometCloud
  • Data-driven workflows
  • Software-defined infrastructure

Fingerprint Dive into the research topics of 'Supporting Data-Intensive Workflows in Software-Defined Federated Multi-Clouds'. Together they form a unique fingerprint.

Cite this