Autonomic management of application workflows on hybrid computing infrastructure

Hyunjoo Kim, Yaakoub El-Khamra, Ivan Rodero, Shantenu Jha, Manish Parashar

Research output: Contribution to journalArticle

43 Citations (Scopus)

Abstract

In this paper, we present a programming and runtime framework that enables the autonomic management of complex application workflows on hybrid computing infrastructures. The framework is designed to address system and application heterogeneity and dynamics to ensure that application objectives and constraints are satisfied. The need for such autonomic system and application management is becoming critical as computing infrastructures become increasingly heterogeneous, integrating different classes of resources from high-end HPC systems to commodity clusters and clouds. For example, the framework presented in this paper can be used to provision the appropriate mix of resources based on application requirements and constraints. The framework also monitors the system/application state and adapts the application and/or resources to respond to changing requirements or environment. To demonstrate the operation of the framework and to evaluate its ability, we employ a workflow used to characterize an oil reservoir executing on a hybrid infrastructure composed of TeraGrid nodes and Amazon EC2 instances of various types. Specifically, we show how different applications objectives such as acceleration, conservation and resilience can be effectively achieved while satisfying deadline and budget constraints, using an appropriate mix of dynamically provisioned resources. Our evaluations also demonstrate that public clouds can be used to complement and reinforce the scheduling and usage of traditional high performance computing infrastructure.

Original languageEnglish (US)
Pages (from-to)75-89
Number of pages15
JournalScientific Programming
Volume19
Issue number2-3
DOIs
StatePublished - Jan 1 2011

Fingerprint

Conservation
Scheduling
Oils

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications

Cite this

Kim, Hyunjoo ; El-Khamra, Yaakoub ; Rodero, Ivan ; Jha, Shantenu ; Parashar, Manish. / Autonomic management of application workflows on hybrid computing infrastructure. In: Scientific Programming. 2011 ; Vol. 19, No. 2-3. pp. 75-89.
@article{debd33b638da41288a3f5690156be800,
title = "Autonomic management of application workflows on hybrid computing infrastructure",
abstract = "In this paper, we present a programming and runtime framework that enables the autonomic management of complex application workflows on hybrid computing infrastructures. The framework is designed to address system and application heterogeneity and dynamics to ensure that application objectives and constraints are satisfied. The need for such autonomic system and application management is becoming critical as computing infrastructures become increasingly heterogeneous, integrating different classes of resources from high-end HPC systems to commodity clusters and clouds. For example, the framework presented in this paper can be used to provision the appropriate mix of resources based on application requirements and constraints. The framework also monitors the system/application state and adapts the application and/or resources to respond to changing requirements or environment. To demonstrate the operation of the framework and to evaluate its ability, we employ a workflow used to characterize an oil reservoir executing on a hybrid infrastructure composed of TeraGrid nodes and Amazon EC2 instances of various types. Specifically, we show how different applications objectives such as acceleration, conservation and resilience can be effectively achieved while satisfying deadline and budget constraints, using an appropriate mix of dynamically provisioned resources. Our evaluations also demonstrate that public clouds can be used to complement and reinforce the scheduling and usage of traditional high performance computing infrastructure.",
author = "Hyunjoo Kim and Yaakoub El-Khamra and Ivan Rodero and Shantenu Jha and Manish Parashar",
year = "2011",
month = "1",
day = "1",
doi = "10.3233/SPR-2011-0319",
language = "English (US)",
volume = "19",
pages = "75--89",
journal = "Scientific Programming",
issn = "1058-9244",
publisher = "IOS Press",
number = "2-3",

}

Autonomic management of application workflows on hybrid computing infrastructure. / Kim, Hyunjoo; El-Khamra, Yaakoub; Rodero, Ivan; Jha, Shantenu; Parashar, Manish.

In: Scientific Programming, Vol. 19, No. 2-3, 01.01.2011, p. 75-89.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Autonomic management of application workflows on hybrid computing infrastructure

AU - Kim, Hyunjoo

AU - El-Khamra, Yaakoub

AU - Rodero, Ivan

AU - Jha, Shantenu

AU - Parashar, Manish

PY - 2011/1/1

Y1 - 2011/1/1

N2 - In this paper, we present a programming and runtime framework that enables the autonomic management of complex application workflows on hybrid computing infrastructures. The framework is designed to address system and application heterogeneity and dynamics to ensure that application objectives and constraints are satisfied. The need for such autonomic system and application management is becoming critical as computing infrastructures become increasingly heterogeneous, integrating different classes of resources from high-end HPC systems to commodity clusters and clouds. For example, the framework presented in this paper can be used to provision the appropriate mix of resources based on application requirements and constraints. The framework also monitors the system/application state and adapts the application and/or resources to respond to changing requirements or environment. To demonstrate the operation of the framework and to evaluate its ability, we employ a workflow used to characterize an oil reservoir executing on a hybrid infrastructure composed of TeraGrid nodes and Amazon EC2 instances of various types. Specifically, we show how different applications objectives such as acceleration, conservation and resilience can be effectively achieved while satisfying deadline and budget constraints, using an appropriate mix of dynamically provisioned resources. Our evaluations also demonstrate that public clouds can be used to complement and reinforce the scheduling and usage of traditional high performance computing infrastructure.

AB - In this paper, we present a programming and runtime framework that enables the autonomic management of complex application workflows on hybrid computing infrastructures. The framework is designed to address system and application heterogeneity and dynamics to ensure that application objectives and constraints are satisfied. The need for such autonomic system and application management is becoming critical as computing infrastructures become increasingly heterogeneous, integrating different classes of resources from high-end HPC systems to commodity clusters and clouds. For example, the framework presented in this paper can be used to provision the appropriate mix of resources based on application requirements and constraints. The framework also monitors the system/application state and adapts the application and/or resources to respond to changing requirements or environment. To demonstrate the operation of the framework and to evaluate its ability, we employ a workflow used to characterize an oil reservoir executing on a hybrid infrastructure composed of TeraGrid nodes and Amazon EC2 instances of various types. Specifically, we show how different applications objectives such as acceleration, conservation and resilience can be effectively achieved while satisfying deadline and budget constraints, using an appropriate mix of dynamically provisioned resources. Our evaluations also demonstrate that public clouds can be used to complement and reinforce the scheduling and usage of traditional high performance computing infrastructure.

UR - http://www.scopus.com/inward/record.url?scp=79960794179&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79960794179&partnerID=8YFLogxK

U2 - 10.3233/SPR-2011-0319

DO - 10.3233/SPR-2011-0319

M3 - Article

AN - SCOPUS:79960794179

VL - 19

SP - 75

EP - 89

JO - Scientific Programming

JF - Scientific Programming

SN - 1058-9244

IS - 2-3

ER -