Ensemble Toolkit: Scalable and Flexible Execution of Ensembles of Tasks

Vivekanandan Balasubramanian, Antons Treikalis, Ole Weidner, Shantenu Jha

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Scopus citations


There are many science applications that requirescalable task-level parallelism, support for flexible execution andcoupling of ensembles of simulations. Most high-performancesystem software and middleware, however, are designed tosupport the execution and optimization of single tasks. Motivatedby the missing capabilities of these computing systems and theincreasing importance of task-level parallelism, we introduce theEnsemble toolkit which has the following application developmentfeatures: (i) abstractions that enable the expression of ensemblesas primary entities, and (ii) support for ensemble-based executionpatterns that capture the majority of application scenarios. Ensemble toolkit uses a scalable pilot-based runtime systemthat decouples workload execution and resource managementdetails from the expression of the application, and enables theefficient and dynamic execution of ensembles on heterogeneouscomputing resources. We investigate three execution patterns andcharacterize the scalability and overhead of Ensemble toolkit forthese patterns. We investigate scaling properties for up to O(1000)concurrent ensembles and O(1000) cores and find linear weakand strong scaling behaviour.

Original languageEnglish (US)
Title of host publicationProceedings - 45th International Conference on Parallel Processing, ICPP 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages6
ISBN (Electronic)9781509028238
StatePublished - Sep 21 2016
Event45th International Conference on Parallel Processing, ICPP 2016 - Philadelphia, United States
Duration: Aug 16 2016Aug 19 2016

Publication series

NameProceedings of the International Conference on Parallel Processing
ISSN (Print)0190-3918


Other45th International Conference on Parallel Processing, ICPP 2016
Country/TerritoryUnited States

All Science Journal Classification (ASJC) codes

  • Software
  • Mathematics(all)
  • Hardware and Architecture


Dive into the research topics of 'Ensemble Toolkit: Scalable and Flexible Execution of Ensembles of Tasks'. Together they form a unique fingerprint.

Cite this