HPC-ABDS high performance computing enhanced apache big data stack

Geoffrey C. Fox, Judy Qiu, Supun Kamburugamuve, Shantenu Jha, Andre Luckow

Research output: Chapter in Book/Report/Conference proceedingConference contribution

13 Citations (Scopus)

Abstract

We review the High Performance Computing Enhanced Apache Big Data Stack HPC-ABDS and summarize the capabilities in 21 identified architecture layers. These cover Message and Data Protocols, Distributed Coordination, Security & Privacy, Monitoring, Infrastructure Management, DevOps, Interoperability, File Systems, Cluster & Resource management, Data Transport, File management, NoSQL, SQL (NewSQL), Extraction Tools, Object-relational mapping, In-memory caching and databases, Inter-process Communication, Batch Programming model and Runtime, Stream Processing, High-level Programming, Application Hosting and PaaS, Libraries and Applications, Workflow and Orchestration. We summarize status of these layers focusing on issues of importance for data analytics. We highlight areas where HPC and ABDS have good opportunities for integration.

Original languageEnglish (US)
Title of host publicationProceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1057-1066
Number of pages10
ISBN (Electronic)9781479980062
DOIs
StatePublished - Jul 7 2015
Event15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015 - Shenzhen, China
Duration: May 4 2015May 7 2015

Publication series

NameProceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015

Other

Other15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015
CountryChina
CityShenzhen
Period5/4/155/7/15

Fingerprint

Interoperability
Information management
Data storage equipment
Monitoring
Communication
Processing
Big data

All Science Journal Classification (ASJC) codes

  • Computer Science (miscellaneous)
  • Computer Networks and Communications
  • Software

Keywords

  • Apache big data stack
  • HPC

Cite this

Fox, G. C., Qiu, J., Kamburugamuve, S., Jha, S., & Luckow, A. (2015). HPC-ABDS high performance computing enhanced apache big data stack. In Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015 (pp. 1057-1066). [7152592] (Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CCGrid.2015.122
Fox, Geoffrey C. ; Qiu, Judy ; Kamburugamuve, Supun ; Jha, Shantenu ; Luckow, Andre. / HPC-ABDS high performance computing enhanced apache big data stack. Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 1057-1066 (Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015).
@inproceedings{7f3ab2a56be64d62bacd45908fb5db5d,
title = "HPC-ABDS high performance computing enhanced apache big data stack",
abstract = "We review the High Performance Computing Enhanced Apache Big Data Stack HPC-ABDS and summarize the capabilities in 21 identified architecture layers. These cover Message and Data Protocols, Distributed Coordination, Security & Privacy, Monitoring, Infrastructure Management, DevOps, Interoperability, File Systems, Cluster & Resource management, Data Transport, File management, NoSQL, SQL (NewSQL), Extraction Tools, Object-relational mapping, In-memory caching and databases, Inter-process Communication, Batch Programming model and Runtime, Stream Processing, High-level Programming, Application Hosting and PaaS, Libraries and Applications, Workflow and Orchestration. We summarize status of these layers focusing on issues of importance for data analytics. We highlight areas where HPC and ABDS have good opportunities for integration.",
keywords = "Apache big data stack, HPC",
author = "Fox, {Geoffrey C.} and Judy Qiu and Supun Kamburugamuve and Shantenu Jha and Andre Luckow",
year = "2015",
month = "7",
day = "7",
doi = "10.1109/CCGrid.2015.122",
language = "English (US)",
series = "Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "1057--1066",
booktitle = "Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015",
address = "United States",

}

Fox, GC, Qiu, J, Kamburugamuve, S, Jha, S & Luckow, A 2015, HPC-ABDS high performance computing enhanced apache big data stack. in Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015., 7152592, Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015, Institute of Electrical and Electronics Engineers Inc., pp. 1057-1066, 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015, Shenzhen, China, 5/4/15. https://doi.org/10.1109/CCGrid.2015.122

HPC-ABDS high performance computing enhanced apache big data stack. / Fox, Geoffrey C.; Qiu, Judy; Kamburugamuve, Supun; Jha, Shantenu; Luckow, Andre.

Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015. Institute of Electrical and Electronics Engineers Inc., 2015. p. 1057-1066 7152592 (Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - HPC-ABDS high performance computing enhanced apache big data stack

AU - Fox, Geoffrey C.

AU - Qiu, Judy

AU - Kamburugamuve, Supun

AU - Jha, Shantenu

AU - Luckow, Andre

PY - 2015/7/7

Y1 - 2015/7/7

N2 - We review the High Performance Computing Enhanced Apache Big Data Stack HPC-ABDS and summarize the capabilities in 21 identified architecture layers. These cover Message and Data Protocols, Distributed Coordination, Security & Privacy, Monitoring, Infrastructure Management, DevOps, Interoperability, File Systems, Cluster & Resource management, Data Transport, File management, NoSQL, SQL (NewSQL), Extraction Tools, Object-relational mapping, In-memory caching and databases, Inter-process Communication, Batch Programming model and Runtime, Stream Processing, High-level Programming, Application Hosting and PaaS, Libraries and Applications, Workflow and Orchestration. We summarize status of these layers focusing on issues of importance for data analytics. We highlight areas where HPC and ABDS have good opportunities for integration.

AB - We review the High Performance Computing Enhanced Apache Big Data Stack HPC-ABDS and summarize the capabilities in 21 identified architecture layers. These cover Message and Data Protocols, Distributed Coordination, Security & Privacy, Monitoring, Infrastructure Management, DevOps, Interoperability, File Systems, Cluster & Resource management, Data Transport, File management, NoSQL, SQL (NewSQL), Extraction Tools, Object-relational mapping, In-memory caching and databases, Inter-process Communication, Batch Programming model and Runtime, Stream Processing, High-level Programming, Application Hosting and PaaS, Libraries and Applications, Workflow and Orchestration. We summarize status of these layers focusing on issues of importance for data analytics. We highlight areas where HPC and ABDS have good opportunities for integration.

KW - Apache big data stack

KW - HPC

UR - http://www.scopus.com/inward/record.url?scp=84941248013&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84941248013&partnerID=8YFLogxK

U2 - 10.1109/CCGrid.2015.122

DO - 10.1109/CCGrid.2015.122

M3 - Conference contribution

AN - SCOPUS:84941248013

T3 - Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015

SP - 1057

EP - 1066

BT - Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Fox GC, Qiu J, Kamburugamuve S, Jha S, Luckow A. HPC-ABDS high performance computing enhanced apache big data stack. In Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015. Institute of Electrical and Electronics Engineers Inc. 2015. p. 1057-1066. 7152592. (Proceedings - 2015 IEEE/ACM 15th International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015). https://doi.org/10.1109/CCGrid.2015.122