Building data civilizer pipelines with an advanced workflow engine

Essam Mansour, Dong Deng, Raul Castro Fernandez, Abdulhakim A. Qahtan, Wenbo Tao, Ziawasch Abedjan, Ahmed Elmagarmid, Ihab F. Ilyas, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, Nan Tang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

In order for an enterprise to gain insight into its internal business and the changing outside environment, it is essential to provide the relevant data for in-depth analysis. Enterprise data is usually scattered across departments and geographic regions and is often inconsistent. Data scientists spend the majority of their time finding, preparing, integrating, and cleaning relevant data sets. Data Civilizer is an end-To-end data preparation system. In this paper, we present the complete system, focusing on our new workflow engine, a superior system for entity matching and consolidation, and new cleaning tools. Our workflow engine allows data scientists to author, execute and retrofit data preparation pipelines of different data discovery and cleaning services. Our end-To-end demo scenario is based on data from the MIT data warehouse and e-commerce data sets.

Original languageEnglish (US)
Title of host publicationProceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1593-1596
Number of pages4
ISBN (Electronic)9781538655207
DOIs
StatePublished - Oct 24 2018
Externally publishedYes
Event34th IEEE International Conference on Data Engineering, ICDE 2018 - Paris, France
Duration: Apr 16 2018Apr 19 2018

Publication series

NameProceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018

Conference

Conference34th IEEE International Conference on Data Engineering, ICDE 2018
Country/TerritoryFrance
CityParis
Period4/16/184/19/18

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Information Systems and Management
  • Hardware and Architecture

Keywords

  • Data Cleaning
  • Data Discovery
  • Data Integration

Fingerprint

Dive into the research topics of 'Building data civilizer pipelines with an advanced workflow engine'. Together they form a unique fingerprint.

Cite this