Paradoxical Correlation Pattern Mining

Wenjun Zhou, Hui Xiong, Lian Duan, Keli Xiao, Robert Mee

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

Given a large transactional database, correlation computing/association analysis aims at efficiently finding strongly correlated items. For traditional association analysis, relationships among variables are usually measured at a global level. In this study, we investigate confounding factors that can help to capture abnormal correlation behaviors at a local level. Indeed, many real-world phenomena are localized to specific markets or subpopulations. Such local relationships may not be visible or may be miscalculated when collectively analyzing the entire data. In particular, confounding effects that change the direction of correlation are a most severe problem because the global correlations alone leads to errant conclusions. To this end, we propose CONFOUND, an efficient algorithm to identify paradoxical correlation patterns (i.e., where controlling for a third item changes the direction of association for strongly correlated pairs) using effective pruning strategies. Moreover, we also provide an enhanced version of this algorithm, called CONFOUND+, which substantially speeds up the confounder search step. Finally, experimental results showed that our proposed CONFOUND and CONFOUND+ algorithms can effectively identify confounders and the computational performance is orders of magnitude faster than benchmark methods.

Original languageEnglish (US)
Pages (from-to)1561-1574
Number of pages14
JournalIEEE Transactions on Knowledge and Data Engineering
Volume30
Issue number8
DOIs
StatePublished - Aug 1 2018

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Science Applications
  • Computational Theory and Mathematics

Keywords

  • Correlation coefficient
  • confounder
  • correlation computing
  • local patterns
  • partial correlation

Cite this