Mining and tracking massive text data: Classification, construction of tracking statistics, and inference under misclassification

Daniel R. Jeske; Regina Y. Liu

doi:10.1198/004017006000000471

Mining and tracking massive text data: Classification, construction of tracking statistics, and inference under misclassification

Daniel R. Jeske, Regina Y. Liu

School of Arts and Sciences, Statistics

Research output: Contribution to journal › Article › peer-review

12 Scopus citations

Abstract

This article presents a comprehensive data-mining procedure for exploring large freestyle text datasets to discover useful features and develop suitable tracking statistics (often referred to as performance measures or risk indicators). The procedure includes text classification, construction of tracking statistics, inference under error measurements, and risk analysis. Some specific text analysis methodologies and tracking statistics are discussed. Several approaches for incorporating misclassified data or error measurements into the inference for tracking statistics are proposed and evaluated. Finally, as an illustrative example, the proposed data-mining procedure is applied to analyzing an aviation safety report repository to show its utility in aviation risk management or general decision-support systems.

Original language	English (US)
Pages (from-to)	116-128
Number of pages	13
Journal	Technometrics
Volume	49
Issue number	2
DOIs	https://doi.org/10.1198/004017006000000471
State	Published - May 2007

All Science Journal Classification (ASJC) codes

Statistics and Probability
Modeling and Simulation
Applied Mathematics

Keywords

Data mining
Misclassification
Risk indicator
Text classification
Tracking statistic

Access to Document

10.1198/004017006000000471

Cite this

@article{1e18215c6aa84619a0eb631495d21a68,

title = "Mining and tracking massive text data: Classification, construction of tracking statistics, and inference under misclassification",

abstract = "This article presents a comprehensive data-mining procedure for exploring large freestyle text datasets to discover useful features and develop suitable tracking statistics (often referred to as performance measures or risk indicators). The procedure includes text classification, construction of tracking statistics, inference under error measurements, and risk analysis. Some specific text analysis methodologies and tracking statistics are discussed. Several approaches for incorporating misclassified data or error measurements into the inference for tracking statistics are proposed and evaluated. Finally, as an illustrative example, the proposed data-mining procedure is applied to analyzing an aviation safety report repository to show its utility in aviation risk management or general decision-support systems.",

keywords = "Data mining, Misclassification, Risk indicator, Text classification, Tracking statistic",

author = "Jeske, {Daniel R.} and Liu, {Regina Y.}",

note = "Funding Information: This research is supported in part by grants from the National Science Foundation and the Federal Aviation Administration. The authors thank Yingtao Bi, Jun Li, and Qi Xia for their computing assistance. They also thank the referees and the associate editor for helpful comments that have helped improve this article. The discussion on aviation safety reflects the views of the authors, who are solely responsible for the accuracy of the analysis results presented herein, and does not necessarily reflect the official view or policy of the FAA. The dataset used in this article has been partially masked to protect confidentiality.",

year = "2007",

month = may,

doi = "10.1198/004017006000000471",

language = "English (US)",

volume = "49",

pages = "116--128",

journal = "Technometrics",

issn = "0040-1706",

publisher = "Taylor and Francis Ltd.",

number = "2",

}

TY - JOUR

T1 - Mining and tracking massive text data

T2 - Classification, construction of tracking statistics, and inference under misclassification

AU - Jeske, Daniel R.

AU - Liu, Regina Y.

N1 - Funding Information: This research is supported in part by grants from the National Science Foundation and the Federal Aviation Administration. The authors thank Yingtao Bi, Jun Li, and Qi Xia for their computing assistance. They also thank the referees and the associate editor for helpful comments that have helped improve this article. The discussion on aviation safety reflects the views of the authors, who are solely responsible for the accuracy of the analysis results presented herein, and does not necessarily reflect the official view or policy of the FAA. The dataset used in this article has been partially masked to protect confidentiality.

PY - 2007/5

Y1 - 2007/5

N2 - This article presents a comprehensive data-mining procedure for exploring large freestyle text datasets to discover useful features and develop suitable tracking statistics (often referred to as performance measures or risk indicators). The procedure includes text classification, construction of tracking statistics, inference under error measurements, and risk analysis. Some specific text analysis methodologies and tracking statistics are discussed. Several approaches for incorporating misclassified data or error measurements into the inference for tracking statistics are proposed and evaluated. Finally, as an illustrative example, the proposed data-mining procedure is applied to analyzing an aviation safety report repository to show its utility in aviation risk management or general decision-support systems.

AB - This article presents a comprehensive data-mining procedure for exploring large freestyle text datasets to discover useful features and develop suitable tracking statistics (often referred to as performance measures or risk indicators). The procedure includes text classification, construction of tracking statistics, inference under error measurements, and risk analysis. Some specific text analysis methodologies and tracking statistics are discussed. Several approaches for incorporating misclassified data or error measurements into the inference for tracking statistics are proposed and evaluated. Finally, as an illustrative example, the proposed data-mining procedure is applied to analyzing an aviation safety report repository to show its utility in aviation risk management or general decision-support systems.

KW - Data mining

KW - Misclassification

KW - Risk indicator

KW - Text classification

KW - Tracking statistic

UR - http://www.scopus.com/inward/record.url?scp=34249083426&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=34249083426&partnerID=8YFLogxK

U2 - 10.1198/004017006000000471

DO - 10.1198/004017006000000471

M3 - Article

AN - SCOPUS:34249083426

SN - 0040-1706

VL - 49

SP - 116

EP - 128

JO - Technometrics

JF - Technometrics

IS - 2

ER -

Mining and tracking massive text data: Classification, construction of tracking statistics, and inference under misclassification

Abstract

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this