Class imbalance in out-of-distribution datasets: Improving the robustness of the TextCNN for the classification of rare cancer types

Kevin De Angeli, Shang Gao, Ioana Danciu, Eric B. Durbin, Xiao Cheng Wu, Antoinette Stroup, Jennifer Doherty, Stephen Schwartz, Charles Wiggins, Mark Damesyn, Linda Coyle, Lynne Penberthy, Georgia D. Tourassi, Hong Jun Yoon

Research output: Contribution to journalArticlepeer-review

Abstract

In the last decade, the widespread adoption of electronic health record documentation has created huge opportunities for information mining. Natural language processing (NLP) techniques using machine and deep learning are becoming increasingly widespread for information extraction tasks from unstructured clinical notes. Disparities in performance when deploying machine learning models in the real world have recently received considerable attention. In the clinical NLP domain, the robustness of convolutional neural networks (CNNs) for classifying cancer pathology reports under natural distribution shifts remains understudied. In this research, we aim to quantify and improve the performance of the CNN for text classification on out-of-distribution (OOD) datasets resulting from the natural evolution of clinical text in pathology reports. We identified class imbalance due to different prevalence of cancer types as one of the sources of performance drop and analyzed the impact of previous methods for addressing class imbalance when deploying models in real-world domains. Our results show that our novel class-specialized ensemble technique outperforms other methods for the classification of rare cancer types in terms of macro F1 scores. We also found that traditional ensemble methods perform better in top classes, leading to higher micro F1 scores. Based on our findings, we formulate a series of recommendations for other ML practitioners on how to build robust models with extremely imbalanced datasets in biomedical NLP applications.

Original languageEnglish (US)
Article number103957
JournalJournal of Biomedical Informatics
Volume125
DOIs
StatePublished - Jan 2022

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Health Informatics

Keywords

  • Class Imbalance
  • CNN
  • Deep Learning
  • Ensemble
  • NLP
  • Text Classification

Fingerprint

Dive into the research topics of 'Class imbalance in out-of-distribution datasets: Improving the robustness of the TextCNN for the classification of rare cancer types'. Together they form a unique fingerprint.

Cite this