Misinformation Detection Algorithms and Fairness across Political Ideologies: The Impact of Article Level Labeling

Jinkyung Park, Rahul Dev Ellezhuthil, Joseph Isaac, Christoph Mergerson, Lauren Feldman, Vivek Singh

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Multiple recent efforts have used large-scale data and computational models to automatically detect misinformation in online news articles. Given the potential impact of misinformation on democracy, many of these efforts have also used the political ideology of these articles to better model misinformation and study political bias in such algorithms. However, almost all such efforts have used source level labels for credibility and political alignment, thereby assigning the same credibility and political alignment label to all articles from the same source (e.g., the New York Times or Breitbart). Here, we report on the impact of journalistic best practices to label individual news articles for their credibility and political alignment. We found that while source level labels are decent proxies for political alignment labeling, they are very poor proxies-almost the same as flipping a coin-for credibility ratings. Next, we study the implications of such source level labeling on downstream processes such as the development of automated misinformation detection algorithms and political fairness audits therein. We find that the automated misinformation detection and fairness algorithms can be suitably revised to support their intended goals but might require different assumptions and methods than those which are appropriate using source level labeling. The results suggest caution in generalizing recent results on misinformation detection and political bias therein. On a positive note, this work shares a new dataset of journalistic quality individually labeled articles and an approach for misinformation detection and fairness audits.

Original languageEnglish (US)
Title of host publicationWebSci 2023 - Proceedings of the 15th ACM Web Science Conference
PublisherAssociation for Computing Machinery
Number of pages10
ISBN (Electronic)9798400700897
StatePublished - Apr 30 2023
Event15th ACM Web Science Conference, WebSci 2023 - Austin, United States
Duration: Apr 30 2023May 1 2023

Publication series

NameACM International Conference Proceeding Series


Conference15th ACM Web Science Conference, WebSci 2023
Country/TerritoryUnited States

All Science Journal Classification (ASJC) codes

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software


  • algorithmic fairness
  • article level labeling
  • misinformation detection
  • political bias


Dive into the research topics of 'Misinformation Detection Algorithms and Fairness across Political Ideologies: The Impact of Article Level Labeling'. Together they form a unique fingerprint.

Cite this