Is interactional dissynchrony a clue to deception? Insights from automated analysis of nonverbal visual cues

Xiang Yu, Shaoting Zhang, Zhennan Yan, Fei Yang, Junzhou Huang, Norah E. Dunbar, Matthew L. Jensen, Judee K. Burgoon, Dimitri Metaxas

Research output: Contribution to journalArticle

16 Scopus citations

Abstract

Detecting deception in interpersonal dialog is challenging since deceivers take advantage of the give-and-take of interaction to adapt to any sign of skepticism in an interlocutor's verbal and nonverbal feedback. Human detection accuracy is poor, often with no better than chance performance. In this investigation, we consider whether automated methods can produce better results and if emphasizing the possible disruption in interactional synchrony can signal whether an interactant is truthful or deceptive. We propose a data-driven and unobtrusive framework using visual cues that consists of face tracking, head movement detection, facial expression recognition, and interactional synchrony estimation. Analysis were conducted on 242 video samples from an experiment in which deceivers and truth-tellers interacted with professional interviewers either face-to-face or through computer mediation. Results revealed that the framework is able to automatically track head movements and expressions of both interlocutors to extract normalized meaningful synchrony features and to learn classification models for deception recognition. Further experiments show that these features reliably capture interactional synchrony and efficiently discriminate deception from truth.

Original languageEnglish (US)
Article number06845335
Pages (from-to)506-520
Number of pages15
JournalIEEE Transactions on Cybernetics
Volume45
Issue number3
DOIs
StatePublished - Mar 1 2015

All Science Journal Classification (ASJC) codes

  • Software
  • Control and Systems Engineering
  • Information Systems
  • Human-Computer Interaction
  • Computer Science Applications
  • Electrical and Electronic Engineering

Keywords

  • Deception detection
  • expression recognition
  • face tracking
  • gesture detection
  • interactional synchrony

Fingerprint Dive into the research topics of 'Is interactional dissynchrony a clue to deception? Insights from automated analysis of nonverbal visual cues'. Together they form a unique fingerprint.

  • Cite this

    Yu, X., Zhang, S., Yan, Z., Yang, F., Huang, J., Dunbar, N. E., Jensen, M. L., Burgoon, J. K., & Metaxas, D. (2015). Is interactional dissynchrony a clue to deception? Insights from automated analysis of nonverbal visual cues. IEEE Transactions on Cybernetics, 45(3), 506-520. [06845335]. https://doi.org/10.1109/TCYB.2014.2329673