EmoLeak: Smartphone Motions Reveal Emotions

Ahmed Tanvir Mahdad, Cong Shi, Zhengkun Ye, Tianming Zhao, Yan Wang, Yingying Chen, Nitesh Saxena

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Emotional state leakage attracts increasing concerns as it reveals rich sensitive information, such as intent, demo graphic, personality, and health information. Existing emotion recognition techniques rely on vision and audio data, which have limited threat due to the requirements of accessing restricted sensors (e.g., cameras and microphones). In this work, we first investigate the feasibility of detecting the emotional state of people in the vibration domain via zero-permission motion sensors. We find that when voice is being played through a smartphone's loudspeaker or ear speaker, it generates vibration signals on the smartphone surface, which encodes rich emotional information. As the smartphone is the go-to device for almost everyone nowadays, our attack based only on motion sensors raises severe concerns about emotion state leakage. We comprehensively study the relationship between vibration data and human emotion based on several publicly available emotion datasets (e.g., SAVEE, TESS). Time-frequency features and machine learning techniques are developed to determine the emotion of the victim based on speech vibrations. We evaluate our attack on both the ear speakers and loudspeakers on a diverse set of smartphones. The results demonstrate our attack can achieve a high accuracy, with around 95.3% (random guess 14.3%) accuracy for the loudspeaker setting and 60.52% (random guess 14.3%) accuracy for the ear speaker setting.

Original languageEnglish (US)
Title of host publicationProceedings - 2023 IEEE 43rd International Conference on Distributed Computing Systems, ICDCS 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages316-326
Number of pages11
ISBN (Electronic)9798350339864
DOIs
StatePublished - 2023
Externally publishedYes
Event43rd IEEE International Conference on Distributed Computing Systems, ICDCS 2023 - Hong Kong, China
Duration: Jul 18 2023Jul 21 2023

Publication series

NameProceedings - International Conference on Distributed Computing Systems
Volume2023-July

Conference

Conference43rd IEEE International Conference on Distributed Computing Systems, ICDCS 2023
Country/TerritoryChina
CityHong Kong
Period7/18/237/21/23

All Science Journal Classification (ASJC) codes

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Keywords

  • emotion recognition
  • motion sensor
  • side channel
  • speech privacy

Fingerprint

Dive into the research topics of 'EmoLeak: Smartphone Motions Reveal Emotions'. Together they form a unique fingerprint.

Cite this