Speech-Based Activity Recognition for Trauma Resuscitation

Jalal Abdulbaqi, Yue Gu, Zhichao Xu, Chenyang Gao, Ivan Marsic, Randall S. Burd

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We present a speech-based approach to recognize team activities in the context of trauma resuscitation. We first analyzed the audio recordings of trauma resuscitations in terms of activity frequency, noise-level, and activity-related keyword frequency to determine the dataset characteristics. We next evaluated different audio-preprocessing parameters (spectral feature types and audio channels) to find the optimal configuration. We then introduced a novel neural network to recognize the trauma activities using a modified VGG network that extracts features from the audio input. The output of the modified VGG network is combined with the output of a network that takes keyword text as input, and the combination is used to generate activity labels. We compared our system with several baselines and performed a detailed analysis of the performance results for specific activities. Our results show that our proposed architecture that uses Mel-spectrum spectral coefficients features with a stereo channel and activity-specific frequent keywords achieve the highest accuracy and average F1-score.

Original languageEnglish (US)
Title of host publication2020 IEEE International Conference on Healthcare Informatics, ICHI 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728153827
StatePublished - Nov 2020
Event8th IEEE International Conference on Healthcare Informatics, ICHI 2020 - Virtual, Oldenburg, Germany
Duration: Nov 30 2020Dec 3 2020

Publication series

Name2020 IEEE International Conference on Healthcare Informatics, ICHI 2020


Conference8th IEEE International Conference on Healthcare Informatics, ICHI 2020
CityVirtual, Oldenburg

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Decision Sciences (miscellaneous)
  • Modeling and Simulation
  • Medicine (miscellaneous)
  • Health Informatics
  • Health(social science)


  • activity recognition
  • audio classification
  • keyword
  • speech processing
  • trauma resuscitation


Dive into the research topics of 'Speech-Based Activity Recognition for Trauma Resuscitation'. Together they form a unique fingerprint.

Cite this