An analysis of speech as a modality for activity recognition during complex medical teamwork

Swathi Jagannath, Aleksandra Sarcevic, Ivan Marsic

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

We analyzed the nature of verbal communication among team members in a dynamic medical setting of trauma resuscitation to inform the design of a speech-based automatic activity recognition system. Using speech transcripts from 20 resuscitations, we identified common keywords and speech patterns for different resuscitation activities. Based on these patterns, we developed narrative schemas (speech -workflow- models) for five most frequently performed activities and applied linguistic models to represent relationships between sentences. We evaluated the narrative schemas with 17 new cases, finding that all five schemas adequately represented speech during activities and could serve as a basis for speech-based activity recognition. We also identified similarities between narrative schemas of different activities. We conclude with design implications and challenges associated with speechbased activity recognition in complex medical processes.

Original languageEnglish (US)
Title of host publicationProceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018
PublisherAssociation for Computing Machinery
Pages88-97
Number of pages10
ISBN (Electronic)9781450364508
DOIs
StatePublished - May 21 2018
Event12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018 - New York, United States
Duration: May 21 2018May 24 2018

Publication series

NameACM International Conference Proceeding Series

Other

Other12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018
CountryUnited States
CityNew York
Period5/21/185/24/18

Fingerprint

Resuscitation
Linguistics
Communication

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications

Keywords

  • Activity recognition
  • Decision support
  • Emergency medicine
  • Narrative schema
  • Speech analysis
  • Speech modeling

Cite this

Jagannath, S., Sarcevic, A., & Marsic, I. (2018). An analysis of speech as a modality for activity recognition during complex medical teamwork. In Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018 (pp. 88-97). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3240925.3240941
Jagannath, Swathi ; Sarcevic, Aleksandra ; Marsic, Ivan. / An analysis of speech as a modality for activity recognition during complex medical teamwork. Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018. Association for Computing Machinery, 2018. pp. 88-97 (ACM International Conference Proceeding Series).
@inproceedings{0fb263fef85945dd9c640712027c091e,
title = "An analysis of speech as a modality for activity recognition during complex medical teamwork",
abstract = "We analyzed the nature of verbal communication among team members in a dynamic medical setting of trauma resuscitation to inform the design of a speech-based automatic activity recognition system. Using speech transcripts from 20 resuscitations, we identified common keywords and speech patterns for different resuscitation activities. Based on these patterns, we developed narrative schemas (speech -workflow- models) for five most frequently performed activities and applied linguistic models to represent relationships between sentences. We evaluated the narrative schemas with 17 new cases, finding that all five schemas adequately represented speech during activities and could serve as a basis for speech-based activity recognition. We also identified similarities between narrative schemas of different activities. We conclude with design implications and challenges associated with speechbased activity recognition in complex medical processes.",
keywords = "Activity recognition, Decision support, Emergency medicine, Narrative schema, Speech analysis, Speech modeling",
author = "Swathi Jagannath and Aleksandra Sarcevic and Ivan Marsic",
year = "2018",
month = "5",
day = "21",
doi = "10.1145/3240925.3240941",
language = "English (US)",
series = "ACM International Conference Proceeding Series",
publisher = "Association for Computing Machinery",
pages = "88--97",
booktitle = "Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018",

}

Jagannath, S, Sarcevic, A & Marsic, I 2018, An analysis of speech as a modality for activity recognition during complex medical teamwork. in Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 88-97, 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018, New York, United States, 5/21/18. https://doi.org/10.1145/3240925.3240941

An analysis of speech as a modality for activity recognition during complex medical teamwork. / Jagannath, Swathi; Sarcevic, Aleksandra; Marsic, Ivan.

Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018. Association for Computing Machinery, 2018. p. 88-97 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - An analysis of speech as a modality for activity recognition during complex medical teamwork

AU - Jagannath, Swathi

AU - Sarcevic, Aleksandra

AU - Marsic, Ivan

PY - 2018/5/21

Y1 - 2018/5/21

N2 - We analyzed the nature of verbal communication among team members in a dynamic medical setting of trauma resuscitation to inform the design of a speech-based automatic activity recognition system. Using speech transcripts from 20 resuscitations, we identified common keywords and speech patterns for different resuscitation activities. Based on these patterns, we developed narrative schemas (speech -workflow- models) for five most frequently performed activities and applied linguistic models to represent relationships between sentences. We evaluated the narrative schemas with 17 new cases, finding that all five schemas adequately represented speech during activities and could serve as a basis for speech-based activity recognition. We also identified similarities between narrative schemas of different activities. We conclude with design implications and challenges associated with speechbased activity recognition in complex medical processes.

AB - We analyzed the nature of verbal communication among team members in a dynamic medical setting of trauma resuscitation to inform the design of a speech-based automatic activity recognition system. Using speech transcripts from 20 resuscitations, we identified common keywords and speech patterns for different resuscitation activities. Based on these patterns, we developed narrative schemas (speech -workflow- models) for five most frequently performed activities and applied linguistic models to represent relationships between sentences. We evaluated the narrative schemas with 17 new cases, finding that all five schemas adequately represented speech during activities and could serve as a basis for speech-based activity recognition. We also identified similarities between narrative schemas of different activities. We conclude with design implications and challenges associated with speechbased activity recognition in complex medical processes.

KW - Activity recognition

KW - Decision support

KW - Emergency medicine

KW - Narrative schema

KW - Speech analysis

KW - Speech modeling

UR - http://www.scopus.com/inward/record.url?scp=85055683866&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055683866&partnerID=8YFLogxK

U2 - 10.1145/3240925.3240941

DO - 10.1145/3240925.3240941

M3 - Conference contribution

AN - SCOPUS:85055683866

T3 - ACM International Conference Proceeding Series

SP - 88

EP - 97

BT - Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018

PB - Association for Computing Machinery

ER -

Jagannath S, Sarcevic A, Marsic I. An analysis of speech as a modality for activity recognition during complex medical teamwork. In Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2018. Association for Computing Machinery. 2018. p. 88-97. (ACM International Conference Proceeding Series). https://doi.org/10.1145/3240925.3240941