3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL

Bo Liu; Jingjing Liu; Xiang Yu; Dimitris Metaxas; Carol Neidle

3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL

Bo Liu, Jingjing Liu, Xiang Yu, Dimitris Metaxas, Carol Neidle

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Essential grammatical information is conveyed in signed languages by clusters of events involving facial expressions and movements of the head and upper body. This poses a significant challenge for computer-based sign language recognition. Here, we present new methods for the recognition of nonmanual grammatical markers in American Sign Language (ASL) based on: (1) new 3D tracking methods for the estimation of 3D head pose and facial expressions to determine the relevant low-level features; (2) methods for higher-level analysis of component events (raised/lowered eyebrows, periodic head nods and head shakes) used in grammatical markings - with differentiation of temporal phases (onset, core, offset, where appropriate), analysis of their characteristic properties, and extraction of corresponding features; (3) a 2-level learning framework to combine low- and high-level features of differing spatio-temporal scales. This new approach achieves significantly better tracking and recognition results than our previous methods.

Original language	English (US)
Title of host publication	Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014
Editors	Nicoletta Calzolari, Khalid Choukri, Sara Goggi, Thierry Declerck, Joseph Mariani, Bente Maegaard, Asuncion Moreno, Jan Odijk, Helene Mazo, Stelios Piperidis, Hrafn Loftsson
Publisher	European Language Resources Association (ELRA)
Pages	4512-4518
Number of pages	7
ISBN (Electronic)	9782951740884
State	Published - 2014
Event	9th International Conference on Language Resources and Evaluation, LREC 2014 - Reykjavik, Iceland Duration: May 26 2014 → May 31 2014

Publication series

Name	Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014

Other

Other	9th International Conference on Language Resources and Evaluation, LREC 2014
Country/Territory	Iceland
City	Reykjavik
Period	5/26/14 → 5/31/14

All Science Journal Classification (ASJC) codes

Linguistics and Language
Library and Information Sciences
Education
Language and Linguistics

Keywords

ASL
Facial expression
Head position
Nonmanual Marker
Spatio-temporal

Cite this

Liu, B., Liu, J., Yu, X., Metaxas, D., & Neidle, C. (2014). 3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL. In N. Calzolari, K. Choukri, S. Goggi, T. Declerck, J. Mariani, B. Maegaard, A. Moreno, J. Odijk, H. Mazo, S. Piperidis, & H. Loftsson (Eds.), Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014 (pp. 4512-4518). (Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014). European Language Resources Association (ELRA).

Liu, Bo ; Liu, Jingjing ; Yu, Xiang ; Metaxas, Dimitris ; Neidle, Carol. / 3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL. Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014. editor / Nicoletta Calzolari ; Khalid Choukri ; Sara Goggi ; Thierry Declerck ; Joseph Mariani ; Bente Maegaard ; Asuncion Moreno ; Jan Odijk ; Helene Mazo ; Stelios Piperidis ; Hrafn Loftsson. European Language Resources Association (ELRA), 2014. pp. 4512-4518 (Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014).

@inproceedings{c82d28881fac45ecb4d0d5f106fc3761,

title = "3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL",

abstract = "Essential grammatical information is conveyed in signed languages by clusters of events involving facial expressions and movements of the head and upper body. This poses a significant challenge for computer-based sign language recognition. Here, we present new methods for the recognition of nonmanual grammatical markers in American Sign Language (ASL) based on: (1) new 3D tracking methods for the estimation of 3D head pose and facial expressions to determine the relevant low-level features; (2) methods for higher-level analysis of component events (raised/lowered eyebrows, periodic head nods and head shakes) used in grammatical markings - with differentiation of temporal phases (onset, core, offset, where appropriate), analysis of their characteristic properties, and extraction of corresponding features; (3) a 2-level learning framework to combine low- and high-level features of differing spatio-temporal scales. This new approach achieves significantly better tracking and recognition results than our previous methods.",

keywords = "ASL, Facial expression, Head position, Nonmanual Marker, Spatio-temporal",

author = "Bo Liu and Jingjing Liu and Xiang Yu and Dimitris Metaxas and Carol Neidle",

note = "Funding Information: The research reported here was partially funded by grants from the National Science Foundation (CNS-1059281, IIS-1064965, IIS-0964597, IIS-1065013, and CNS-0964385). We gratefully acknowledge invaluable assistance from Rachel Benedict, Braden Painter, Iryna Zhuravlova, Jessica Scott, Joan Nash, Tory Sampson, Indya Oliver, Corbin Kuntze, Amelia Wisniewski-Barker, and many other BU students.; 9th International Conference on Language Resources and Evaluation, LREC 2014 ; Conference date: 26-05-2014 Through 31-05-2014",

year = "2014",

language = "English (US)",

series = "Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014",

publisher = "European Language Resources Association (ELRA)",

pages = "4512--4518",

editor = "Nicoletta Calzolari and Khalid Choukri and Sara Goggi and Thierry Declerck and Joseph Mariani and Bente Maegaard and Asuncion Moreno and Jan Odijk and Helene Mazo and Stelios Piperidis and Hrafn Loftsson",

booktitle = "Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014",

}

Liu, B, Liu, J, Yu, X, Metaxas, D & Neidle, C 2014, 3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL. in N Calzolari, K Choukri, S Goggi, T Declerck, J Mariani, B Maegaard, A Moreno, J Odijk, H Mazo, S Piperidis & H Loftsson (eds), Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014. Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014, European Language Resources Association (ELRA), pp. 4512-4518, 9th International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, 5/26/14.

3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL. / Liu, Bo; Liu, Jingjing; Yu, Xiang; Metaxas, Dimitris; Neidle, Carol.

Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014. ed. / Nicoletta Calzolari; Khalid Choukri; Sara Goggi; Thierry Declerck; Joseph Mariani; Bente Maegaard; Asuncion Moreno; Jan Odijk; Helene Mazo; Stelios Piperidis; Hrafn Loftsson. European Language Resources Association (ELRA), 2014. p. 4512-4518 (Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - 3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL

AU - Liu, Bo

AU - Liu, Jingjing

AU - Yu, Xiang

AU - Metaxas, Dimitris

AU - Neidle, Carol

N1 - Funding Information: The research reported here was partially funded by grants from the National Science Foundation (CNS-1059281, IIS-1064965, IIS-0964597, IIS-1065013, and CNS-0964385). We gratefully acknowledge invaluable assistance from Rachel Benedict, Braden Painter, Iryna Zhuravlova, Jessica Scott, Joan Nash, Tory Sampson, Indya Oliver, Corbin Kuntze, Amelia Wisniewski-Barker, and many other BU students.

PY - 2014

Y1 - 2014

N2 - Essential grammatical information is conveyed in signed languages by clusters of events involving facial expressions and movements of the head and upper body. This poses a significant challenge for computer-based sign language recognition. Here, we present new methods for the recognition of nonmanual grammatical markers in American Sign Language (ASL) based on: (1) new 3D tracking methods for the estimation of 3D head pose and facial expressions to determine the relevant low-level features; (2) methods for higher-level analysis of component events (raised/lowered eyebrows, periodic head nods and head shakes) used in grammatical markings - with differentiation of temporal phases (onset, core, offset, where appropriate), analysis of their characteristic properties, and extraction of corresponding features; (3) a 2-level learning framework to combine low- and high-level features of differing spatio-temporal scales. This new approach achieves significantly better tracking and recognition results than our previous methods.

AB - Essential grammatical information is conveyed in signed languages by clusters of events involving facial expressions and movements of the head and upper body. This poses a significant challenge for computer-based sign language recognition. Here, we present new methods for the recognition of nonmanual grammatical markers in American Sign Language (ASL) based on: (1) new 3D tracking methods for the estimation of 3D head pose and facial expressions to determine the relevant low-level features; (2) methods for higher-level analysis of component events (raised/lowered eyebrows, periodic head nods and head shakes) used in grammatical markings - with differentiation of temporal phases (onset, core, offset, where appropriate), analysis of their characteristic properties, and extraction of corresponding features; (3) a 2-level learning framework to combine low- and high-level features of differing spatio-temporal scales. This new approach achieves significantly better tracking and recognition results than our previous methods.

KW - ASL

KW - Facial expression

KW - Head position

KW - Nonmanual Marker

KW - Spatio-temporal

UR - http://www.scopus.com/inward/record.url?scp=85037148075&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85037148075&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85037148075

T3 - Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014

SP - 4512

EP - 4518

BT - Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014

A2 - Calzolari, Nicoletta

A2 - Choukri, Khalid

A2 - Goggi, Sara

A2 - Declerck, Thierry

A2 - Mariani, Joseph

A2 - Maegaard, Bente

A2 - Moreno, Asuncion

A2 - Odijk, Jan

A2 - Mazo, Helene

A2 - Piperidis, Stelios

A2 - Loftsson, Hrafn

PB - European Language Resources Association (ELRA)

T2 - 9th International Conference on Language Resources and Evaluation, LREC 2014

Y2 - 26 May 2014 through 31 May 2014

ER -

Liu B, Liu J, Yu X, Metaxas D, Neidle C. 3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL. In Calzolari N, Choukri K, Goggi S, Declerck T, Mariani J, Maegaard B, Moreno A, Odijk J, Mazo H, Piperidis S, Loftsson H, editors, Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014. European Language Resources Association (ELRA). 2014. p. 4512-4518. (Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014).

3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL

Abstract

Publication series

Other

All Science Journal Classification (ASJC) codes

Keywords

Other files and links

Fingerprint

Cite this