Customized expression recognition for performance-driven cutout character animation

Xiang Yu, Jianchao Yang, Linjie Luo, Wilmot Li, Jonathan Brandt, Dimitris Metaxas

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Scopus citations

Abstract

Performance-driven character animation enables users to create expressive results by performing the desired motion of the character with their face and/or body. However, for cutout animations where continuous motion is combined with discrete artwork replacements, supporting a performance-driven workflow has some unique requirements. To trigger the appropriate artwork replacements, the system must reliably detect a wide range of customized facial expressions that are challenging for existing recognition methods, which focus on a few canonical expressions (e.g., angry, disgusted, scared, happy, sad and surprised). Also, real usage scenarios require the system to work in realtime with minimal training. In this paper, we propose a novel customized expression recognition technique that meets all of these requirements. We first use a set of handcrafted features combining geometric features derived from facial landmarks and patch-based appearance features through group sparsity-based facial component learning. To improve discrimination and generalization, these handcrafted features are integrated into a custom-designed Deep Convolutional Neural Network (CNN) structure trained from publicly available facial expression datasets. The combined features are fed to an online ensemble of SVMs designed for the few training sample problem and performs in realtime. To improve temporal coherence, we also apply a Hidden Markov Model (HMM) to smooth the recognition results. Our system achieves state-of-the-art performance on canonical expression datasets and promising results on our collected dataset of customized expressions.

Original languageEnglish (US)
Title of host publication2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781509006410
DOIs
StatePublished - May 23 2016
EventIEEE Winter Conference on Applications of Computer Vision, WACV 2016 - Lake Placid, United States
Duration: Mar 7 2016Mar 10 2016

Publication series

Name2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016

Other

OtherIEEE Winter Conference on Applications of Computer Vision, WACV 2016
Country/TerritoryUnited States
CityLake Placid
Period3/7/163/10/16

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Computer Vision and Pattern Recognition

Fingerprint

Dive into the research topics of 'Customized expression recognition for performance-driven cutout character animation'. Together they form a unique fingerprint.

Cite this