Speaking with hands: Creating animated conversational characters from recordings of human performance

Matthew Stone, Doug DeCarlo, Insuk Oh, Christian Rodriguez, Adrian Stere, Alyssa Lees, Chris Bregler

Research output: Contribution to journalConference articlepeer-review

34 Scopus citations


We describe a method for using a database of recorded speech and captured motion to create an animated conversational character. People's utterances are composed of short, clearly-delimited phrases; in each phrase, gesture and speech go together meaningfully and synchronize at a common point of maximum emphasis. We develop tools for collecting and managing performance data that exploit this structure. The tools help create scripts for performers, help annotate and segment performance data, and structure specific messages for characters to use within application contexts. Our animations then reproduce this structure. They recombine motion samples with new speech samples to recreate coherent phrases, and blend segments of speech and motion together phrase-by-phrase into extended utterances. By framing problems for utterance generation and synthesis so that they can draw closely on a talented performance, our techniques support the rapid construction of animated characters with rich and appropriate expression.

Original languageEnglish (US)
Pages (from-to)506-513
Number of pages8
JournalACM Transactions on Graphics
Issue number3
StatePublished - 2004
EventACM Transactions on Graphics - Proceedings of ACM SIGGRAPH 2004 -
Duration: Aug 9 2004Aug 12 2004

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design


  • Animation
  • Conversational agents
  • Language generation
  • Motion capture
  • Motion synthesis
  • Speech synthesis


Dive into the research topics of 'Speaking with hands: Creating animated conversational characters from recordings of human performance'. Together they form a unique fingerprint.

Cite this