Speaking with Hands: Creating Animated Conversational Characters
from Recordings of Human Performance

Matthew Stone
Rutgers University 


Friday, September 10
2:30PM          
Room 102       
Weaver Hall
251 Mercer Street
NYU, NYC, NYS 

ABSTRACT
--------

People's utterances in conversation are composed of short,
clearly-delimited phrases; in each phrase, gesture and speech go
together meaningfully and synchronize at a common point of maximum
emphasis.  In our work on RUTH, the Rutgers University Talking Head,
we rely on this structure to create general-purpose annotation and
synthesis to animate an expressive talking face.  In our ongoing work,
we also exploit this structure in methods to create animated
conversational characters using databases of recorded speech and
captured motion.  We start with tools that help create scripts for
performers, help annotate and segment performance data, and construct
specific messages for characters to use within application contexts.
Our animations then reproduce the structure of natural utterances.
They recombine motion samples with new speech samples to recreate
coherent phrases, and blend segments of speech and motion together
phrase-by-phrase into extended utterances.  By framing problems for
utterance generation and synthesis so that they can draw closely on a
talented performance, our techniques support the rapid construction of
animated characters with rich and appropriate expression.