Converting Text into Agent Animations: Assigning Gestures to Text

Yukiko I. Nakano, Masashi Okamoto, Daisuke Kawahara, Qing Li, Toyoaki Nishida

TL;DR

This paper introduces a novel method for automatically generating synchronized gestures for animated agents based on plain text. By analyzing lexical and syntactic features of text, their system can predict and assign appropriate gestures, making agent presentations more natural and comprehensible.

What Problem Does This Paper Address?

As animated characters and interface agents become more sophisticated, there's a growing need for them to communicate naturally, much like humans do. Human communication isn't just about words; gestures play a crucial role in conveying meaning, enhancing understanding, and making interactions more engaging. While previous work has focused on generating gestures based on complex semantic representations or specific task contexts, this paper tackles the challenge of generating gestures directly from plain text. The core problem is identifying which parts of a sentence are likely to be accompanied by gestures and how to extract the necessary linguistic information from text to make these decisions automatically.

Methods

The authors propose a two-pronged approach:

  1. Empirical Study: They conducted an empirical study to identify linguistic features that correlate with gesture occurrence. This involved analyzing videotaped presentations, transcribing them, and annotating both gestures and linguistic information.
  2. System Implementation: Based on the empirical findings, they developed a system called CAST (Conversational Agent System for neTwork applications). CAST takes plain text as input and uses a Language Tagging Module (LTM) to extract linguistic features. An Agent Behavior Selection Module (ABS) then uses these features to decide whether to assign a gesture to a given text segment (bunsetsu unit) and selects an appropriate gesture from a library.

The linguistic features analyzed include:

  • Syntactic Information: NP modification by clauses/complements, case markers (e.g., "wa," "ga," "wo"), wh-interrogatives, coordination, and dependency structures.
  • Lexical Information: Emphatic adverbial phrases, cue words (e.g., "now," "therefore"), and numerals.
  • Information Structure: Identifying "given" vs. "new" information.

Datasets

The study used data from seven videotaped presentation talks. Three minutes of each talk were transcribed, resulting in 2124 "bunsetsu units" (linguistic phrases) and 343 gestures. This data was used for both the empirical analysis and the development of the gesture assignment rules. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Syntactic Structure is Key: The most significant finding is that syntactic structure, particularly coordination and clause modification of noun phrases, is a stronger predictor of gesture occurrence than simpler cues like theme/rheme or given/new information. This suggests that the way ideas are grammatically linked is crucial for gesturing.
  • Coordination is a Gesture Hotspot: Bunsetsu units that are part of a coordination (linked by conjunctions) are significantly more likely to be accompanied by gestures (47.7% probability).
  • Modifiers Matter: Noun phrases modified by clauses or complements also show a higher likelihood of gesture occurrence.
  • Specific Lexical Cues: Wh-interrogatives, cue words, and numerals act as strong indicators for gesture generation.
  • Emphatic Adverbs' Role: Gestures tend to occur after emphatic adverbs rather than with the adverb itself, suggesting a temporal relationship in how emphasis is conveyed through speech and gesture.
  • Baseline Gesture Rate: The baseline probability of a gesture occurring per bunsetsu unit was relatively low (10.1%), highlighting the importance of identifying specific predictive features.

Key Results

The empirical study identified several features that significantly increased the probability of gesture occurrence, some by 3 to 5 times the baseline rate. These features accounted for 75% of all observed gestures. Specifically:

  • Coordination: 47.7% gesture occurrence.
  • NP modified by clause: 38.2% gesture occurrence.
  • New information with "wo" case marker: 28.1% gesture occurrence.
  • Wh-interrogatives: 41.4% gesture occurrence.
  • Cue words: 41.5% gesture occurrence.
  • Numerals: 39.3% gesture occurrence.
  • Following emphatic adverb: 35% gesture occurrence.

The implemented CAST system uses these probabilities to decide whether to assign a gesture. If a specific gesture is available in the library (e.g., for "big"), it's preferred; otherwise, a default "beat gesture" (a simple hand movement) is used.

Limitations and Future Work

  • Discourse Level Information: The current model primarily relies on local lexical and syntactic information. The authors acknowledge that incorporating broader discourse-level information (e.g., episode boundaries, discourse segment structure) could further improve gesture generation.
  • User Study Needed: The authors plan to conduct user studies to evaluate the effectiveness of their agent gestures in actual human-agent interactions, aiming to confirm if presentations feel more alive and comprehensible.
  • Gesture Library: The system's ability to generate specific, meaningful gestures depends on the richness of its gesture animation library. The current implementation defaults to beat gestures when specific ones aren't defined.

Summary updated

Abstract

This paper proposes a method for assigning gestures to text based on lexical and syntactic information. First, our empirical study identified lexical and syntactic information strongly correlated with gesture occurrence and suggested that syntactic structure is more useful for judging gesture occurrence than local syntactic cues. Based on the empirical results, we have implemented a system that converts text into an animated agent that gestures and speaks synchronously.
BibTeX
@inproceedings{nakano2004converting,
  author = {Yukiko I. Nakano and Masashi Okamoto and Daisuke Kawahara and Qing Li and Toyoaki Nishida},
  title = {{Converting Text into Agent Animations: Assigning Gestures to Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure