Social Interaction: Multimodal Conversation with Social Agents

Katashi Nagao, Akikazu TakeuchiAAAI 1994

Introduction

This 1994 paper from Sony Computer Science Laboratory represents a foundational moment in the shift from "tools" to "partners" in artificial intelligence. Long before modern voice assistants, Nagao and Takeuchi argued that for computers to truly interact with humans, they must be more than autonomous—they must be social.

The Problem: Beyond Passive Interfaces

In the early 1990s, human-computer interaction (HCI) was primarily "dialogical" (one-to-one), "transformational" (input-output), and "passive." This model failed to capture the co-constructive nature of real human conversation, which is often messy, involves multiple participants, and relies heavily on non-verbal cues. The authors sought to build an agent capable of existing in this "ill-structured" social environment as an active participant.

System Architecture and Methods

The researchers developed a prototype social agent equipped with a 3D animated face, a video camera ("eyes"), and a microphone ("ears"). The architecture rests on three pillars:

1. Horizontal Decomposition

Inspired by Rodney Brooks’ subsumption architecture, the agent’s cognitive structure is organized into parallel layers. Lower layers manage reactive behaviors, such as detecting movement or wandering gaze, while upper layers handle deliberate reasoning, such as topic identification and planning.

The horizontal decomposition of the social agent's architecture, showing parallel layers from reactive detection to deliberate reasoning.

2. Multi-agent Plan Recognition

Unlike standard models that track only one user’s goals, this system maintains a "belief space" for every participant. It uses logic to infer what one agent believes another knows, allowing it to track complex group dynamics. This is supported by domain knowledge, such as Kautz’s Cooking Plan Library, to model the context of human activities.

3. Multimodal Synthesis

The agent features a 3D face model with 16 simulated muscles based on the Facial Action Coding System (FACS). This allows it to express emotions and "communicative facial displays," such as shrugging when it lacks information.

Key Insights: The "Overhearing" Agent

A central innovation is the concept of an agent that monitors human-to-human conversation without waiting for a direct command. By detecting "communication mismatches"—where participants misunderstand each other's intentions—the agent can intervene to clarify. To remain efficient, the agent uses "minimal utterances"—short phrases and facial expressions—to flag these misunderstandings quickly.

Results and Implementation

The prototype was implemented using SGI and Sony NEWS workstations. In qualitative tests, the agent successfully "overheard" two humans discussing dinner plans.

The prototype social agent system featuring a 3D animated face interacting with two human participants.

When the agent detected confusion through conversation flow or facial cues, it intervened with statements like, "I think that he is thinking of a pasta dish." This demonstrated the system's ability to maintain separate belief models and identify the exact point where human plans diverged.

Limitations and Future Work

The 1994 implementation faced significant hardware bottlenecks, requiring a trade-off between real-time reaction and semantic depth. Additionally, while the agent could track face positions, it could not yet automatically interpret human facial expressions. Future work aimed to incorporate "social stereotypes" and dynamic personal properties to better model complex human relationships.

Summary updated

BibTeX
@inproceedings{nagao1994social,
  author = {Katashi Nagao and Akikazu Takeuchi},
  title = {{Social Interaction: Multimodal Conversation with Social Agents}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1994},
}
Figure