EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold

Junjie Chen, Fei Wang, Kun Li, Yiqi Nie, Xun Yang, Yanbin Hao, Linfeng Zhang, Meng WangarXiv 2026 · arXiv

TL;DR

EvolvingAvatar is a novel 3D head generation framework that learns to adapt its behavior during a conversation. By using self-supervised test-time training on dyadic audiovisual context, the model mimics the human "chameleon effect," improving its coordination and naturalness as an interaction unfolds without requiring ground-truth motion labels at inference time.

What Problem Does This Paper Address?

Generating realistic 3D avatars for interactive applications (like virtual tutors or therapeutic agents) requires more than just lip-syncing; it requires a coordinated dance of speaking and listening behaviors that respond to a partner's unique style.

Current state-of-the-art generators are typically "static" at deployment—their parameters are fixed after training. While they can take user audio or video as input context, they cannot learn from the specific patterns of a new conversation, such as a user's unique nodding habits or turn-taking rhythms. This leaves a wealth of "dyadic" (two-person) interaction data unused. The challenge is: how do you update a model during a live interaction when you don't have the "correct" motion labels to train on?

Methods

The authors introduce EvolvingAvatar, which bridges the gap between fixed models and adaptive human behavior through three key innovations:

  1. Test-Time Training (TTT) with Dyadic Context Prediction (DCP): The model uses a self-supervised objective during inference. It attempts to predict future user video and dyadic audio features. The prediction error serves as a learning signal to update "fast weights" in the generator, allowing it to adapt to the specific conversation's dynamics.
  2. Persistent vs. Transient Adaptation: The architecture separates adaptation into two streams. Persistent fast weights accumulate information throughout the conversation to capture long-term stylistic regularities. Conversely, transient jaw adaptation responds to immediate, short-term speech articulation to ensure precise lip-syncing.
  3. Region-Structured Causal FLAME Codec: To coordinate complex motions, the model uses a specialized codec that separates the face into regions (expression, neck, jaw). This allows for more granular control and ensures that adaptation in one area (like a nodding neck) doesn't negatively interfere with another (like speech articulation).
Figure 2: EVOLVINGAVATAR learns from the ongoing conversation at deployment without motion labels. Dyadic context prediction drives two complementary forms of adaptation: persistent fast weights capture conversational regularities, while transient jaw adaptation tracks current articulation. Regional latent routing combines their predictions through a shared causal decoder.

Datasets

The researchers also addressed the lack of large-scale dyadic data by introducing INTERHEAD-BENCH, a massive 455.95-hour benchmark.

  • Construction: They developed DIALOG3D-FACTORY, a pipeline that unifies single-view (mixed audio) and dual-view (separate audio/video) recordings into a standardized format with FLAME motion parameters, transcripts, and interaction states.
  • Scale: It includes 4,365 source participant IDs and covers a wide range of interaction durations and turn counts.
  • Splits: The benchmark features In-Distribution (ID), Out-of-Distribution (OOD), and "OOD-Hard" (RealTalk) splits to test how well models generalize to entirely new recording environments and participants.
Figure 3: DIALOG3D-FACTORY processes dual-view recordings with separate audio and single-view recordings with mixed audio. A shared workflow extracts face video, participant speech, FLAME motion, transcripts, speaking and interaction states, and turn statistics on a common timeline.

Key Insights and Ideas

  • Interaction as a Learning Signal: The core "aha" moment is treating the ongoing conversation not just as an input, but as a continuous stream of unlabeled training data.
  • The Chameleon Effect: By updating parameters online, the model can naturally mirror the user's nonverbal behavior, a key component of human rapport.
  • Activity-Gated Adaptation: The model predicts "speaking" vs. "listening" states and uses this to gate how the persistent adaptation affects motion, preventing listening-style adaptations from distorting speech.
  • Evolution Over Time: Unlike static models that maintain a constant error rate, EvolvingAvatar actually improves its performance as the conversation progresses.

Key Results

EvolvingAvatar consistently outperformed strong baselines (like DualTalk and UniLS) across multiple metrics:

  • Mismatch Reduction: On the challenging OOD-Hard split, the model reduced the mismatch with recorded user-avatar expression statistics by up to 11.1% as the conversation unfolded.
  • Quantitative Gains: It achieved the lowest Frechet Distance (FD) for expression and neck motion across nearly all test splits, indicating more realistic global motion statistics.
  • Human Preference: In blind A/B testing, human raters preferred EvolvingAvatar over the strongest baseline (DualTalk) 86% of the time on the OOD-Hard split, citing better naturalness and responsiveness.
Figure 4: INTERHEAD-BENCH composition by split, including scale and the distributions of inter- action duration and conversational turns.

Limitations and Future Work

  • Closed-Loop Dynamics: The current study uses recorded conversations. In a live setting, the user would also react to the avatar's generated motion, creating a feedback loop that isn't fully captured here.
  • Memory Management: Future work is needed to determine which expressive habits should be retained long-term versus which are temporary emotional responses that should be "forgotten" to avoid catastrophic interference.
  • Clinical Validation: While the model shows promise for psychological support, specific studies are needed to see if this adaptation actually leads to better perceived empathy and support in therapeutic contexts.

Summary updated

Abstract

Interactive 3D head generation requires coordinated speaking and listening motion that responds to an evolving conversation. Existing generators use incoming observations as context but keep their parameters fixed, leaving conversational patterns unused as a learning signal. We introduce EvolvingAvatar, a causal generator that uses test-time training to adapt to user face video and dyadic audio during interaction. Its dyadic context prediction objective provides a self-supervised learning signal from audiovisual context without target motion labels at test time. Persistent fast weights accumulate these updates within each conversation to guide motion generation, while transient jaw adaptation responds to current audiovisual context. Predicted speech activity controls how persistent adaptation guides motion. We also introduce InterHead-Bench, a unified 455.95-hour benchmark built from single-view and dual-view conversation videos. Experiments show improved conversational motion statistics over strong baselines. On the hardest out-of-distribution split, generation improves as conversations unfold, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.
BibTeX
@misc{chen2026evolvingavatar,
  author = {Junjie Chen and Fei Wang and Kun Li and Yiqi Nie and Xun Yang and Yanbin Hao and Linfeng Zhang and Meng Wang},
  title = {{EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold}},
  year = {2026},
  eprint = {2609.35616},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.35616},
}
Figure