RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation

Peng Chen, Xiaobao Wei, Yi Yang, Naiming Yao, Hui Chen, Feng Tian

TL;DR

RSATalker is a novel framework that generates high-fidelity, photorealistic talking heads capable of natural multi-turn conversations. By combining the efficiency of 3D Gaussian Splatting with a "socially-aware" module, the system produces avatars that adjust their facial expressions and listening behaviors based on their relationship with the interlocutor (e.g., sibling vs. boss).

What Problem Does This Paper Address?

Creating realistic digital avatars for Virtual Reality (VR) is a balancing act between visual quality and computational efficiency. Current methods generally fall into two camps:

  1. 3D Mesh-based methods: These are great at modeling the "give-and-take" of a conversation, but they often look like plastic or video game characters, falling straight into the "uncanny valley."
  2. 2D Image-based methods (like Diffusion): These produce stunning, photorealistic results but are far too slow and computationally expensive for real-time VR applications.

Furthermore, most existing models are "speaker-only." They focus on syncing lips to audio but ignore the listener. In a real conversation, how you react while listening—and how you speak—is heavily influenced by your social relationship with the other person. You don't use the same micro-expressions with your manager as you do with your spouse. RSATalker aims to bridge this gap by creating a system that is realistic, efficient, and socially intelligent.

Methods

The RSATalker architecture consists of three primary components:

  • Speaker-Listener Motion Generator: Unlike standard models that only look at the current speaker's audio, this module processes audio and motion data from both participants. It uses a cross-attention mechanism to ensure the avatar’s facial movements are responsive to the partner's cues.
  • Realistic Head Avatar Renderer: The authors leverage 3D Gaussian Splatting (3DGS). They "bind" 3D Gaussians to the triangular facets of a standard 3D facial mesh (FLAME). As the underlying mesh moves and deforms based on the speech, the Gaussians follow, allowing for high-fidelity rendering that maintains structural integrity.
  • Socially-Aware Module (SAM): This is the core innovation. SAM categorizes relationships along two axes: Blood vs. Non-blood and Equal vs. Unequal. These relationships are encoded into high-level embeddings. These embeddings then influence both the motion (how the head moves) and the rendering (applying "Gaussian offsets" to fine-tune facial details like the tension in a smile or the depth of a frown).

The team uses a three-stage training paradigm: first training the motion generator on large-scale data, then pre-training the speaker-specific avatar renderer, and finally performing end-to-end joint training with the social module.

Datasets

The researchers introduced the RSATalker dataset, a new benchmark for socially-conditioned talking head generation.

  • It consists of speech–mesh–image triplets annotated with social relationships.
  • It was constructed by merging data from RealTalk (which provides 2D video/audio) and DualTalk (which provides 3D mesh/audio).
  • The dataset covers four typical scenarios: Sibling (blood/equal), Mother/Son (blood/unequal), Couple (non-blood/equal), and Colleague (non-blood/unequal).

Key Insights and Ideas

  • Social Context as a Prior: The authors demonstrate that social relationship isn't just a "label"—it’s a vital "relational prior" that modulates facial dynamics. Conditioning on these relationships helps the model produce observable, interpretable behavioral cues like role-appropriate head poses.
  • Bridging 3DGS and Meshes: By rigging 3D Gaussians to a 3D mesh, the authors solve the "drifting" problem often seen in pure 3DGS methods, where facial landmarks can slide unnaturally across the face during intense expressions.
  • The "Listener" Matters: A major "aha moment" is the qualitative difference in listener behavior. While baselines often look passive or "dead" when not speaking, RSATalker generates responsive nods and gaze shifts that make the interaction feel authentic.

Key Results

RSATalker significantly outperforms state-of-the-art baselines like ER-NeRF, SyncTalk, and GaussianTalker in both quantitative and qualitative metrics:

  • Image Quality: Achieved a PSNR of 22.99 and an SSIM of 0.9377, surpassing GaussianTalker (21.32 / 0.9164).
  • Social Accuracy: In a user study, RSATalker scored 84.5 in Social Relationship Accuracy (SRA), a massive 45.4% relative increase over the next best baseline (58.1).
  • Efficiency: By using 3DGS, the model maintains high rendering speeds suitable for VR, unlike heavy diffusion-based models.

Limitations and Future Work

While a significant step forward, the authors note a few hurdles:

  • Extreme Poses: Very sharp head rotations or heavy occlusions (like a hand covering the face) can still degrade visual fidelity.
  • Long-term Memory: In very long, multi-turn dialogues, the model occasionally "forgets" or attenuates social cues from earlier in the conversation.
  • Evaluation Bias: The current user study relied on domain experts (CV/HCI researchers). Future work should involve a broader, more diverse population to assess how "lay" viewers perceive these social nuances.

Summary updated

Abstract

Talking head generation is increasingly important in virtual reality (VR), especially for social scenarios involving multi-turn conversation. Existing approaches face notable limitations: mesh-based 3D methods can model dual-person dialogue but lack realistic textures, while large-model-based 2D methods produce natural appearances but incur prohibitive computational costs. Recently, 3D Gaussian Splatting (3DGS) based methods achieve efficient and realistic rendering but remain speaker-only and ignore social relationships. We introduce RSATalker, the first framework that leverages 3DGS for realistic and socially-aware talking head generation with support for multi-turn conversation. Our method first drives mesh-based 3D facial motion from speech, then binds 3D Gaussians to mesh facets to render high-fidelity 2D avatar videos. To capture interpersonal dynamics, we propose a socially-aware module that encodes social relationships, including blood and non-blood as well as equal and unequal, into high-level embeddings through a learnable query mechanism. We design a three-stage training paradigm and construct the RSATalker dataset with speech-mesh-image triplets annotated with social relationships. Extensive experiments demonstrate that RSATalker achieves state-of-the-art performance in both realism and social awareness. The code and dataset will be released.
BibTeX
@article{chen2026rsatalker,
  author = {Peng Chen and Xiaobao Wei and Yi Yang and Naiming Yao and Hui Chen and Feng Tian},
  title = {{RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2026},
  doi = {10.1109/tvcg.2026.3679919},
}
Figure