RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation

Peng Chen, Xiaobao Wei, Yi Yang, Naiming Yao, Hui Chen, Feng Tian

Abstract

Talking head generation is increasingly important in virtual reality (VR), especially for social scenarios involving multi-turn conversation. Existing approaches face notable limitations: mesh-based 3D methods can model dual-person dialogue but lack realistic textures, while large-model-based 2D methods produce natural appearances but incur prohibitive computational costs. Recently, 3D Gaussian Splatting (3DGS) based methods achieve efficient and realistic rendering but remain speaker-only and ignore social relationships. We introduce RSATalker, the first framework that leverages 3DGS for realistic and socially-aware talking head generation with support for multi-turn conversation. Our method first drives mesh-based 3D facial motion from speech, then binds 3D Gaussians to mesh facets to render high-fidelity 2D avatar videos. To capture interpersonal dynamics, we propose a socially-aware module that encodes social relationships, including blood and non-blood as well as equal and unequal, into high-level embeddings through a learnable query mechanism. We design a three-stage training paradigm and construct the RSATalker dataset with speech-mesh-image triplets annotated with social relationships. Extensive experiments demonstrate that RSATalker achieves state-of-the-art performance in both realism and social awareness.

Introduction

Creating realistic digital avatars for Virtual Reality (VR) requires a delicate balance between visual quality and computational efficiency. Current methodologies typically fall into two categories: 3D mesh-based methods, which excel at modeling the dynamics of conversation but often suffer from an "uncanny valley" effect due to unrealistic textures, and 2D image-based methods, which produce photorealistic results but are too computationally intensive for real-time applications.

Furthermore, most models are "speaker-only," focusing on lip-syncing while ignoring the listener. In authentic human interaction, facial expressions and reactions are heavily modulated by the social relationship between participants. RSATalker bridges this gap by introducing a system that is realistic, efficient, and socially intelligent.

Methodology

The RSATalker architecture integrates three primary components to achieve its goals:

  • Speaker-Listener Motion Generator: This module processes audio and motion data from both participants simultaneously. Using a cross-attention mechanism, it ensures the avatar’s facial movements are responsive to the partner's cues, facilitating natural multi-turn dialogue.
  • Realistic Head Avatar Renderer: By leveraging 3D Gaussian Splatting (3DGS), the authors "bind" 3D Gaussians to the triangular facets of a standard FLAME facial mesh. As the mesh deforms based on speech, the Gaussians follow, maintaining structural integrity while providing high-fidelity rendering.
  • Socially-Aware Module (SAM): SAM categorizes relationships along two axes (Blood vs. Non-blood and Equal vs. Unequal). These relationships are encoded into high-level embeddings that influence both head motion and fine-grained facial details, such as the tension in a smile.
The Speaker-Listener Motion Generator processes audio and motion from both participants to ensure responsive facial movements during multi-turn conversation.

The RSATalker Dataset

To support this research, the team introduced the RSATalker dataset, a new benchmark for socially-conditioned talking head generation. It consists of speech–mesh–image triplets annotated with four typical social scenarios: Sibling (blood/equal), Mother/Son (blood/unequal), Couple (non-blood/equal), and Colleague (non-blood/unequal).

Key Results and Insights

RSATalker significantly outperforms state-of-the-art baselines like ER-NeRF and GaussianTalker. Quantitatively, it achieved a PSNR of 22.99 and an SSIM of 0.9377. Qualitatively, the model produces observable behavioral cues, such as role-appropriate head poses and responsive listener behaviors like nodding.

RSATalker achieves high-fidelity, photorealistic rendering by binding 3D Gaussians to a 3D facial mesh.

In a user study, RSATalker scored 84.5 in Social Relationship Accuracy (SRA), representing a 45.4% relative increase over the next best baseline. This demonstrates that social context acts as a vital "relational prior" that modulates facial dynamics effectively.

Conclusion and Future Work

While RSATalker represents a significant advancement, challenges remain regarding extreme head poses and long-term memory in extended dialogues. Future work will focus on improving visual fidelity during sharp rotations and assessing social nuances across a broader, more diverse population to minimize evaluation bias.

Summary updated

Abstract

Talking head generation is increasingly important in virtual reality (VR), especially for social scenarios involving multi-turn conversation. Existing approaches face notable limitations: mesh-based 3D methods can model dual-person dialogue but lack realistic textures, while large-model-based 2D methods produce natural appearances but incur prohibitive computational costs. Recently, 3D Gaussian Splatting (3DGS) based methods achieve efficient and realistic rendering but remain speaker-only and ignore social relationships. We introduce RSATalker, the first framework that leverages 3DGS for realistic and socially-aware talking head generation with support for multi-turn conversation. Our method first drives mesh-based 3D facial motion from speech, then binds 3D Gaussians to mesh facets to render high-fidelity 2D avatar videos. To capture interpersonal dynamics, we propose a socially-aware module that encodes social relationships, including blood and non-blood as well as equal and unequal, into high-level embeddings through a learnable query mechanism. We design a three-stage training paradigm and construct the RSATalker dataset with speech-mesh-image triplets annotated with social relationships. Extensive experiments demonstrate that RSATalker achieves state-of-the-art performance in both realism and social awareness. The code and dataset will be released.
BibTeX
@article{chen2026rsatalker,
  author = {Peng Chen and Xiaobao Wei and Yi Yang and Naiming Yao and Hui Chen and Feng Tian},
  title = {{RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2026},
  doi = {10.1109/tvcg.2026.3679919},
}
Figure