Abstract
Talking head generation is increasingly important in virtual reality (VR), especially for social scenarios involving multi-turn conversation. Existing approaches face notable limitations: mesh-based 3D methods can model dual-person dialogue but lack realistic textures, while large-model-based 2D methods produce natural appearances but incur prohibitive computational costs. Recently, 3D Gaussian Splatting (3DGS) based methods achieve efficient and realistic rendering but remain speaker-only and ignore social relationships. We introduce RSATalker, the first framework that leverages 3DGS for realistic and socially-aware talking head generation with support for multi-turn conversation. Our method first drives mesh-based 3D facial motion from speech, then binds 3D Gaussians to mesh facets to render high-fidelity 2D avatar videos. To capture interpersonal dynamics, we propose a socially-aware module that encodes social relationships, including blood and non-blood as well as equal and unequal, into high-level embeddings through a learnable query mechanism. We design a three-stage training paradigm and construct the RSATalker dataset with speech-mesh-image triplets annotated with social relationships. Extensive experiments demonstrate that RSATalker achieves state-of-the-art performance in both realism and social awareness.
Introduction
Creating realistic digital avatars for Virtual Reality (VR) requires a delicate balance between visual quality and computational efficiency. Current methodologies typically fall into two categories: 3D mesh-based methods, which excel at modeling the dynamics of conversation but often suffer from an "uncanny valley" effect due to unrealistic textures, and 2D image-based methods, which produce photorealistic results but are too computationally intensive for real-time applications.
Furthermore, most models are "speaker-only," focusing on lip-syncing while ignoring the listener. In authentic human interaction, facial expressions and reactions are heavily modulated by the social relationship between participants. RSATalker bridges this gap by introducing a system that is realistic, efficient, and socially intelligent.
Methodology
The RSATalker architecture integrates three primary components to achieve its goals:
- Speaker-Listener Motion Generator: This module processes audio and motion data from both participants simultaneously. Using a cross-attention mechanism, it ensures the avatar’s facial movements are responsive to the partner's cues, facilitating natural multi-turn dialogue.
- Realistic Head Avatar Renderer: By leveraging 3D Gaussian Splatting (3DGS), the authors "bind" 3D Gaussians to the triangular facets of a standard FLAME facial mesh. As the mesh deforms based on speech, the Gaussians follow, maintaining structural integrity while providing high-fidelity rendering.
- Socially-Aware Module (SAM): SAM categorizes relationships along two axes (Blood vs. Non-blood and Equal vs. Unequal). These relationships are encoded into high-level embeddings that influence both head motion and fine-grained facial details, such as the tension in a smile.
The RSATalker Dataset
To support this research, the team introduced the RSATalker dataset, a new benchmark for socially-conditioned talking head generation. It consists of speech–mesh–image triplets annotated with four typical social scenarios: Sibling (blood/equal), Mother/Son (blood/unequal), Couple (non-blood/equal), and Colleague (non-blood/unequal).
Key Results and Insights
RSATalker significantly outperforms state-of-the-art baselines like ER-NeRF and GaussianTalker. Quantitatively, it achieved a PSNR of 22.99 and an SSIM of 0.9377. Qualitatively, the model produces observable behavioral cues, such as role-appropriate head poses and responsive listener behaviors like nodding.
In a user study, RSATalker scored 84.5 in Social Relationship Accuracy (SRA), representing a 45.4% relative increase over the next best baseline. This demonstrates that social context acts as a vital "relational prior" that modulates facial dynamics effectively.
Conclusion and Future Work
While RSATalker represents a significant advancement, challenges remain regarding extreme head poses and long-term memory in extended dialogues. Future work will focus on improving visual fidelity during sharp rotations and assessing social nuances across a broader, more diverse population to minimize evaluation bias.
Summary updated