RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation
Lists
Only you see this
Abstract
Talking head generation is increasingly important in virtual reality (VR), especially for social scenarios involving multi-turn conversation. Existing approaches face notable limitations: mesh-based 3D methods can model dual-person dialogue but lack realistic textures, while large-model-based 2D methods produce natural appearances but incur prohibitive computational costs. Recently, 3D Gaussian Splatting (3DGS) based methods achieve efficient and realistic rendering but remain speaker-only and ignore social relationships. We introduce RSATalker, the first framework that leverages 3DGS for realistic and socially-aware talking head generation with support for multi-turn conversation. Our method first drives mesh-based 3D facial motion from speech, then binds 3D Gaussians to mesh facets to render high-fidelity 2D avatar videos. To capture interpersonal dynamics, we propose a socially-aware module that encodes social relationships, including blood and non-blood as well as equal and unequal, into high-level embeddings through a learnable query mechanism. We design a three-stage training paradigm and construct the RSATalker dataset with speech-mesh-image triplets annotated with social relationships. Extensive experiments demonstrate that RSATalker achieves state-of-the-art performance in both realism and social awareness. The code and dataset will be released.
BibTeX
@article{chen2026rsatalker,
author = {Peng Chen and Xiaobao Wei and Yi Yang and Naiming Yao and Hui Chen and Feng Tian},
title = {{RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation}},
journal = {IEEE Trans. Vis. Comput. Graph.},
year = {2026},
doi = {10.1109/tvcg.2026.3679919},
}