MSSG: Multi-Scale Speaker Graph Network for Active Speaker Detection

Guanjun Li, Jiangyan Yi, Zhengqi Wen, Ruibo Fu, Yuwang Wang, Jianhua Tao

Abstract

The active speaker detection task is to determine whether a person is speaking or not across a series of video frames. Existing methods heavily rely on facial information within the annotated face bounding boxes for cross-modal learning with audio. This leads to a substantial decline in detection performance when facial cues are unclear, such as in cases of face occlusion or low-resolution facial appearances. In this paper, we extend the perception scale using only face bounding box annotations to model both facial and gestural cues, addressing the over-reliance on facial cues in active speaker detection. We propose a novel graph neural network that models inter-speaker interactions and integrates various cues from individual speakers. The final detection results are obtained through a binary graph node classification task. Our method achieves state-of-the-art performance on the AVA-ActiveSpeaker dataset (mAP: 95.6%) and the ASW dataset (mAP: 99.4%), with a model size only 21% that of the second-best method. Additionally, when facial cues are of poor quality, our method demonstrates a significant performance advantage over existing approaches. The code and model weights will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sdqdlgj/MSSG</uri>.
BibTeX
@article{li2026mssg,
  author = {Guanjun Li and Jiangyan Yi and Zhengqi Wen and Ruibo Fu and Yuwang Wang and Jianhua Tao},
  title = {{MSSG: Multi-Scale Speaker Graph Network for Active Speaker Detection}},
  journal = {IEEE Trans. Multimedia},
  year = {2026},
  doi = {10.1109/tmm.2025.3608949},
}

Loading related content...

Figure