Supplementary Material: AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection

Joseph Roth, Zhonghua Xi, Caroline Pantofaru, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew C. Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia SchmidWorkshops @ ICCV 2019 · OpenReview

Abstract

Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual active speaker dataset has limited algorithm evaluation in terms of data diversity, environments, and accuracy. In this paper, we present the AVA Active Speaker detection dataset (AVA-ActiveSpeaker) which has been publicly released to facilitate algorithm development and comparison. It contains temporally labeled face tracks in video, where each face instance is labeled as speaking or not, and whether the speech is audible. This dataset contains about 3.65 million human labeled frames spanning 38.5 hours. We also introduce a state-of-the-art approach for real-time active speaker detection and compare several variants. This evaluation clearly demonstrates a significant gain due to audio-visual modeling and temporal integration over multiple frames.
BibTeX
@inproceedings{roth2019supplementary,
  author = {Joseph Roth and Zhonghua Xi and Caroline Pantofaru and Sourish Chaudhuri and Ondrej Klejch and Radhika Marvin and Andrew C. Gallagher and Liat Kaver and Sharadh Ramaswamy and Arkadiusz Stopczynski and Cordelia Schmid},
  title = {{Supplementary Material: AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  url = {https://openreview.net/forum?id=8BrmEnJN4Gu},
}
Figure