Visual speaker localization aided by acoustic models

Gerald Friedland, Chuohao Yeo, Hayley Hung

Abstract

The following paper presents a novel audio-visual approach for unsupervised speaker locationing. Using recordings from a single, low-resolution room overview camera and a single far-field microphone, a state-of-the art audio-only speaker localization system (traditionally called speaker diarization) is extended so that both acoustic and visual models are estimated as part of a joint unsupervised optimization problem. The speaker diarization system first automatically determines the number of speakers and estimates "who spoke when", then, in a second step, the visual models are used to infer the location of the speakers in the video. The experiments were performed on real-world meetings using 4.5 hours of the publicly available AMI meeting corpus. The proposed system is able to exploit audio-visual integration to not only improve the accuracy of a state-of-the-art (audio-only) speaker diarization, but also adds visual speaker locationing at little incremental engineering and computation costs.
BibTeX
@inproceedings{friedland2009visual,
  author = {Gerald Friedland and Chuohao Yeo and Hayley Hung},
  title = {{Visual speaker localization aided by acoustic models}},
  booktitle = {ACM Multimedia},
  year = {2009},
  doi = {10.1145/1631272.1631301},
}

Loading related content...

Figure