TL;DR
This paper introduces a novel unsupervised approach to identify speakers in TV broadcasts by fusing audio-based speaker diarization with text extracted from on-screen overlays (like lower thirds). Their best method achieves a 70.2% F-measure, outperforming a supervised baseline, demonstrating the power of multimodal fusion for this challenging task.
What Problem Does This Paper Address?
Identifying who is speaking in video content is crucial for searching and browsing large archives. Traditional speaker identification methods require extensive manual annotation to train speaker models, which is time-consuming and expensive, especially for new video sources. This paper tackles the problem of building speaker identification systems without relying on such manual labels, aiming for an unsupervised approach that leverages readily available visual cues.
Methods
The core of the proposed system lies in combining two independent "monomodal" components:
- Acoustic Speaker Diarization: This component segments the audio stream into segments and assigns anonymous speaker labels to each segment. The authors use a multi-stage system that first segments the audio and then clusters these segments using BIC criterion and cross-likelihood ratio for improved accuracy.
- Overlaid Name Detection (Video OCR): This component uses a video OCR system to detect and recognize text appearing on screen, specifically focusing on "lower thirds" that often introduce people. They employ a three-step process: coarse text detection, refinement of text box coordinates, and temporal tracking. A key observation is that when a person's name appears on screen, it's highly likely to be associated with the speaker at that moment.
These two components are then fused using three different "name propagation" strategies to map the detected names to the speaker clusters identified by the diarization system. The goal is to find a mapping function that assigns a name (or "unknown") to each speech turn.
- Method 1 (One-to-One Speaker Tagging): Assumes perfect speaker diarization and finds a one-to-one mapping between speaker clusters and detected names to maximize co-occurrence duration.
- Method 2 (Direct Speech Turn Tagging): Leverages the high precision of directly associating a name on screen with the current speaker. Speech turns with a single, clear name overlay are tagged first. Then, Method 1 is applied to the remaining unnamed turns.
- Method 3 (One-to-Many Speaker Tagging): Addresses potential over-segmentation in speaker diarization by allowing a single name to be propagated to multiple speaker clusters. It uses a TF-IDF-like weighting scheme to find the best name assignment for remaining unnamed turns.
Datasets
The experiments were conducted on the REPERE evaluation campaign dry-run corpus. This corpus consists of approximately 3 hours of annotated videos from seven different TV shows (news, talk shows) broadcast on two French TV channels. The data was used in both "full condition" (allowing processing of the entire video) and "standard condition" (limited to annotated sections). A development set was used for training speaker models for the supervised baseline, and the evaluation was performed on a test set.
Key Insights and Ideas
- Multimodal Fusion is Powerful: Combining audio (speaker diarization) and visual (OCR of overlaid text) information significantly boosts speaker identification performance, especially in an unsupervised setting.
- Overlaid Text is a Reliable Source: On-screen text, particularly lower thirds, provides a high-confidence source of speaker names, even when automatic speech recognition might struggle with proper nouns.
- Handling Diarization Imperfections: Method 3's ability to propagate names to multiple clusters is crucial for dealing with the inherent errors and over-segmentation common in speaker diarization systems.
- Anchor Speakers are Tricky: Identifying "anchor" speakers (e.g., main news anchors) is more challenging because their names are less frequently displayed on screen compared to guests.
Key Results
- The best unsupervised system (Method 3) achieved an F-measure of 70.2% when considering all speakers and 81.7% when excluding anchor speakers.
- This unsupervised performance outperformed a supervised mono-modal speaker identification baseline (57.5% F-measure for all speakers, 45.7% without anchors).
- A combination of the unsupervised approach with the supervised baseline yielded the best overall results, reaching 77.5% F-measure for all speakers.
- The "full condition" (using the entire video) generally led to better results than the "standard condition" (using only annotated sections), particularly for anchor speakers.
Limitations and Future Work
- Anchor Speaker Identification: The paper acknowledges that identifying anchor speakers remains a challenge due to their names appearing less frequently on screen.
- Diarization Errors: While Method 3 is more robust, errors in speaker diarization still impact overall performance.
- Future Work: The authors suggest exploring early fusion methods, using cross-show information, unsupervised training of collection of speaker models, and applying these techniques to face recognition.
Summary updated