Abstract
As digital news archives grow, the ability to navigate video content at a semantic level—rather than just by visual cuts—is essential for effective retrieval. This paper introduces a multi-modal, two-level framework designed to segment news broadcasts into individual stories. By combining Decision Trees for shot-level classification with Hidden Markov Models (HMM) for sequential modeling, the system leverages the structured "grammar" of news programs. Experimental results on Singaporean news broadcasts demonstrate a shot tagging accuracy of 95.1% and a story boundary detection accuracy of 89%.
Introduction
While "shot detection" (identifying camera cuts) is a mature technology, a single news story typically spans multiple shots. Navigating raw video remains difficult because these syntactic units do not correspond directly to semantic units. The challenge lies in bridging this gap. Many news segments, such as live reports or studio interviews, may share visual characteristics with non-news content. To address this, we propose a framework that incorporates visual, audio, and temporal features alongside the inherent structural patterns of news broadcasting.
System Methodology
The proposed framework operates on two distinct levels, analogous to how natural language processing treats words and sentences.
Level 1: Shot Classification
The system first decomposes the video into shots and extracts a multi-modal feature set:
- Visual: Color histograms and face detection (tracking the number and size of faces).
- Temporal: Shot duration and motion intensity.
- Audio: Classification into speech, music, noise, or silence.
- Object-based: Detection of videotext (captions) and its spatial distribution.
A Decision Tree (DT) is employed to classify each shot into one of 13 categories, including Anchor, People-Meeting, Sports, and Weather.
Level 2: Story Segmentation
Shot-by-shot classification is susceptible to local noise. To improve robustness, a Hidden Markov Model (HMM) treats the shot sequence as a time series. The HMM utilizes the "grammar" of news—such as the tendency for stories to be bookended by anchor shots—to correct misclassifications and identify precise story boundaries.
Key Insights: The Grammar of News
A central finding of this research is that news broadcasts are highly structured. The system leverages this predictability to resolve ambiguities. For instance, while a Decision Tree might mislabel a shot based on visual cues alone, the HMM acts as a statistical "sanity check," recognizing that certain transitions (e.g., from a weather report to a live news report) are more or less likely.
Furthermore, feature analysis revealed that face detection is the most critical visual cue for classification. Using face data alone provided nearly 60% accuracy, which jumped to 78% when combined with audio data.
Experimental Results
The system was evaluated using two days of half-hour news broadcasts from MediaCorp of Singapore.
- Shot Classification: The Decision Tree achieved 95.1% accuracy. Most errors involved confusion between Anchor and Speech shots due to visual similarities.
- Story Segmentation: The HMM-based approach achieved an 89% accuracy in boundary detection. Notably, the HMM corrected several tagging errors made by the initial Decision Tree phase, proving the efficacy of sequential modeling.
Conclusion and Future Work
This two-level approach successfully bridges the gap between low-level video features and high-level semantic stories. Future research will focus on integrating Automatic Speech Recognition (ASR) to utilize textual keywords for further refinement. Additionally, the authors aim to scale the system to diverse broadcast styles and automate the remaining manual indexing steps to create a fully end-to-end personalized news delivery pipeline.
Summary updated