Using Closed Captions as Supervision for Video Activity Recognition

Sonal Gupta, Raymond J. MooneyAAAI 2010 · DOI

Abstract

This paper presents a novel framework for training video activity recognition models using weak supervision from closed captions, bypassing the need for labor-intensive manual labeling. The approach utilizes a dual-stream architecture: a visual stream extracting spatio-temporal features and a linguistic stream employing a specialized classifier to filter out irrelevant commentary. Evaluated on real-world soccer broadcasts, the combined system significantly improves event retrieval precision, demonstrating that noisy, automatically generated labels can effectively train robust recognition models.

Introduction

Recognizing human actions such as "kicking" or "saving" in sports broadcasts is a complex task due to environmental clutter, dynamic camera movements, and varying scales. Traditional supervised learning requires thousands of manually annotated clips, which is prohibitively expensive. Closed captions offer a potential alternative; however, they are inherently "noisy." Commentators often use action-related keywords in non-literal contexts or with significant temporal lag. This research addresses the challenge of leveraging this "free" but messy data to build reliable visual recognizers.

Methodology

The proposed system follows a dual-stream approach to process both visual and linguistic information.

1. Automatic Clip Extraction

The system identifies keywords (e.g., kick, save, throw, touch) in captions and extracts 8-second video segments. A heuristic adjustment is applied to synchronize the visual action with the typically delayed caption.

2. Dual-Stream Architecture

The overall workflow is divided into training and testing phases, integrating both visual and linguistic classifiers to refine event detection.

System architecture for training and testing using caption-based supervision.
  • Visual Stream: The system extracts Space-Time Interest Points (STIPs) from the video clips. These are represented as a "bag of visual words" using Histograms of Oriented Gradients (HoG) and Optical Flow (HoF). To handle the high noise in automatic labels, the DECORATE ensemble meta-learner is used for classification.
Space-Time Interest Points (STIPs) extracted for 'kick' and 'save' actions.
  • Linguistic Stream: To distinguish between live action descriptions and extraneous commentary (e.g., "save their season"), a String Subsequence Kernel (SSK) classifier is trained. This model analyzes word sequences and Part-of-Speech (POS) tags to identify relevant captions.

3. Reranking

During testing, the system ranks retrieved clips by fusing the probability scores from both the video and caption classifiers to produce a final ranked list of video clips.

Results and Discussion

The system was evaluated on a dataset of 23 soccer games. The combined model achieved a Mean Average Precision (MAP) of 72.11, a significant improvement over the keyword-based baseline of 65.68.

Key findings include:

  • Noise as a Regularizer: Models trained on "noisy" automatic labels sometimes outperformed those trained on "clean" human labels, suggesting that noise helps ensemble models like DECORATE avoid overfitting.
  • Linguistic Context: The SSK classifier was crucial, achieving 80% accuracy in identifying live-action commentary, far surpassing standard bag-of-words approaches (69%).
  • Cross-Domain Generalization: A caption classifier trained on soccer data generalized well to baseball broadcasts, indicating that the structure of play-by-play commentary is consistent across different sports.

Conclusion

This work demonstrates the viability of using closed captions for weak supervision in video activity recognition. While the system is currently limited by its reliance on commentator mentions (the "recall" problem), future iterations could involve scanning entire video streams using the trained recognizers. Further improvements may include temporal modeling via Hidden Markov Models (HMMs) and enhanced visual preprocessing to better isolate players in cluttered scenes.

Summary updated

BibTeX
@inproceedings{gupta2010using,
  author = {Sonal Gupta and Raymond J. Mooney},
  title = {{Using Closed Captions as Supervision for Video Activity Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2010},
  doi = {10.1609/aaai.v24i1.7738},
}
Figure