A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching

Pradipto Das, Chenliang Xu, Richard F. Doell, Jason J. CorsoCVPR 2013 · OpenReview

Abstract

The problem of describing images through natural language has gained importance in the computer vision community. Solutions to image description have either focused on a top-down approach of generating language through combinations of object detections and language models or bottom-up propagation of keyword tags from training images to test images through probabilistic or nearest neighbor techniques. In contrast, describing videos with natural language is a less studied problem. In this paper, we combine ideas from the bottom-up and top-down approaches to image description and propose a method for video description that captures the most relevant contents of a video in a natural language description. We propose a hybrid system consisting of a low level multimodal latent topic model for initial keyword annotation, a middle level of concept detectors and a high level module to produce final lingual descriptions. We compare the results of our system to human descriptions in both short and long forms on two datasets, and demonstrate that final system output has greater agreement with the human descriptions than any single level.

Introduction

While describing static images has seen significant progress, video description remains a "less studied" and much harder problem. Videos introduce temporal variations, complex articulations, and long-range dependencies that static images lack. Existing solutions generally fall into two camps: bottom-up methods that propagate keyword tags but often produce disconnected "word clouds," and top-down methods that use object detectors to fill templates but are limited by detector accuracy and narrow vocabularies. This work bridges the gap by creating a system that is both semantically rich and syntactically structured, robust enough to handle "in-the-wild" YouTube videos.

Proposed Framework

The system architecture is a three-tiered hybrid model designed to integrate bottom-up keyword generation with top-down structured language generation.

Overview of the hybrid video description system, illustrating the flow from raw video to final semantic verification.

Low Level: Multimodal Latent Topic Model

The bottom-up component adapts a multimodal latent topic model called MMLDA (MultiModal Latent Dirichlet Allocation). This model treats visual features (such as HOG3D and color histograms) and text as joint memberships over latent topics. It generates a "proposal distribution" of keywords—subjects, verbs, and objects—for a given video.

Graphical representation of the MMLDA model used for low-level keyword annotation.

Middle Level: Sparse Object Stitching (SOS)

To handle the difficulty of tracking objects in shaky, unpredictable video, the authors introduce Sparse Object Stitching (SOS). Instead of continuous tracking, detectors (Deformable Parts Models) are run on sparse frames and grouped into "concept shots" based on Jaccard distance. These detections are then mapped to a Tripartite Template Graph (Subjects $\to$ Tools $\to$ Objects) to generate candidate sentences.

High Level: Semantic Verification

The high-level module acts as the "glue," comparing the ranked list of keywords from the low level with the detected concepts from the middle level. By measuring "topic rank inversions," the system verifies if the proposed words match the visual evidence, filtering out noise and ranking the most plausible sentences.

Datasets and Evaluation

The system was evaluated on two distinct datasets:

  1. TRECVID MED12: A large-scale dataset for Multimedia Event Detection containing complex real-world events like "Renovating a home" or "Rock climbing."
  2. YouCook: A specialized dataset of 88 YouTube cooking videos featuring third-person viewpoints and dynamic camera changes across six different cooking styles.

Evaluation utilized ROUGE scores to measure overlapping n-grams between system-generated and human descriptions.

Results and Discussion

The integrated "High Level" system significantly outperformed individual components. On the TRECVID dataset, the full system achieved a recall of 65.84 in the "Rock Climbing" category, compared to 59.22 for the low-level keyword model alone. Qualitatively, the system successfully generated complex, descriptive sentences such as "A person is using power drill to renovate a house" or "A woman adds ingredients to a blender."

The semantic verification step proved crucial for robustness, successfully filtering out "noise phrases" that lacked strong visual support. While the system is currently limited by the scalability of the concept space and the precision of underlying computer vision primitives, it demonstrates a powerful approach to bridging the gap between visual features and natural language.

Conclusion

By combining the strengths of bottom-up keyword propagation and top-down template-based generation, this hybrid framework produces coherent and semantically accurate video descriptions. Future work will focus on scaling the concept library to cover a wider array of human activities and environments.

Summary updated

Abstract

The problem of describing images through natural language has gained importance in the computer vision community. Solutions to image description have either focused on a top-down approach of generating language through combinations of object detections and language models or bottom-up propagation of keyword tags from training images to test images through probabilistic or nearest neighbor techniques. In contrast, describing videos with natural language is a less studied problem. In this paper, we combine ideas from the bottom-up and top-down approaches to image description and propose a method for video description that captures the most relevant contents of a video in a natural language description. We propose a hybrid system consisting of a low level multimodal latent topic model for initial keyword annotation, a middle level of concept detectors and a high level module to produce final lingual descriptions. We compare the results of our system to human descriptions in both short and long forms on two datasets, and demonstrate that final system output has greater agreement with the human descriptions than any single level.
BibTeX
@inproceedings{das2013thousand,
  author = {Pradipto Das and Chenliang Xu and Richard F. Doell and Jason J. Corso},
  title = {{A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2013},
  url = {https://openreview.net/forum?id=SkZoWT-OWS},
}
Figure