An Efficient Framework for Dense Video Captioning

Maitreya Suin, A. N. RajagopalanAAAI 2020 · DOI

Abstract

Dense video captioning (DVC) involves identifying and describing multiple events within long, untrimmed videos. Traditional approaches are computationally expensive, often processing every frame or using rigid uniform sampling. We propose a reinforcement learning-based framework that teaches an agent to "skim" videos by dynamically selecting only the most informative frames. Our approach reduces computational overhead by over 90% while outperforming standard uniform sampling methods on the ActivityNet Captions dataset.

Introduction

Dense video captioning (DVC) is a challenging task that requires both temporal event localization and natural language generation. Unlike standard video captioning, DVC must handle long-form content where events may overlap or occur sequentially over several minutes. The primary bottleneck in current DVC systems is the "brute-force" processing of video frames through heavy Convolutional Neural Networks (CNNs). Even fixed-interval sampling fails to account for the varying information density in real-world videos—some segments are static and redundant, while others contain rapid, critical actions. We address this by introducing a content-aware efficiency model that treats frame selection as a sequential decision-making process.

Methods

The core of our framework is a reinforcement learning (RL) agent designed to navigate through a video by selecting a sparse subset of frames.

The Agent and State Encoder

The system utilizes an Actor-Critic architecture. The Actor (Frame-Selection Network) determines the next jump interval (ranging from -4s to +32s), while the Critic (Guide Network) evaluates the quality of these transitions. To maintain temporal awareness, a State-Encoder (SE)—implemented as a lightweight LSTM—tracks the visual history and previous actions. This prevents the agent from missing segments or entering infinite loops.

The proposed reinforcement learning-based architecture for efficient dense video captioning.

Reward Function and Training

The agent is trained using a multi-objective reward function designed to balance accuracy and efficiency:

  • Language Score: Based on METEOR scores of generated captions.
  • Event Proposal Score: Measures the accuracy of identified event boundaries.
  • Global Representation Score: Ensures the selected frames accurately represent the full video context.
  • Efficiency Penalty: A small negative reward for every frame processed, encouraging the agent to skip redundant content.

To further enhance performance, we employ Knowledge Distillation, where a "teacher" model (trained on all frames) guides the "student" agent to produce similar captioning probability distributions despite seeing significantly less data.

Key Insights and Results

Our experiments on the ActivityNet Captions dataset reveal several critical advantages of dynamic skimming over static sampling.

Dynamic Skimming vs. Static Sampling

Videos exhibit non-linear information density. Our agent learns to accelerate through simple or static scenes and decelerate to capture nuances in complex sequences, such as a person interacting with pets or performing gymnastics.

Qualitative example showing the agent's dynamic frame selection (picked frames) relative to ground truth (GT) and predicted events.

Performance and Efficiency

  • Computational Reduction: The framework achieves a 90% reduction in GFLOPs compared to full-video processing.
  • Superior Accuracy: When compared to models using an equivalent number of uniformly sampled frames, our RL-based selection achieves higher METEOR and Average Recall scores. This confirms that the quality of selected frames is more impactful than the quantity.
  • Competitive Quality: Despite relying solely on appearance features (ResNet-200) and omitting computationally expensive motion features (optical flow), the model remains highly competitive with state-of-the-art DVC systems.

Conclusion

We have demonstrated that reinforcement learning can significantly improve the efficiency of dense video captioning by enabling content-aware frame selection. By learning to "skim" redundant segments, our model maintains high captioning quality while drastically reducing computational costs. Future work will explore integrating lightweight motion features and improving the generalizability of the skimming policy across diverse domains such as medical or surveillance video.

Summary updated

BibTeX
@inproceedings{suin2020efficient,
  author = {Maitreya Suin and A. N. Rajagopalan},
  title = {{An Efficient Framework for Dense Video Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6881},
}
Figure