Introduction
Video captioning—the task of automatically describing video clips in natural language—sits at the complex intersection of Computer Vision and Natural Language Processing. While recent research has heavily focused on developing more sophisticated language decoders, this paper argues that the primary bottleneck often lies in the visual encoder.
Standard practices, such as mean-pooling CNN features across all frames, effectively flatten the video and wash away critical temporal dynamics and fine-grained object information. This paper introduces a novel approach that prioritizes richer visual features to achieve superior captioning performance even with a relatively simple language model.
Methods: Enriched Visual Encoding (GRU-EVE)
The authors propose a visual encoding technique that produces a "visual code" enriched with both spatio-temporal dynamics and high-level semantics. The architecture, dubbed GRU-EVE, is built on three main pillars:
- Hierarchical Short Fourier Transform (hft): Instead of simple averaging, a recursive, tree-like Fourier Transform captures how visual features change over time at multiple scales (the whole video, halves, quarters, etc.). This preserves the temporal "rhythm" and evolution of the scene for both 2D and 3D CNN features.
- Object Semantics and Spatial Evolution: By employing an object detector (YOLO), the model tracks specific objects along with their plurality, frequency, and velocity. This allows the system to distinguish between subtle differences, such as "a person walking" versus "people running."
- Action Semantics: Explicit action labels are extracted from the final output layer of a 3D CNN and intersected with the language model's vocabulary to ensure the visual features are aligned with the words the model can generate.
These enriched features are compressed through a fully connected layer and fed into a two-layer Gated Recurrent Unit (GRU) for sequence modeling.
Experimental Setup and Results
The model was evaluated on two benchmark datasets: MSVD (1,970 YouTube clips) and MSR-VTT (10,000 clips across 20 categories). GRU-EVE established new state-of-the-art performance on both benchmarks at the time of publication. On MSVD, it achieved a 35.0 METEOR score and a 71.5 ROUGE-L score. On MSR-VTT, it reached a 28.4 METEOR score, outperforming significantly more complex models.
Qualitative analysis shows that the model is particularly adept at identifying the correct number of subjects and using precise action verbs. For instance, it can correctly identify "a group of elephants" where simpler models might only see "an elephant."
Conclusion
The success of GRU-EVE demonstrates that the secret to better captions often lies in the quality of the visual input rather than the complexity of the decoder. By capturing fine temporal details and explicit semantics, the model gains a better "common sense" understanding of the scene. Future work may involve combining these enriched encodings with more advanced language architectures like Transformers or optimizing the computational efficiency of the hierarchical transforms.
Summary updated