Hybrid Reasoning Network for Video-based Commonsense Captioning

Weijiang Yu, Jian Liang, Lei Ji, Lu Li, Yuejian Fang, Nong Xiao, Nan Duan

Abstract

Standard video captioning models typically focus on describing observable actions, often failing to capture the underlying "commonsense" elements such as intentions, effects, and attributes. We present HybridNet, a framework that integrates semantic-level joint learning with a memory-augmented attention mechanism to perform video-based commonsense captioning. By reasoning across interconnected dimensions simultaneously, HybridNet achieves state-of-the-art performance on the Video-to-Commonsense (V2C) benchmark.

Introduction

While modern computer vision models excel at identifying "what" is happening in a video (e.g., "a person is running"), they often lack the human-like ability to infer "why" (intentions) or "what happens next" (effects). This gap in commonsense reasoning limits the depth of automated video understanding. Previous attempts at commonsense captioning typically employed isolated networks for different tasks, ignoring the inherent correlations between intentions, effects, and attributes. For instance, an agent's intention to win a race directly informs the likely effect of their actions. HybridNet addresses this by creating a unified network that reasons across these dimensions in parallel.

Methodology

The Hybrid Reasoning Network (HybridNet) employs a dual-level reasoning strategy to bridge visual perception and linguistic commonsense.

Overview of the HybridNet framework illustrating the integration of commonsense semantics and memory-routed word prediction.

Semantic-level Reasoning

HybridNet utilizes a shared architecture to jointly train decoders for event captions, intentions, effects, and attributes. This multi-commonsense learning approach allows the model to exploit semantic correlations. For example, features from the "Intention" decoder can provide valuable context for the "Effect" decoder, leading to more logically consistent outputs.

Word-level Reasoning

To enhance text generation, the authors introduced a Memory-routed Multi-Head Attention (MMHA) module. This module incorporates:

  • Memory Module: A "history memory base" that records information from previous generation steps, ensuring temporal and contextual coherence in the predicted word sequence.
  • Triangle Convolution: A specialized layer within the attention mechanism designed to capture local patterns while preventing "look-ahead" bias during training.

Multimodal Encoding

The model processes video through three distinct streams: 3D Spatio-temporal CNNs for motion, 1D Sound CNNs for audio, and 2D Image CNNs for appearance. These features are fused into a single multimodal vector to provide a comprehensive representation of the scene.

Experimental Results

The model was evaluated on the Video-to-Commonsense (V2C) benchmark, which consists of 9,721 video scenes and over 121,000 captions across four tasks: standard captioning, intentions, effects, and attributes.

Quantitative Performance

HybridNet outperformed previous state-of-the-art (SOTA) models across multiple metrics, including CIDEr, BLEU, and METEOR. Notably, on the "Intention" completion task, HybridNet achieved a CIDEr score of 92.6, a significant improvement over the previous best of 62.0. For the joint generation of captions and intentions, the CIDEr score rose from 37.8 to 40.4.

Key Insights

  • Holistic Perception: Jointly learning multiple commonsense tasks improves individual task performance by leveraging shared semantic structures.
  • The Role of Audio: Ablation studies revealed that audio is essential for distinguishing between visually similar actions, such as "singing" versus "talking."
  • Contextual Logic: The memory module effectively mitigates common issues like "looping" or "forgetting" in long-form generation, maintaining better linguistic coherence.

Conclusion and Future Work

HybridNet demonstrates that unified, multi-level reasoning is superior to isolated task-specific models for video commonsense captioning. While the model is more parameter-efficient than some competitors (103.4M vs 159.1M parameters), the multi-decoder setup remains computationally intensive. Future work will focus on more sophisticated multimodal fusion techniques and expanding the scope of reasoning to include complex social dynamics beyond the current categories.

Summary updated

BibTeX
@inproceedings{yu2021hybrid,
  author = {Weijiang Yu and Jian Liang and Lei Ji and Lu Li and Yuejian Fang and Nong Xiao and Nan Duan},
  title = {{Hybrid Reasoning Network for Video-based Commonsense Captioning}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=7Q4R_RhdT7},
}
Figure