Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Hassan Akbari, Hamid Palangi, Jianwei Yang, Sudha Rao, Asli Celikyilmaz, Roland Fernandez, Paul Smolensky, Jianfeng Gao, Shih-Fu ChangarXiv 2020 · arXiv

Abstract

Video captioning requires models to move beyond simple object recognition to understand complex temporal interactions. We present the R3-Transformer (Relative Role Representations), a neuro-symbolic architecture that incorporates inductive biases to explicitly model the roles entities play within a scene. By utilizing Vector Quantization (VQ) to learn a discrete dictionary of "relative roles," the model disentangles specific actions from general context. Our approach achieves state-of-the-art performance on YouCook2 and ActivityNet Captions, producing more accurate, action-oriented descriptions that are significantly preferred by human evaluators.

Introduction

Current video captioning models often struggle to distinguish between primary actions and background context, treating all visual and textual tokens as uniform data points. This lack of structural inductive bias makes it difficult for models to capture the fluid, temporal interactions essential for high-quality captions. We propose bridging the gap between connectionist neural networks and symbolic logic by explicitly modeling these relationships using Tensor Product Representations (TPRs) and discrete role induction.

Methodology: The R3-Transformer

The R3-Transformer architecture bifurcates the attention mechanism into two distinct modules to better represent scene dynamics:

  1. Special Relativity: This module maps input tokens into a "relativity space" and employs Vector Quantization (VQ) to assign tokens to a learnable dictionary of discrete "roles." This forces the model to categorize interactions into a finite set of concepts, effectively performing neuro-symbolic "binding."
  2. General Relativity: A standard multi-head attention module that captures broader contextual information without discrete constraints.
Figure 1: The R3-Transformer architecture, featuring a Video Encoder and Text Decoder with integrated Role Induction modules.

The outputs of these modules are combined via a Hadamard product to form the final Relative Role Representation (R3). The system is trained end-to-end with a combination of captioning loss and a commitment loss to stabilize dictionary learning.

Experimental Setup

The model was evaluated on two benchmark datasets without large-scale vision-language pre-training to isolate the impact of the architectural changes:

  • YouCook2: Focused on procedural actions in 2,000 cooking videos.
  • ActivityNet Captions: A broad dataset containing 100,000 dense descriptions of diverse human activities.

Key Insights and Results

The "Action Attraction" Phenomenon

Analysis of the learned roles reveals that the discrete dictionary naturally converges on predicates and verbs, such as "crush," "pour," and "melt." Despite no explicit linguistic supervision for parts of speech, the model's inductive bias identifies these actions as the most efficient structural representations for video content.

Figure 4: Analysis of learned roles shows a strong preference for action-oriented predicates like 'crush', 'pour', and 'add'.

Quantitative and Qualitative Performance

The R3-Transformer consistently outperformed baseline Transformers and previous state-of-the-art methods. On YouCook2, the CIDEr score improved from 41.68 to 50.22, while ActivityNet saw a jump from 20.72 to 29.68. Human evaluations further validated these results, with judges preferring R3-Transformer captions 69% of the time for ActivityNet, citing better relevance and action accuracy.

Conclusion

The R3-Transformer demonstrates that incorporating neuro-symbolic inductive biases can significantly enhance a model's ability to describe complex video sequences. Future work will explore non-deterministic dictionary lookups and the integration of these biases into large-scale pre-trained models to further improve spatial granularity and handling of crowded scenes.

Summary updated

Abstract

Neuro-symbolic representations have proved effective in learning structure information in vision and language. In this paper, we propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning. Our approach uses a dictionary learning-based method of learning relations between videos and their paired text descriptions. We refer to these relations as relative roles and leverage them to make each token role-aware using attention. This results in a more structured and interpretable architecture that incorporates modality-specific inductive biases for the captioning task. Intuitively, the model is able to learn spatial, temporal, and cross-modal relations in a given pair of video and text. The disentanglement achieved by our proposal gives the model more capacity to capture multi-modal structures which result in captions with higher quality for videos. Our experiments on two established video captioning datasets verifies the effectiveness of the proposed approach based on automatic metrics. We further conduct a human evaluation to measure the grounding and relevance of the generated captions and observe consistent improvement for the proposed model. The codes and trained models can be found at https://github.com/hassanhub/R3Transformer
BibTeX
@misc{akbari2020neuro,
  author = {Hassan Akbari and Hamid Palangi and Jianwei Yang and Sudha Rao and Asli Celikyilmaz and Roland Fernandez and Paul Smolensky and Jianfeng Gao and Shih-Fu Chang},
  title = {{Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language}},
  year = {2020},
  eprint = {2011.09530},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.09530},
}
Figure