Multi-modal Circulant Fusion for Video-to-Language and Backward

Aming Wu, Yahong Han

Introduction

In multi-modal artificial intelligence, "fusion" is the critical process of combining disparate data types—such as video frames and text queries—into a unified representation. Traditional fusion techniques typically rely on simple operations: element-wise addition, multiplication, or concatenation. While computationally efficient, these methods are often "shallow," assuming that correlations only exist between corresponding dimensions of the feature vectors.

Conversely, more sophisticated approaches like Bilinear Pooling capture exhaustive cross-dimensional interactions but suffer from a massive parameter overhead, making them memory-intensive and difficult to scale. Researchers have introduced Multi-modal Circulant Fusion (MCF) as a "Goldilocks" solution: a method that provides the expressive power of bilinear pooling with the efficiency of simple addition.

Comparison of traditional fusion methods (a, b, c) versus the proposed Multi-modal Circulant Fusion (d).

The MCF Module

The core innovation of MCF lies in its use of circulant matrices to facilitate complex feature interactions without adding learnable parameters to the fusion layer itself.

Architectural Flow

As illustrated in the architecture diagram below, the process begins by passing visual and textual features through fully connected (FC) layers to project them into a shared d-dimensional space. These vectors are then reshaped into circulant matrices, where each row is a cyclically shifted version of the previous one.

Detailed architecture of the MCF module showing the flow from input features to the final fused representation M.

By interacting a vector from one modality with the circulant matrix of another, the model effectively forces every element of the text feature to interact with every element of the video feature. This "sliding window" effect captures correlations across all possible dimensional offsets. The authors explored two primary interaction types: standard matrix multiplication and a more efficient element-wise product across shifted rows.

Convolutional Decoding

For video captioning (Video-to-Language), the researchers replaced standard Recurrent Neural Networks (RNNs) with a multi-stage decoder utilizing dilated convolutions. This allows the model to refine descriptions hierarchically, moving from coarse actions to fine-grained object details.

Evaluation and Results

The MCF module was evaluated across three major benchmarks: MSVD and MSRVTT for video captioning, and TACoS for temporal activity localization (the "backward" task of finding text-described actions in long videos).

  • State-of-the-Art Performance: On the MSVD dataset, MCF achieved top-tier results, reaching a METEOR score of 33.72% and a CIDEr score of 75.46%.
  • Localization Gains: When integrated into the TALL (Temporal Activity Localization) framework, MCF provided a 4.2% performance boost on the TACoS dataset (R@5, IoU=0.1) compared to baseline fusion methods.
  • Qualitative Improvements: The model demonstrated a superior ability to identify specific entities (e.g., "pasta" or "makeup") where simpler models often defaulted to generic descriptors like "something."

Key Insights

  1. Parameter-Free Sophistication: MCF is defined a priori. It uses the mathematical structure of circulant matrices to generate interactions, avoiding the "parameter explosion" typical of bilinear methods.
  2. Bidirectional Utility: The same fusion logic proved equally effective for both generating text from video and searching through video using text queries.
  3. Efficiency vs. Complexity: While MCF avoids extra parameters, it does introduce a processing cost due to matrix operations. However, this cost is significantly lower than that of full bilinear pooling while maintaining high expressivity.

Conclusion and Future Work

Multi-modal Circulant Fusion represents a significant step forward in balancing model complexity with performance. While the current circulant structure is fixed, future research may explore learnable shifts or more flexible matrix structures to capture even more nuanced multi-modal relationships. Additionally, testing the robustness of MCF on hour-long video streams remains a key challenge for scaling these insights to real-world applications.

Summary updated

Abstract

Multi-modal fusion has been widely involved in focuses of the modern artificial intelligence research, e.g., from visual content to languages and backward. Common-used multi-modal fusion methods mainly include element-wise product, element-wise sum, or even simply concatenation between different types of features, which are somewhat straightforward but lack in-depth analysis. Recent studies have shown fully exploiting interactions among elements of multi-modal features will lead to a further performance gain. In this paper, we put forward a new approach of multi-modal fusion, namely Multi-modal Circulant Fusion (MCF). Particularly, after reshaping feature vectors into circulant matrices, we define two types of interaction operations between vectors and matrices. As each row of the circulant matrix shifts one elements, with newly-defined interaction operations, we almost explore all possible interactions between vectors of different modalities. Moreover, as only regular operations are involved and defined a priori, MCF avoids increasing parameters or computational costs for multi-modal fusion. We evaluate MCF with tasks of video captioning and temporal activity localization via language (TALL). Experiments on MSVD and MSRVTT show our method obtains the state-of-the-art for video captioning. For TALL, by plugging into MCF, we achieve a performance gain of roughly 4.2% on TACoS.
BibTeX
@inproceedings{wu2018multi,
  author = {Aming Wu and Yahong Han},
  title = {{Multi-modal Circulant Fusion for Video-to-Language and Backward}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/143},
}
Figure