Deep Multimodal Feature Encoding for Video Ordering

Vivek Sharma, Makarand Tapaswi, Rainer StiefelhagenarXiv 2020 · arXiv

Abstract

This paper addresses the challenge of efficiently fusing multiple data modalities—video, audio, and text—for video understanding. We introduce Temporal Compact Bilinear Pooling (TCBP), a method that captures complex pairwise interactions across modalities while maintaining a compact representation. By training on a self-supervised "proxy task" of ordering shuffled video clips, the model learns rich, transferable multimodal embeddings. Our results demonstrate that TCBP significantly outperforms standard pooling methods in both temporal ordering and downstream action recognition tasks.

Introduction

While humans naturally integrate sight, sound, and context (text/dialogue) to understand a story, AI models often struggle to fuse these modalities efficiently. Standard fusion techniques, such as simple concatenation or averaging, often fail to capture the complex, pairwise interactions between different types of data. Furthermore, training these models usually requires massive amounts of manually labeled data.

To bypass this, researchers use self-supervised learning, where the model solves a "proxy task" using the inherent structure of the data. Previous temporal proxy tasks often looked at very short sequences (e.g., 3 frames), which fails to capture the high-level semantic "flow" of a scene. This paper aims to learn clip-level representations by ordering sequences of clips lasting several seconds each.

The multi-modal ordering task involves taking shuffled clips and predicting their correct chronological sequence.

Methods

The core technical contribution is Temporal Compact Bilinear Pooling (TCBP).

Feature Extraction

The authors extract features from five distinct sources to capture a holistic view of the video content:

  • Objects (I): ResNet50 trained on ImageNet.
  • Scenes (P): ResNet50 trained on Places365.
  • Motion (R): 3D ResNet50 trained on Kinetics.
  • Audio (A): SoundNet.
  • Text (S): GloVe word embeddings from closed captions.

The TCBP Operator and Proxy Task

Standard bilinear pooling captures all pairwise interactions between features but suffers from a "curse of dimensionality." TCBP uses the Tensor Sketch algorithm to project these interactions into a much smaller space. Crucially, TCBP extends this to the temporal dimension, ensuring consistent encoding regardless of when a feature appears in a clip.

The architecture of the proposed multimodal encoding and ordering loss pipeline.

The model is given a set of shuffled clips from a movie scene and must predict their correct chronological order. It uses an order violation loss, which penalizes the model if the learned "time score" of an earlier clip is not lower than that of a later clip.

Key Results and Insights

  • Ordering Performance: TCBP achieved an ordering accuracy of 86.23% on the LSMDC test set, vastly outperforming Mean Pooling (69.58%) and standard CBP (85.06%).
  • Multimodal Synergy: Combining modalities (e.g., Audio + Places + ImageNet) consistently outperforms single-modality models. Interestingly, scene layout (Places) and object presence were stronger cues for long-term narrative progression than short-term movement (Motion).
  • Action Recognition Transferability: When evaluated on UCF101 and HMDB51, features learned via the ordering task set new state-of-the-art benchmarks for C3D-based models, reaching 88.03% and 61.58% accuracy respectively.

Limitations and Future Work

The current inference approach uses a brute-force check of all permutations, which is computationally expensive for longer sequences. Future work could explore more scalable sorting algorithms like Beam Search. Additionally, addressing text sparsity through speech-to-text could provide a richer training signal for the multimodal embeddings.

Summary updated

Abstract

True understanding of videos comes from a joint analysis of all its modalities: the video frames, the audio track, and any accompanying text such as closed captions. We present a way to learn a compact multimodal feature representation that encodes all these modalities. Our model parameters are learned through a proxy task of inferring the temporal ordering of a set of unordered videos in a timeline. To this end, we create a new multimodal dataset for temporal ordering that consists of approximately 30K scenes (2-6 clips per scene) based on the "Large Scale Movie Description Challenge". We analyze and evaluate the individual and joint modalities on three challenging tasks: (i) inferring the temporal ordering of a set of videos; and (ii) action recognition. We demonstrate empirically that multimodal representations are indeed complementary, and can play a key role in improving the performance of many applications.
BibTeX
@misc{sharma2020deep,
  author = {Vivek Sharma and Makarand Tapaswi and Rainer Stiefelhagen},
  title = {{Deep Multimodal Feature Encoding for Video Ordering}},
  year = {2020},
  eprint = {2004.02205},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.02205},
}
Figure