Contrastive Bidirectional Transformer for Temporal Representation Learning

Chen Sun, Fabien Baradel, Kevin Murphy, Cordelia SchmidarXiv 2019 · OpenReview

Abstract

This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.

Introduction

The success of self-supervised learning in Natural Language Processing (NLP), particularly through models like BERT, has largely been driven by "masked language modeling"—predicting missing tokens based on bidirectional context. However, extending this paradigm to video is challenging. Unlike text, which consists of discrete words, video is a continuous stream of high-dimensional pixel data.

Previous attempts, such as VideoBERT, relied on "quantizing" video frames into a discrete vocabulary of "visual words." This process is inherently lossy, discarding fine-grained visual details—like subtle textures or precise hand movements—that are often critical for understanding complex actions. The Contrastive Bidirectional Transformer (CBT) addresses this by applying bidirectional context learning directly to raw, real-valued video feature vectors, avoiding the information bottleneck of quantization.

The Contrastive Bidirectional Transformer (CBT)

The CBT architecture leverages the power of transformers for continuous temporal sequences. The process begins with visual feature extraction, where short windows of video frames are processed using a 3D Convolutional Neural Network (S3D) to generate initial feature vectors.

The CBT architecture for pre-training and downstream tasks.

The core innovation lies in the training objective. Instead of a traditional softmax layer used for discrete token prediction, CBT employs Noise Contrastive Estimation (NCE). During pre-training, the model is presented with a masked sequence of video features and must identify the "correct" hidden feature from a set of "noise" distractors (other frames within the same batch). This contrastive approach encourages the model to learn the underlying temporal structure and semantics of the video without requiring a discrete vocabulary.

Furthermore, the authors introduce a cross-modal learning component. By utilizing Automatic Speech Recognition (ASR) to extract transcripts from narrated videos, a cross-modal transformer learns to maximize mutual information between video and text sequences. Crucially, this alignment occurs at the sequence level rather than the frame level, making the model robust to the natural temporal misalignment between speech and visual action.

Experimental Results

The CBT model was pre-trained on massive unlabeled datasets, including Kinetics and the HowTo100M dataset, which contains 1.2 million instructional videos. Evaluation across diverse downstream tasks demonstrated significant improvements over existing self-supervised methods.

Comparison of CBT with state-of-the-art self-supervised methods on action recognition benchmarks.

Action Recognition and Segmentation

On the UCF101 benchmark, CBT improved accuracy from the previous best of 75.7% to 79.5%. On HMDB51, the performance jumped from 35.7% to 44.6%. The model also showed a massive leap in action segmentation on the COIN dataset, reaching 53.9% frame accuracy compared to the previous state-of-the-art of 34.3%.

Action Anticipation and Captioning

CBT significantly outperformed VideoBERT in action anticipation. For instance, on the 50Salads dataset, it achieved 39.1% accuracy, a stark contrast to VideoBERT's 5.5%. In video captioning on the YouCook2 dataset, CBT surpassed previous methods across all standard metrics (BLEU-4, METEOR, ROUGE-L, and CIDEr), indicating that richer visual representations directly translate to better natural language descriptions.

Discussion and Key Insights

The results highlight several critical findings:

  • Continuous vs. Discrete: Moving from discrete "visual words" to continuous vectors via NCE is the primary driver of performance, as it preserves fine-grained details lost during quantization.
  • Instructional Synergy: Instructional videos are particularly effective for cross-modal learning because the narration is highly correlated with the visual actions.
  • Sequence-Level Alignment: Aligning modalities at the sequence level effectively handles the temporal lag often found in narrated content.
  • Bidirectional Context: The transformer's ability to look both forward and backward in time allows it to define actions based on their preceding and succeeding context (e.g., "pouring batter" before "flipping a pancake").

While the benefits of cross-modal learning are clear in instructional settings, its effectiveness in "arbitrary" videos remains a challenge. Additionally, the authors noted that scaling the number of transformer layers beyond a certain point did not yield further gains, likely due to the relative size of the video datasets compared to the massive corpora used in NLP. Future work will focus on scaling to even larger datasets and refining cross-modal architectures to further advance self-supervised video understanding.

Summary updated

Abstract

This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.
BibTeX
@misc{sun2019contrastive,
  author = {Chen Sun and Fabien Baradel and Kevin Murphy and Cordelia Schmid},
  title = {{Contrastive Bidirectional Transformer for Temporal Representation Learning}},
  year = {2019},
  eprint = {1906.05743},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.05743},
}
Figure