Look Before you Speak: Visually Contextualized Utterances

Paul Hongsuck Seo, Arsha Nagrani, Cordelia Schmid

Abstract

Conversational AI systems typically rely on textual dialogue alone, but human communication is inherently multimodal. Conditioning utterances on visual context can lead to more realistic and grounded interactions. A significant barrier to this goal is the scarcity of large-scale labeled datasets for visual dialogue. We propose the Future Utterance Prediction (FUP) task, which requires predicting the next spoken sentence in a video using both visual frames and transcribed speech. By leveraging the abundance of instructional videos online, we train a model at scale without manual annotations. Our Co-attentional Multimodal Video Transformer (CoMVT) outperforms text-only baselines and achieves state-of-the-art performance on downstream VideoQA benchmarks.

Introduction: The Need for Visual Grounding

Human language is deeply situated in the physical world. Deictic expressions like "this" or "that" are ubiquitous in natural speech but remain ambiguous without visual grounding. While existing dialogue datasets are often small or artificially constructed, instructional videos provide a naturally occurring, large-scale source of visually grounded narration. This work explores using these videos to train models that "look before they speak," resolving linguistic ambiguities through visual context.

Methods: Co-attentional Multimodal Video Transformer (CoMVT)

The proposed CoMVT architecture employs a two-stream approach to process text and video. Text is processed via a BERT-based encoder, while video features combine global scene information (via S3D) and local object-level features (via Graph-RISE). To handle the high dimensionality of video, an attention-based mechanism aggregates redundant frames into a compact feature set. The core of the model is the Co-attentional Transformer (CoTRM) block, which enables deep, layer-wise interaction between modalities.

The CoMVT architecture uses two streams and co-attention to fuse text and visual features efficiently.

Datasets and Training

We introduce How2FUP, a massive dataset derived from HowTo100M containing 35 million training clips. The model is trained using a ranking loss for next-utterance prediction and a masked language modeling (MLM) loss. This self-supervised approach allows the model to learn complex visual-linguistic relationships without manual labels, effectively repurposing the natural co-occurrence of narration and action in "how-to" videos.

Key Results and Insights

Our experiments yield several critical findings:

  1. Visual Context as a Force Multiplier: The multimodal CoMVT model achieved a 68.34% R@1 on How2FUP, an 8% absolute improvement over text-only baselines. The gain is most pronounced when the transcript contains expressions requiring visual grounding.
  2. Objects Matter: Adding object-level features to global scene features provided a significant performance boost, suggesting that "looking" at specific tools or ingredients is key to understanding the dialogue.
  3. FUP as a Pretraining Task: Pretraining on FUP transfers exceptionally well to VideoQA. After finetuning, our model achieved state-of-the-art results on benchmarks such as MSVD-QA (42.6% accuracy) and MSRVTT-QA (39.5% accuracy).
Qualitative examples show the model correctly identifying objects like "carrots" that text-only models miss.

Conclusion and Future Work

This work highlights the potential of using instructional videos for large-scale multimodal pretraining. By grounding dialogue in visual context, we move closer to conversational agents that can understand and interact with the physical world. Future research will explore generative response synthesis, improved temporal alignment between speech and action, and the incorporation of temporal object tracking.

Summary updated

Abstract

While most conversational AI systems focus on textual dialogue only, conditioning utterances on visual context (when it's available) can lead to more realistic conversations. Unfortunately, a major challenge for incorporating visual context into conversational dialogue is the lack of large-scale labeled datasets. We provide a solution in the form of a new visually conditioned Future Utterance Prediction task. Our task involves predicting the next utterance in a video, using both visual frames and transcribed speech as context. By exploiting the large number of instructional videos online, we train a model to solve this task at scale, without the need for manual annotations. Leveraging recent advances in multimodal learning, our model consists of a novel co-attentional multimodal video transformer, and when trained on both textual and visual context, outperforms baselines that use textual inputs alone. Further, we demonstrate that our model trained for this task on unlabelled videos achieves state-of-the-art performance on a number of downstream VideoQA benchmarks such as MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA.
BibTeX
@inproceedings{seo2021look,
  author = {Paul Hongsuck Seo and Arsha Nagrani and Cordelia Schmid},
  title = {{Look Before you Speak: Visually Contextualized Utterances}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=OJxM69OKBU},
}
Figure