Abstract
Conversational AI systems typically rely on textual dialogue alone, but human communication is inherently multimodal. Conditioning utterances on visual context can lead to more realistic and grounded interactions. A significant barrier to this goal is the scarcity of large-scale labeled datasets for visual dialogue. We propose the Future Utterance Prediction (FUP) task, which requires predicting the next spoken sentence in a video using both visual frames and transcribed speech. By leveraging the abundance of instructional videos online, we train a model at scale without manual annotations. Our Co-attentional Multimodal Video Transformer (CoMVT) outperforms text-only baselines and achieves state-of-the-art performance on downstream VideoQA benchmarks.
Introduction: The Need for Visual Grounding
Human language is deeply situated in the physical world. Deictic expressions like "this" or "that" are ubiquitous in natural speech but remain ambiguous without visual grounding. While existing dialogue datasets are often small or artificially constructed, instructional videos provide a naturally occurring, large-scale source of visually grounded narration. This work explores using these videos to train models that "look before they speak," resolving linguistic ambiguities through visual context.
Methods: Co-attentional Multimodal Video Transformer (CoMVT)
The proposed CoMVT architecture employs a two-stream approach to process text and video. Text is processed via a BERT-based encoder, while video features combine global scene information (via S3D) and local object-level features (via Graph-RISE). To handle the high dimensionality of video, an attention-based mechanism aggregates redundant frames into a compact feature set. The core of the model is the Co-attentional Transformer (CoTRM) block, which enables deep, layer-wise interaction between modalities.
Datasets and Training
We introduce How2FUP, a massive dataset derived from HowTo100M containing 35 million training clips. The model is trained using a ranking loss for next-utterance prediction and a masked language modeling (MLM) loss. This self-supervised approach allows the model to learn complex visual-linguistic relationships without manual labels, effectively repurposing the natural co-occurrence of narration and action in "how-to" videos.
Key Results and Insights
Our experiments yield several critical findings:
- Visual Context as a Force Multiplier: The multimodal CoMVT model achieved a 68.34% R@1 on How2FUP, an 8% absolute improvement over text-only baselines. The gain is most pronounced when the transcript contains expressions requiring visual grounding.
- Objects Matter: Adding object-level features to global scene features provided a significant performance boost, suggesting that "looking" at specific tools or ingredients is key to understanding the dialogue.
- FUP as a Pretraining Task: Pretraining on FUP transfers exceptionally well to VideoQA. After finetuning, our model achieved state-of-the-art results on benchmarks such as MSVD-QA (42.6% accuracy) and MSRVTT-QA (39.5% accuracy).
Conclusion and Future Work
This work highlights the potential of using instructional videos for large-scale multimodal pretraining. By grounding dialogue in visual context, we move closer to conversational agents that can understand and interact with the physical world. Future research will explore generative response synthesis, improved temporal alignment between speech and action, and the incorporation of temporal object tracking.
Summary updated