Abstract
While most conversational AI systems focus on textual dialogue only, conditioning utterances on visual context (when it's available) can lead to more realistic conversations. Unfortunately, a major challenge for incorporating visual context into conversational dialogue is the lack of large-scale labeled datasets. We provide a solution in the form of a new visually conditioned Future Utterance Prediction task. Our task involves predicting the next utterance in a video, using both visual frames and transcribed speech as context. By exploiting the large number of instructional videos online, we train a model to solve this task at scale, without the need for manual annotations. Leveraging recent advances in multimodal learning, our model consists of a novel co-attentional multimodal video transformer, and when trained on both textual and visual context, outperforms baselines that use textual inputs alone. Further, we demonstrate that our model trained for this task on unlabelled videos achieves state-of-the-art performance on a number of downstream VideoQA benchmarks such as MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA.
Introduction
Most current conversational AI systems are "blind" — they process text but lack awareness of the physical environment. While humans seamlessly combine sight and speech to communicate (e.g., "Pass me that wrench"), AI struggles with this "visual grounding." The primary bottleneck has been data; existing datasets for visual dialogue are often small or manually annotated.
To address this, we propose the Future Utterance Prediction (FUP) task. Instead of manual labeling, we leverage the natural pairing of narration and action in millions of online instructional videos. These videos serve as a nearly infinite source of "free" supervision, where speakers often describe their actions in real-time, providing perfect visual context for the spoken words.
Methodology: The CoMVT Model
We introduce the Co-attentional Multimodal Video Transformer (CoMVT), designed to handle the complexity of video data through several key innovations.
Architecture
CoMVT utilizes a Two-Stream Architecture that maintains separate paths for text and vision. This allows the use of specialized encoders — BERT for text and S3D for video — while enabling the modalities to interact through lateral co-attention blocks.
To achieve Object-Level Awareness, the model extracts features for individual objects within frames. To manage computational costs, a Compact Feature Set Extraction module uses attention to aggregate redundant object features into a manageable set, reducing FLOPs by up to 39% while maintaining performance.
Training Objectives
The model is trained on two self-supervised tasks:
- Next Utterance Prediction: Selecting the correct next sentence from a pool of 100 candidates given a video clip and transcript.
- Masked Language Modeling (MLM): Predicting missing words in the transcript based on both surrounding text and visual cues.
Experimental Results
The researchers repurposed large-scale video datasets into new benchmarks, including HowToFUP (35 million clips) and COIN-FUP.
Performance Gains
The full CoMVT model achieved a Recall@1 of 68.34% on HowToFUP, significantly outperforming the text-only baseline (59.21%). This improvement underscores the importance of visual context. In many instances, the model correctly predicted sentences referring to objects visible on screen but not previously mentioned in the dialogue.
Downstream Transfer
A model trained only for FUP on unlabelled videos achieves state-of-the-art performance on standard Video Question Answering (VideoQA) benchmarks. On MSVD-QA, CoMVT reached 42.6% accuracy, a substantial jump over the previous state-of-the-art of 36.1%.
Discussion and Future Work
While CoMVT demonstrates the power of visual grounding, several challenges remain. The current model focuses on ranking candidates rather than generating original sentences. Additionally, performance is affected by noise in Automated Speech Recognition (ASR) and the temporal misalignment that occurs when speakers describe actions before or after they happen. Future work will focus on generative capabilities and improving multimodal alignment across longer time horizons.
Summary updated