Abstract
Long-form video language grounding (LFVLG) requires identifying brief moments of interest within extensive video streams, a task characterized by extremely low ground truth coverage. Traditional models often struggle with visual domain shifts and the vast search space. We present EI-VLG (Environment Infusion for Video-Language Grounding), a framework that leverages Multi-modal Large Language Models (MLLMs) to generate descriptive environmental captions. These captions serve as semantic anchors, allowing the model to utilize "common sense" knowledge to prune irrelevant scenes. Our approach achieves state-of-the-art performance on the EgoNLQ benchmark, demonstrating the efficacy of infusing textual environmental cues into video features.
The Challenge of Long-Form Video Grounding
Video Language Grounding (VLG) involves localizing a specific moment in a video based on a natural language query. While short-form VLG is relatively mature, Long-Form VLG (LFVLG) presents a "needle-in-a-haystack" scenario. In datasets like EgoNLQ, the target moment typically covers only 2.3% of the total video duration, compared to 27% in short-form datasets like Charades-STA.
Existing models often rely on superficial visual patterns. For instance, a model trained on wooden chopping boards might fail to identify a red plastic one in a kitchen. Humans overcome this by using contextual experience—knowing that a chopping board is defined by its function and typical environment. EI-VLG aims to bridge this gap by providing models with a proxy for human-like environmental understanding.
Proposed Method: EI-VLG
The EI-VLG framework consists of four primary components designed to integrate high-level semantic descriptions into the grounding process.
1. Environment Encoder (EE)
To capture the context of the video, we utilize a powerful off-the-shelf MLLM, LLaVA-v1.6 34B, to generate detailed captions at 10-second intervals. These captions describe the scene's layout, present objects, and spatial relationships, effectively translating visual context into a rich textual representation.
2. Contrastive Learning
To ensure these environmental cues are relevant to the user's query, we fine-tune the text encoder using a marginal log-likelihood loss. This process aligns the query (e.g., "Where did I put the chopping board?") with the specific environmental captions that describe the relevant scene, helping the model focus its search.
3. Environment Infuser (EI)
This module merges textual environment features with standard video features. Empirical results showed that simple concatenation followed by a linear projection outperformed more complex cross-attention mechanisms, providing a robust way to "infuse" the video representation with semantic context.
4. VLG Model
The infused features are processed by a standard grounding backbone, such as VSLNet or ActionFormer, to predict the precise start and end timestamps of the target action.
Experimental Results
We evaluated EI-VLG on the EgoNLQ dataset, which contains egocentric videos averaging 8 minutes in length. This dataset is particularly challenging due to its massive search space.
EI-VLG achieved new state-of-the-art results on the EgoNLQ validation set:
- [email protected]: 10.0% (surpassing GroundVQA's 9.4%)
- [email protected]: 35.2% (surpassing 34.3%)
Ablation studies highlighted that the scale of the MLLM is critical; the 34B parameter model provided the necessary detail to distinguish between similar scenes, whereas smaller models produced captions that were too generic to be useful.
Discussion and Limitations
While EI-VLG significantly improves grounding accuracy, it introduces a high computational overhead. Preprocessing the 260-hour EgoNLQ dataset required approximately 1,300 GPU hours on NVIDIA RTX A5000 hardware. Additionally, the system's performance is intrinsically linked to the quality of the MLLM's captions; hallucinations or omissions in the text can lead to grounding failures.
Future work will focus on optimizing caption generation efficiency and improving the robustness of the model in scenarios where environmental descriptions may be incomplete or inaccurate.
Summary updated