Introduction
Spatio-temporal video grounding (STVG) aims to localize a target object in a video sequence based on a natural language query. While fully supervised methods achieve high accuracy, they require dense, frame-by-frame bounding box annotations, which are prohibitively expensive to collect at scale. Weakly-supervised STVG (WS-STVG) offers a more scalable alternative by learning from video-sentence pairs alone. However, these models often struggle with spurious correlations—mistakenly associating a word with the wrong object in a scene containing multiple similar entities.
As shown in the figure above, a query like "the woman points at another woman" presents a significant challenge. A flat model might associate the word "woman" with both individuals, failing to distinguish the subject (target) from the object (interference). WINNER addresses this by decomposing both modalities into matching hierarchical structures.
The WINNER Framework
The core philosophy of WINNER is that structural relationships within language can guide the discovery of corresponding structures in video, thereby resolving ambiguity. The framework consists of four primary components:
- Language Structure Decomposition: The input sentence is decomposed into a hierarchical tree. Starting from individual words, the model merges tokens into increasingly complex phrases based on semantic compatibility.
- Structure-guided Video Hierarchical Understanding (SVHU): Instead of treating the video as a flat sequence, WINNER uses the language tree to extract relevant visual features. A structural attention mechanism focuses on visual cues that match specific nodes in the language hierarchy (e.g., differentiating the visual representation of an action like "points at" from the actors involved).
- Top-down Feature Backtracking: To maintain global-to-local consistency, the model performs a top-down pass. This ensures that the representation of a parent node (a phrase) is consistent with its children (individual words or sub-phrases), refining the grounding of specific components.
- Decomposition Structure Alignment (DSA): A hierarchical contrastive learning objective is employed to align the language tree with video "tubes." By matching every level of the hierarchy, the model learns to distinguish between similar objects by leveraging their unique roles in the sentence structure.
Experimental Results
WINNER was evaluated on two benchmark datasets: VidSTG and HC-STVG.
- VidSTG: On declarative sentences, WINNER achieved a Mean Video Intersection over Union (m_vIoU) of 11.61, a significant improvement over the previous state-of-the-art weakly-supervised method, Vis-Ctx (9.34).
- HC-STVG: This movie-based dataset is particularly challenging due to crowded scenes. WINNER reached 14.20 m_vIoU, nearly 50% higher than the baseline.
Notably, WINNER outperformed several older fully supervised models, demonstrating that structural inductive biases can be more effective than dense manual annotations in certain contexts.
Conclusion and Future Work
WINNER demonstrates that hierarchical decomposition is a powerful tool for overcoming the limitations of weak supervision in video grounding. By explicitly modeling the relationships between entities, the framework effectively filters out spurious correlations. Currently, the model relies on pre-trained object detectors to generate candidate tubes. Future research aims to develop end-to-end systems where language queries can directly influence the detection process, potentially improving the grounding of small or partially occluded objects.
Summary updated