Abstract
The goal of visual answering localization (VAL) in the video is to obtain a relevant and concise time clip from a video as the answer to the given natural language question. Early methods are based on the interaction modelling between video and text to predict the visual answer by the visual predictor. Later, using the textual predictor with subtitles for the VAL proves to be more precise. However, these existing methods still have cross-modal knowledge deviations from visual frames or textual subtitles. In this paper, we propose a cross-modal mutual knowledge transfer span localization (MutualSL) method to reduce the knowledge deviation. MutualSL has both visual predictor and textual predictor, where we expect the prediction results of these both to be consistent, so as to promote semantic knowledge understanding between cross-modalities. On this basis, we design a one-way dynamic loss function to dynamically adjust the proportion of knowledge transfer. We have conducted extensive experiments on three public datasets for evaluation. The experimental results show that our method outperforms other competitive state-of-the-art (SOTA) methods, demonstrating its effectiveness.
Introduction
Visual Answer Localization (VAL) is a critical task in video understanding, requiring a model to identify the specific temporal segment (start and end times) within a video that answers a natural language query. For instance, given a video about car maintenance and the question "How do you begin an automatic start?", the model must locate the exact clip where this is explained.
Historically, VAL research has split into two primary approaches. Visual predictors focus on raw video frames, which are robust but can struggle with complex semantic transitions. Textual predictors leverage subtitles, which often provide denser semantic cues but are vulnerable to transcription errors or temporal misalignment with visual actions. The fundamental challenge lies in cross-modal knowledge deviation: the visual and textual streams do not always perfectly synchronize in time or semantic depth.
Methods: The MutualSL Framework
To address these deviations, the authors introduce MutualSL (Mutual Knowledge Transfer Span Localization). This framework treats VAL as a collaborative task between a visual predictor and a textual predictor, forcing them to achieve consistency.
Dual Predictors and Cross-Modal Fusion
MutualSL maintains two distinct predictors. The visual predictor processes frame features extracted via an I3D model, while the textual predictor utilizes subtitle spans processed through a pre-trained language model. A key innovation is the Cross-Modal Look-up Table, which maps subtitle spans to raw timestamps and vice versa, allowing the two predictors to communicate despite operating in different feature spaces (frames vs. text spans).
One-way Dynamic Loss (ODL)
The core of the mutual learning process is the One-way Dynamic Loss (ODL) function. This mechanism uses Intersection over Union (IoU) to evaluate the performance of each predictor in real-time. If the textual predictor demonstrates higher accuracy, the ODL increases the weight of knowledge transfer from the textual to the visual predictor. To prevent model collapse and ensure independent learning, a "stop gradient" (sg) mechanism is employed, treating the other modality's output as a "pseudo-label" rather than a direct gradient source.
Experimental Results
The model was evaluated on three diverse datasets: MedVidQA (medical instructions), TutorialVQA (software tutorials), and VehicleVQA (automotive maintenance).
As shown in the results table, MutualSL establishes new state-of-the-art (SOTA) performance across all benchmarks. Notably, it achieved a significant jump of 3.28 mIoU on TutorialVQA and 1.23 mIoU on VehicleVQA compared to previous best methods like VPTSL.
Key Insights
- Textual Dominance: Analysis of the dynamic loss weights revealed that the textual predictor is generally the stronger "teacher," providing a more reliable anchor for localization.
- Mutual Benefit: Ablation studies confirmed that Mutual Knowledge Transfer (MKT) is vital. Performance for both modalities dropped significantly when MKT was disabled, proving that cross-modal collaboration reduces the inherent deviations between what is seen and what is said.
Conclusion
MutualSL demonstrates that synchronizing visual and textual predictors through mutual knowledge transfer significantly enhances visual answer localization. While the model currently relies on high-quality subtitles, future work aims to explore more efficient knowledge distillation techniques and more sophisticated alignment methods to handle scenarios with missing or noisy textual data.
Summary updated