Visual Answer Localization with Cross-Modal Mutual Knowledge Transfer

Yixuan Weng, Bin Li

TL;DR

Researchers have developed MutualSL, a new framework for Visual Answer Localization (VAL) that synchronizes visual and textual data to find precise answers in videos. By using a "mutual knowledge transfer" approach with a dynamic loss function, the model allows visual and textual predictors to learn from one another, significantly outperforming previous state-of-the-art methods across medical, software, and automotive video datasets.

What Problem Does This Paper Address?

Visual Answer Localization (VAL) is the task of pinpointing the exact start and end times of a video clip that answers a specific natural language question (e.g., "How do you begin an automatic start?").

Current methods generally fall into two camps:

  1. Visual Predictors: These analyze raw video frames. They are robust but often struggle with frequent scene changes and complex semantics.
  2. Textual Predictors: These rely on video subtitles. While often more precise because text contains dense semantic information, they fail if subtitles are missing, poorly transcribed, or misaligned with the visual action.

The core challenge is the cross-modal knowledge deviation. What we see (visual) and what we hear/read (textual) don't always align perfectly in time or meaning. Most models pick one modality as the primary driver, leaving valuable information on the table from the other.

Methods

The authors propose MutualSL (Mutual Knowledge Transfer Span Localization), which treats VAL as a collaborative effort between two "experts": a visual predictor and a textual predictor.

Key Components:

  • Dual Predictors: Unlike previous models that favor one modality, MutualSL maintains both. The visual predictor outputs time points based on frame features (using I3D), while the textual predictor outputs spans based on subtitle timelines (using pre-trained language models).
  • Cross-Modal Look-up Table: To allow the two predictors to "talk," the authors use a look-up table that translates subtitle spans into raw timestamps and vice versa. This ensures that even though the models operate in different "spaces," they are aiming for the same target.
  • One-way Dynamic Loss (ODL): This is the "brain" of the transfer process. It uses Intersection over Union (IoU) to measure how well each predictor is performing. If the textual predictor is performing better, the ODL increases the "knowledge transfer" weight to help the visual predictor catch up.
  • Stop Gradient: To prevent the models from collapsing into a single biased state, they use a "stop gradient" (sg) mechanism. This ensures the predictors learn independently while still using the other's output as a "pseudo-label" or guide.

Datasets

The model was evaluated on three diverse public datasets to prove its versatility:

  1. MedVidQA: 3,010 medical instructional QA pairs (e.g., first aid, surgical procedures).
  2. TutorialVQA: 6,195 QA pairs from software editing tutorials.
  3. VehicleVQA: 9,482 QA pairs regarding practical vehicle instructions and maintenance.

Key Insights and Ideas

  • Text is the Stronger Teacher: The researchers found that the textual predictor generally outperforms the visual one. In their dynamic loss function, the weight for textual knowledge transfer ($\beta$) was consistently higher than the visual weight ($\alpha$), confirming that subtitles provide a more reliable "anchor" for localization.
  • Mutual Benefit: Even though the textual predictor is stronger, the visual predictor still provides essential context. By forcing them to be consistent with each other, the model reduces the "deviation" that occurs when a subtitle is mentioned slightly before or after the relevant visual action occurs.
  • Dynamic Adaptation: Rather than using static weights for loss, the ODL allows the model to focus on the most reliable information source at different stages of training.

Key Results

MutualSL achieved new State-of-the-Art (SOTA) results across all three benchmarks:

  • MedVidQA: Improved mean IoU (mIoU) by 0.51 over the previous best (VPTSL).
  • TutorialVQA: A significant jump of 3.28 mIoU.
  • VehicleVQA: An improvement of 1.23 mIoU.

Qualitatively, the ablation studies showed that the "Mutual Knowledge Transfer" (MKT) was the primary driver of success. Without MKT, the performance of both the visual and textual predictors dropped significantly, proving that cross-modal collaboration is more effective than modality isolation.

Limitations and Future Work

  • Dependency on Subtitles: While the model improves visual prediction, it still benefits heavily from high-quality subtitles. In videos where subtitles are completely absent or nonsensical, the performance gains might diminish.
  • Computational Overhead: Running two predictors and a fusion layer is more computationally intensive than single-modality models.
  • Future Directions: The authors suggest exploring knowledge distillation—a technique where a large, complex "teacher" model trains a smaller, faster "student" model—to make MutualSL more efficient for real-time applications. They also aim to explore more sophisticated cross-modal alignment techniques beyond simple look-up tables.

Summary updated

Abstract

The goal of visual answering localization (VAL) in the video is to obtain a relevant and concise time clip from a video as the answer to the given natural language question. Early methods are based on the interaction modelling between video and text to predict the visual answer by the visual predictor. Later, using the textual predictor with subtitles for the VAL proves to be more precise. However, these existing methods still have cross-modal knowledge deviations from visual frames or textual subtitles. In this paper, we propose a cross-modal mutual knowledge transfer span localization (MutualSL) method to reduce the knowledge deviation. MutualSL has both visual predictor and textual predictor, where we expect the prediction results of these both to be consistent, so as to promote semantic knowledge understanding between cross-modalities. On this basis, we design a one-way dynamic loss function to dynamically adjust the proportion of knowledge transfer. We have conducted extensive experiments on three public datasets for evaluation. The experimental results show that our method outperforms other competitive state-of-the-art (SOTA) methods, demonstrating its effectiveness <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
BibTeX
@inproceedings{weng2023visual,
  author = {Yixuan Weng and Bin Li},
  title = {{Visual Answer Localization with Cross-Modal Mutual Knowledge Transfer}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095026},
}

Loading related content...

Figure