Distraction-free Embeddings for Robust VQA

Atharvan Dogra, Deeksha Varshney, Ashwin Kalyan, Ameet Deshpande, Neeraj KumararXiv 2023 · arXiv

TL;DR

DRAX (Distraction Removal and Attended Cross-Alignment) is a novel framework for Video Question Answering that explicitly filters out "distractor" information in the latent space. By combining attention-guided masking with semantic vector alignment, it achieves state-of-the-art results on complex reasoning tasks within the challenging SUTD-TrafficQA benchmark.

What Problem Does This Paper Address?

Video Question Answering (VQA) is notoriously difficult because it requires models to juggle dense, multi-modal information—appearance, motion, and language—simultaneously. Humans are naturally good at "attention control," the ability to ignore irrelevant background noise (distractors) to focus on a specific task.

Current AI models, however, often suffer from information overload. Even when models use attention mechanisms to weigh features, small amounts of irrelevant information can still leak into the latent embeddings, degrading performance. While some researchers have tried sparse sampling (looking at fewer frames), this paper argues that the real issue lies in how we fuse and refine these representations in the latent space.

Methods

The authors introduce DRAX, which stands for Distraction Removal and Attended Cross-Alignment. The framework operates through a structured hierarchy, conditioning appearance, motion, and linguistic features step-by-step.

Figure 1: Structured Hierarchy of Input Modalities to the Distraction Removal and Attended Cross Alignment (DRAX) blocks. Each solid (or gradient) rectangle represents a set of vectors, until after the output of Answer Candidates DRAX block where the set of vectors is reduced to a single to fuse with answer candidates.

Key technical innovations include:

  1. Distraction Removal (DR): Instead of just weighting features, DRAX identifies "representative relevance scores" for each query. It sets a threshold based on the highest relevance score and completely masks out (zeros) any attention weights that fall below a certain percentage of that peak. This acts as a hard filter for noise.
  2. Attended Cross-Alignment (AX): When fusing two modalities (e.g., motion and appearance), the model doesn't just concatenate them. It treats one as an "anchor" and projects the other (the "tailing" vector) into the anchor's vector space. This ensures that the resulting fused embedding is semantically and dimensionally aligned.
  3. Dynamic Thresholding: The "strictness" of the distraction filter increases at each layer of the encoder. As the model gets deeper, it becomes more ruthless in discarding irrelevant information, focusing only on the most task-critical features.
The internal architecture of the DRAX block showing the DR and AX functions.

Datasets

The primary evaluation is performed on the SUTD-TrafficQA dataset. This is a particularly grueling benchmark consisting of:

  • 10,080 videos of traffic scenes.
  • 62,535 QA pairs categorized into six reasoning tasks: Basic Understanding, Attribution (causal analysis), Event Forecasting, Reverse Reasoning (looking at the past), Counterfactual Inference (hypothetical "what if" scenarios), and Introspection (preventive advice).

Key Insights and Ideas

  • Latent Space Hygiene: The core "aha" moment is that we shouldn't just focus on what data to include, but actively work to exclude noise from the latent representations themselves.
  • Subspace Masking: By applying distraction removal within multi-head attention subspaces, the model can ignore specific parts of a feature vector that are distracting while keeping the relevant parts.
  • Reasoning over Perception: The results suggest that while distraction removal doesn't significantly change "Basic Understanding" (simple object detection), it is transformative for high-level reasoning tasks like counterfactuals and causal attribution.
Figure 4: Different parts (subspaces) of the embeddings masked due to multiple heads. Above example: 3-head space (H = 3)

Key Results

DRAX-large achieved an overall accuracy of 40.4%, outperforming previous state-of-the-art models like Eclipse (37.05%) and HCRN (36.49%).

Comparison of DRAX against previous SOTA methods.

The most dramatic improvements were seen in complex reasoning sub-tasks:

  • Counterfactual Inference: Jumped from a baseline of 0.4% (HCRN) to 43.06%.
  • Introspection: Improved from 0.25% to 31.76%.
  • Attribution: Gained nearly 5 absolute points over the baseline with significantly more robustness (lower standard deviation).

Limitations and Future Work

The authors acknowledge that their study focused on models using offline feature extraction (like ResNet and ResNeXt). They did not directly compare against massive, end-to-end pre-trained vision-language models like BLIP-2 or VIOLET, noting that such comparisons were outside their current scope.

Future work involves applying the distraction removal mechanism to other vision-language tasks beyond VQA, such as video captioning or text-to-video retrieval, to see if "latent space hygiene" provides similar benefits across the board.

Summary updated

Abstract

The generation of effective latent representations and their subsequent refinement to incorporate precise information is an essential prerequisite for Vision-Language Understanding (VLU) tasks such as Video Question Answering (VQA). However, most existing methods for VLU focus on sparsely sampling or fine-graining the input information (e.g., sampling a sparse set of frames or text tokens), or adding external knowledge. We present a novel "DRAX: Distraction Removal and Attended Cross-Alignment" method to rid our cross-modal representations of distractors in the latent space. We do not exclusively confine the perception of any input information from various modalities but instead use an attention-guided distraction removal method to increase focus on task-relevant information in latent embeddings. DRAX also ensures semantic alignment of embeddings during cross-modal fusions. We evaluate our approach on a challenging benchmark (SUTD-TrafficQA dataset), testing the framework's abilities for feature and event queries, temporal relation understanding, forecasting, hypothesis, and causal analysis through extensive experiments.
BibTeX
@misc{dogra2023distraction,
  author = {Atharvan Dogra and Deeksha Varshney and Ashwin Kalyan and Ameet Deshpande and Neeraj Kumar},
  title = {{Distraction-free Embeddings for Robust VQA}},
  year = {2023},
  eprint = {2309.00133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.00133},
}
Figure