TL;DR
DRAX (Distraction Removal and Attended Cross-Alignment) is a novel framework for Video Question Answering that explicitly filters out "distractor" information in the latent space. By combining attention-guided masking with semantic vector alignment, it achieves state-of-the-art results on complex reasoning tasks within the challenging SUTD-TrafficQA benchmark.
What Problem Does This Paper Address?
Video Question Answering (VQA) is notoriously difficult because it requires models to juggle dense, multi-modal information—appearance, motion, and language—simultaneously. Humans are naturally good at "attention control," the ability to ignore irrelevant background noise (distractors) to focus on a specific task.
Current AI models, however, often suffer from information overload. Even when models use attention mechanisms to weigh features, small amounts of irrelevant information can still leak into the latent embeddings, degrading performance. While some researchers have tried sparse sampling (looking at fewer frames), this paper argues that the real issue lies in how we fuse and refine these representations in the latent space.
Methods
The authors introduce DRAX, which stands for Distraction Removal and Attended Cross-Alignment. The framework operates through a structured hierarchy, conditioning appearance, motion, and linguistic features step-by-step.
Key technical innovations include:
- Distraction Removal (DR): Instead of just weighting features, DRAX identifies "representative relevance scores" for each query. It sets a threshold based on the highest relevance score and completely masks out (zeros) any attention weights that fall below a certain percentage of that peak. This acts as a hard filter for noise.
- Attended Cross-Alignment (AX): When fusing two modalities (e.g., motion and appearance), the model doesn't just concatenate them. It treats one as an "anchor" and projects the other (the "tailing" vector) into the anchor's vector space. This ensures that the resulting fused embedding is semantically and dimensionally aligned.
- Dynamic Thresholding: The "strictness" of the distraction filter increases at each layer of the encoder. As the model gets deeper, it becomes more ruthless in discarding irrelevant information, focusing only on the most task-critical features.
Datasets
The primary evaluation is performed on the SUTD-TrafficQA dataset. This is a particularly grueling benchmark consisting of:
- 10,080 videos of traffic scenes.
- 62,535 QA pairs categorized into six reasoning tasks: Basic Understanding, Attribution (causal analysis), Event Forecasting, Reverse Reasoning (looking at the past), Counterfactual Inference (hypothetical "what if" scenarios), and Introspection (preventive advice).
Key Insights and Ideas
- Latent Space Hygiene: The core "aha" moment is that we shouldn't just focus on what data to include, but actively work to exclude noise from the latent representations themselves.
- Subspace Masking: By applying distraction removal within multi-head attention subspaces, the model can ignore specific parts of a feature vector that are distracting while keeping the relevant parts.
- Reasoning over Perception: The results suggest that while distraction removal doesn't significantly change "Basic Understanding" (simple object detection), it is transformative for high-level reasoning tasks like counterfactuals and causal attribution.
Key Results
DRAX-large achieved an overall accuracy of 40.4%, outperforming previous state-of-the-art models like Eclipse (37.05%) and HCRN (36.49%).
The most dramatic improvements were seen in complex reasoning sub-tasks:
- Counterfactual Inference: Jumped from a baseline of 0.4% (HCRN) to 43.06%.
- Introspection: Improved from 0.25% to 31.76%.
- Attribution: Gained nearly 5 absolute points over the baseline with significantly more robustness (lower standard deviation).
Limitations and Future Work
The authors acknowledge that their study focused on models using offline feature extraction (like ResNet and ResNeXt). They did not directly compare against massive, end-to-end pre-trained vision-language models like BLIP-2 or VIOLET, noting that such comparisons were outside their current scope.
Future work involves applying the distraction removal mechanism to other vision-language tasks beyond VQA, such as video captioning or text-to-video retrieval, to see if "latent space hygiene" provides similar benefits across the board.
Summary updated