Top-Down Visual Attention from Analysis by Synthesis

Baifeng Shi, Trevor Darrell, Xin WangCVPR 2023 · DOI · OpenReview

Overview

AbSViT introduces a principled approach to equipping Vision Transformers (ViTs) with "top-down" attention. By framing vision as Analysis-by-Synthesis, the model uses a feedback loop to focus on task-specific objects, leading to enhanced performance in vision-language tasks and significantly improved robustness against image corruptions.

The Need for Top-Down Attention

Most current AI vision models rely on "bottom-up" attention. When presented with an image containing multiple salient objects, a standard Vision Transformer typically highlights all of them. However, human perception is goal-driven. If asked a specific question about an object, our brains employ "top-down" attention to ignore distractors and focus on relevant details.

Top-down attention allows a model to focus on specific objects based on a goal, such as identifying a husky or cat stripes.

AbSViT addresses this by implementing a feedback loop that refines visual focus based on a task or goal, bridging the gap between classic computer vision theory and modern neural architectures.

Methodology: The AbSViT Inference Cycle

The AbSViT model implements top-down attention through a structured four-step inference cycle:

The AbSViT inference cycle: (1) Initial feedforward, (2) Modulation by a prior, (3) Feedback reconstruction, and (4) Final feedforward with top-down signals.
  1. Initial Feedforward: The image is processed by a standard ViT encoder to extract baseline features.
  2. Modulation: A task-specific "prior" (e.g., a text embedding) weights the output tokens, emphasizing those relevant to the goal.
  3. Feedback: These weighted tokens are sent backward through linear decoders to intermediate layers.
  4. Final Feedforward: The image is re-processed, but the "top-down" signals from the feedback path are integrated into the Value (V) matrix of the self-attention layers.

The model is trained using a variational loss, ensuring the feedback path effectively learns to reconstruct target object features, making the "synthesis" part of the theory functionally active.

Key Insights

  • Value Matrix Modulation: Mathematical derivation revealed that top-down signals are most effective when modifying the Value matrix in self-attention, rather than the Query or Key matrices.
  • Generative-Discriminative Balance: To prevent pixel-level reconstruction from degrading classification performance, the authors utilized a "stop-gradient" trick, allowing generative feedback to guide focus without interfering with discriminative features.
  • Attention as Sparse Reconstruction: The paper builds on the insight that visual attention is functionally equivalent to "sparse reconstruction"—describing a scene using the fewest possible templates.

Experimental Results

AbSViT was evaluated across various benchmarks, demonstrating significant improvements:

  • Vision-Language Tasks: On VQAv2, AbSViT improved accuracy to 68.72% (from a 67.89% baseline). Zero-shot image-to-text retrieval (IR@1) on Flickr30K rose from 42.40% to 45.28%.
  • Robustness: The model excelled on ImageNet-A (natural adversarial examples), achieving 28.2% accuracy compared to the baseline's 25.2%. It also showed superior performance against common image corruptions (ImageNet-C).
  • General Backbone Utility: Even without specific task priors, AbSViT improved semantic segmentation mIoU by up to 2.0% on datasets like PASCAL VOC and ADE20K.

Limitations and Future Work

While promising, AbSViT faces challenges. Standard datasets like ImageNet often feature single dominant objects, making them suboptimal for training selective attention. Additionally, the feedback loop increases computational costs (FLOPs). Future research may focus on training with multi-object datasets and unsupervised objectives to further enhance selective attention capabilities without requiring manual labels.

Summary updated

Abstract

Current attention algorithms (e.g., self-attention) are stimulus-driven and highlight all the salient objects in an image. However, intelligent agents like humans often guide their attention based on the high-level task at hand, focusing only on task-related objects. This ability of task-guided top-down attention provides task-adaptive representation and helps the model generalize to various tasks. In this paper, we consider top-down attention from a classic Analysis-by-Synthesis (AbS) perspective of vision. Prior work indicates a functional equivalence between visual attention and sparse reconstruction; we show that an AbS visual system that optimizes a similar sparse reconstruction objective modulated by a goal-directed top-down signal naturally simulates top-down attention. We further propose Analysis-by-Synthesis Vision Transformer (AbSViT), which is a top-down modulated ViT model that variationally approximates AbS, and achieves controllable top-down attention. For real-world applications, AbSViT consistently improves over baselines on Vision-Language tasks such as VQA and zero-shot retrieval where language guides the top-down attention. AbSViT can also serve as a general backbone, improving performance on classification, semantic segmentation, and model robustness.
BibTeX
@inproceedings{shi2023top,
  author = {Baifeng Shi and Trevor Darrell and Xin Wang},
  title = {{Top-Down Visual Attention from Analysis by Synthesis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00209},
}
Figure