TDAN: Top-Down Attention Networks for Enhanced Feature Selectivity in CNNs

Shantanu Jaiswal, Basura Fernando, Cheston Tan

TL;DR

Traditional attention modules in CNNs operate purely "bottom-up," meaning they calculate attention using only local information from a single layer. This paper introduces the Top-Down Attention Module (TDAM), which uses high-level semantic context to guide low-level feature selection through an iterative "visual searchlight." TDAM consistently outperforms state-of-the-art attention mechanisms across multiple benchmarks while remaining highly parameter- and memory-efficient.

What Problem Does This Paper Address?

Most popular attention mechanisms in Convolutional Neural Networks (CNNs)—such as Squeeze-and-Excitation (SE) or CBAM—are strictly feedforward. They modulate features based entirely on the local information available within that specific layer.

However, the human visual cortex relies heavily on top-down feedback. When we look at a cluttered scene, our brain uses high-level context ("what" we are looking for) to tell lower-level sensory areas "where" to focus. Without this top-down flow, standard CNN attention modules are limited by the representational capacity of individual layers, making them less effective at resolving fine-grained ambiguities or localizing objects in complex environments.

Methods

The authors propose TDAM, a lightweight attention module that introduces an iterative feedback loop within a standard convolutional block over $T$ computational steps.

  1. Attentional Searchlight ($\mathcal{S}_t$): High-level semantic features from the top of a block ($\mathbf{X}^N$) and low-level features from the bottom ($\mathbf{X}^0$) are spatially pooled and passed through Multi-Layer Perceptrons (MLPs) to generate a 1D modulation vector called the "searchlight." The authors designed two variants: TDjoint (which models both top and bottom features) and TDtop (which uses only top features for maximum parameter efficiency).
  2. Channel and Spatial Modulation: The searchlight first performs channel attention by scaling the lower-level feature map ("highlighting features of interest"). Next, a pointwise convolution between the searchlight vector and the channel-scaled map generates a 2D spatial map ($\mathcal{A}$) to intensify salient spatial locations.
  3. Iterative Refinement: This contextually modulated feature map is fed forward through the block again. To stabilize training across these iterative loops, the network utilizes unique batch normalization layers for each computational step.

Datasets

The module was rigorously evaluated on several standard computer vision datasets:

  • ImageNet-1k (ILSVRC-12) and ImageNet-V2 for large-scale object classification and weakly-supervised localization.
  • CUB-200 (Caltech-Birds) and Stanford Dogs for fine-grained image classification.
  • MS-COCO for multi-label image classification.

Key Insights and Ideas

  • The "Searchlight" Boosts Resolution Robustness: By using a dynamic pointwise convolution with the top-down searchlight vector to generate spatial attention maps, TDAM avoids the pitfalls of static convolutional kernel-based attention (like CBAM). This makes the network significantly more robust to changes in input resolution during inference.
  • Emergent "Attention Shifting": Without any explicit localization supervision, TDAM naturally learns to sequentially focus on different objects or features across iterative steps. For example, in a cluttered image, it might isolate a primary object in the first step and shift to a secondary object or a finer detail in the next step.
Fig. 1. Illustration of how iterative top-down feedback computation can help increase feature selectivity and thereby process individual salient features (left) or objects (right) at each computation step. Green arrows indicate feedforwarded inputs to TD blocks and red dotted arrows indicate top-down computation within TD block.
  • The Feedback Distance "Goldilocks Zone": The authors discovered that top-down feedback works best when restricted to a short span of layers ($m = 1$ to $3$). Attempting to span the entire network with a single feedback loop causes massive training instability because intermediate layers receive radically shifting input distributions at each step.

Key Results

  • ImageNet Classification: Integrating TDjoint ($t=2, m=1$) into a ResNet50 backbone achieved a 78.96% top-1 accuracy on ImageNet-V1, a 1.45% improvement over the baseline. It outperformed prominent attention modules like SE, CBAM, ECA, and FCA while maintaining superior parameter and memory efficiency.
  • Weakly-Supervised Object Localization: TDAM achieved a remarkable 5% top-1 localization accuracy improvement over baseline ResNet50 on ImageNet. Interestingly, purely channel-based attention methods (like SE or FCA) actually degraded localization performance compared to the baseline.
  • Downstream Transfer: When used as a pretrained backbone for fine-grained tasks, TDAM yielded a 1.5% accuracy boost on CUB-200 and a 2-point mAP increase on MS-COCO over standard ResNet50.

Limitations and Future Work

  • Diminishing Returns with More Steps: Performance peaks at $T=2$ feedback steps. Adding more steps ($T \ge 3$) causes a slight decline in accuracy, which the authors attribute to optimization challenges akin to vanishing gradients during unrolled backpropagation.
  • Increased FLOPs: Because features are iteratively processed through the convolutional block, TDAM inherently increases the number of FLOPs (e.g., a ~20% FLOP increase when moving from feedback distance $m=1$ to $m=3$). However, because the module requires fewer parameters and memory operations, actual throughput (FPS) remains highly competitive with feedforward baselines.

Summary updated

Abstract

Attention modules for Convolutional Neural Networks (CNNs) are an effective method to enhance performance on multiple computer-vision tasks. While existing methods appropriately model channel-, spatial- and self-attention, they primarily operate in a feedforward bottom-up manner. Consequently, the attention mechanism strongly depends on the local information of a single input feature map and does not incorporate relatively semantically-richer contextual information available at higher layers that can specify "what and where to look" in lower-level feature maps through top-down information flow. Accordingly, in this work, we propose a lightweight top-down attention module (TDAM) that iteratively generates a "visual searchlight" to perform channel and spatial modulation of its inputs and outputs more contextually-relevant feature maps at each computation step. Our experiments indicate that TDAM enhances the performance of CNNs across multiple object-recognition benchmarks and outperforms prominent attention modules while being more parameter and memory efficient. Further, TDAM-based models learn to "shift attention" by localizing individual objects or features at each computation step without any explicit supervision resulting in a 5% improvement for ResNet50 on weakly-supervised object localization. Source code and models are publicly available at: https://github.com/shantanuj/TDAM_Top_down_attention_module .
BibTeX
@inproceedings{jaiswal2022tdan,
  author = {Shantanu Jaiswal and Basura Fernando and Cheston Tan},
  title = {{TDAN: Top-Down Attention Networks for Enhanced Feature Selectivity in CNNs}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-19806-9\_15},
}
Figure