Spatial Localization from Image Level Labels

85 concepts72 papers3 proposals

When we train a deep convolutional neural network (CNN) to classify images, we only provide global labels—such as whether an image contains a "backpack," a "smiling face," or a specific "thoracic disease." Yet, internally, the network must discover where these features reside to make an accurate prediction.

As Ziwei Liu et al. demonstrated in Deep Learning Face Attributes in the Wild, individual filters in a deep network naturally learn to represent high-level semantic concepts spatially. Because convolutional layers preserve spatial translation equivariance, the location of a feature in the input image corresponds directly to its location in the final feature maps. Weakly Supervised Object Localization (WSOL) exploits this property to extract precise spatial coordinates or heatmaps without ever showing the model a single bounding box during training.

The Mechanics of Class Activation Mapping

How do we translate a global classification score into a spatial map? The foundational mechanism relies on Global Average Pooling (GAP). Consider a network where the final convolutional layer outputs a set of feature maps $f_k(x, y)$, where $k$ represents the channel index and $(x, y)$ represents the spatial coordinates on the grid.

Instead of flattening these maps into a massive dense vector—which destroys spatial information—a GAP layer computes the spatial average for each channel independently:

$$F_k = \frac{1}{Z} \sum_{x,y} f_k(x, y)$$

where $Z$ is the total number of spatial pixels in the feature map. These aggregated features $F_k$ are then fed into a linear classification layer with weights $w_k^c$ assigned to each class $c$. The final classification score $S_c$ before the softmax activation is given by:

$$S_c = \sum_k w_k^c F_k = \sum_k w_k^c \left( \frac{1}{Z} \sum_{x,y} f_k(x, y) \right)$$

By swapping the order of summation, we can rewrite this expression to expose the spatial contribution of each grid cell:

$$S_c = \frac{1}{Z} \sum_{x,y} \left( \sum_k w_k^c f_k(x, y) \right)$$

The inner term defines the Class Activation Map (CAM) for class $c$:

$$M_c(x, y) = \sum_k w_k^c f_k(x, y)$$

As highlighted by Kenan E. Ak et al. in Learning Attribute Representations With Localization for Flexible Fashion Search, this map directly reveals the spatial regions that contribute most to the classification decision. By thresholding $M_c(x, y)$, we can generate pseudo-bounding boxes or region masks. For instance, Yuanyi Zhong et al. leveraged this in DAP: Detection-Aware Pre-Training With Weak Supervision to pre-train object detectors using only classification datasets, converting noisy activation maps into functional localization signals early in the learning pipeline.

Alternative Pooling and Attribution Strategies

While Global Average Pooling is highly effective, it tends to spread activations across entire regions, sometimes missing fine-grained details or failing when multiple distinct instances are present. Researchers have developed alternative pooling and backpropagation strategies to refine these maps.

Instead of averaging, one can look for local extrema. In Local Extremum Mapping for Weak Supervision Learning on Mammogram Classification and Localization, Minjuan Zhu et al. showed that backpropagating identified extrema from score maps highlights the exact pixels driving the classification. Similarly, Heba Sailem utilized backpropagation-based saliency maps in Discovery of Rare Phenotypes in Cellular Images Using Weakly Supervised Deep Learning to pinpoint rare cellular events from image-level labels alone.

Another prominent paradigm is Multiple Instance Learning (MIL), where an image is treated as a "bag" of local patches. Thibaut Durand et al. introduced the WELDON framework in WELDON: Weakly Supervised Learning of Deep Convolutional Neural Networks, which explicitly selects both the most representative and the most contradictory regions to constrain the localization. To balance the extremes of max-pooling (which only finds the single most discriminative point) and average pooling (which dilutes the signal), Dan Xu proposed a learned top-N average pooling layer in Viraliency: Pooling Local Virality to map complex, non-localized concepts like image virality.

Overcoming the Annotation Bottleneck

The true power of deriving localization from weak labels lies in scaling AI to domains where dense pixel-level annotations are prohibitively expensive or require rare expert knowledge.

In medical imaging, manual segmentation of tumors or pathologies is a massive bottleneck. In ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases, Xiaosong Wang et al. demonstrated that models trained purely on text-mined disease labels could accurately localize thoracic pathologies. This capability was extended by Chaochao Yan et al. in Weakly Supervised Deep Learning for Thoracic Disease Classification and Localization on Chest X-rays by aggregating activations from class-specific feature maps to map 14 distinct diseases. Even when training signals are as sparse as post-hoc patient survival times, Renato Hermoza et al. showed in Post-Hoc Overall Survival Time Prediction From Brain MRI that saliency maps can successfully localize brain tumors.

This framework generalizes beyond standard classification to vision-language models. Instead of mapping discrete class labels, models can align text descriptions to spatial features. Shantanu Ghosh et al. developed Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography, using text-based supervision from clinical reports to localize breast abnormalities. Similarly, in the open-vocabulary setting, Dahun Kim et al. demonstrated in Region-Centric Image-Language Pretraining for Open-Vocabulary Detection that max-pooling over candidate regions matching a text prompt allows a model to discover object boundaries without explicit coordinate-based supervision.

Key Foundational Work

Deep Learning Face Attributes in the Wild (2015) — contributed 1 concept elements to this direction.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_class_activation_mapping_a389f659,
  title = {class activation mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-activation-mapping/rdr_a389f6593a5e4c30bbf8287aed63379a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_class_activation_mapping_a389f659_v1,
  title = {class activation mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-activation-mapping/rdr_a389f6593a5e4c30bbf8287aed63379a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake