Deep learning models excel at semantic recognition when provided with global labels—for instance, identifying whether a medical scan contains a tumor or a satellite image contains an aircraft. However, there is a fundamental disconnect between knowing what is in an image and knowing where it is. When models are trained purely on global, image-level tags, they struggle to delineate precise spatial boundaries. This discrepancy is known as the weakly supervised localization gap.
As noted by George Papandreou et al. in Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation, image-level tags completely lack spatial context, making it incredibly difficult for models to learn precise object boundaries from scratch. This challenge spans domains, from identifying aircraft in remote sensing imagery (Zhize Wu et al. in Convolutional Neural Network Based Weakly Supervised Learning for Aircraft Detection From Remote Sensing Image) to segmenting tumors in digital pathology (Marvin Lerousseau et al. in Weakly supervised multiple instance learning histopathological tumor segmentation).
To close this gap, researchers must design mechanisms that force networks to expand their focus from highly specific visual cues to full, cohesive object structures.
Why Standard Networks Fail to Localize
To understand why the localization gap exists, we must look at how standard classification networks are optimized. Most modern architectures use a Global Average Pooling (GAP) layer right before the final classification decision. GAP collapses spatial dimensions into a single feature vector, forcing the network to find the most discriminative features to minimize classification loss.
If a model can confidently classify an image as containing a "cat" just by seeing its ears and whiskers, the network has no incentive to learn the boundaries of the paws or tail. It zeroes in on a sparse, highly specific region. This creates a severe localization error: the predicted region either covers only a small fraction of the object or bleeds unpredictably into the background.
As Bowen Dong et al. observe in Boosting Weakly Supervised Object Detection via Learning Bounding Box Adjusters, without explicit coordinate-level supervision, models naturally struggle to distinguish between an entire object and its most prominent parts.
Forcing the Network to See the Whole Object
How do we encourage a network trained only on global labels to map out full spatial extents? Researchers have developed several clever algorithmic constraints to counteract this discriminative part collapse.
Entropy Minimization
One approach is to treat object localization as a latent optimization problem. In Min-Entropy Latent Model for Weakly Supervised Object Detection, Fang Wan 0001 et al. introduce a thermodynamic-inspired framework that minimizes the randomness or entropy of the localized regions. By penalizing highly fragmented or overly ambiguous attention maps, the model is forced to aggregate scattered activations into a cohesive, tight bounding box that covers the entire object.
Architectural Redesign
Instead of relying on standard classification heads that discard spatial context, we can redesign the output layers to preserve pixel-level awareness. Ildoo Kim proposes the Spatially Attentive Output Layer for Image Classification, which replaces the standard GAP layer with a dual-branch system. This architecture simultaneously predicts class scores and pixel-wise attention weights, generating high-resolution localization maps as a natural byproduct of the classification process.
Progressive Adaptation
Another strategy involves transitioning the model through stages of increasing spatial granularity. For instance, Dong Li 0025 outlines a progressive domain adaptation framework in Weakly Supervised Object Localization with Progressive Domain Adaptation that mines object proposals and iteratively refines them using a "mask-out" strategy, gradually shifting the network's focus from global classification to local detection.
Leveraging Sparse Spatial Anchors
When pure image-level labels prove too ambiguous, an alternative paradigm is to use ultra-low-cost spatial anchors. These require a fraction of the time needed to draw full bounding boxes or pixel masks, yet they provide just enough structural guidance to anchor the model's attention.
Point Supervision
Instead of drawing a box, annotators simply click a single point on the target object. In Pointly-Supervised Action Localization, Pascal Mettes et al. demonstrate how single-point clicks can be combined with unsupervised spatio-temporal proposals to bridge the localization gap in video action detection, achieving performance competitive with full box supervision at a 150x lower annotation cost.
Scribble Supervision
For dense prediction tasks like semantic segmentation, a single line or scribble provides a powerful structural prior. Jing Zhang et al. leverage this in Weakly-Supervised Salient Object Detection via Scribble Annotations, using an auxiliary edge detection task and structure-aware losses to propagate sparse scribble information across the entire object. This results in sharp, well-defined boundaries without the blurry artifacts typical of pure image-level supervision.