Models that learn to localize objects from image-level tags alone, such as "this photo contains a bird and a car," usually work like this. A classifier is trained on the tags. Its internal feature maps are then read back as a heatmap of where each class is, called a class activation map. These localizers have a well-documented blind spot for small objects. A small car at the edge of a street scene often never appears in the heatmap. This matters because downstream methods that repair the heatmap cannot repair what was never there. Small Objects Matters in Weakly-supervised Semantic Segmentation shows small objects are effectively invisible to standard metrics. It fixes this with size-weighted training on pseudo-labels, and it explicitly leaves open the problem of fixing the initial heatmap stage. We argue the object is lost in two places. First, uniform pooling inside the backbone blends a few-pixel object into surrounding background texture. Second, the loss terms that stop heatmaps from bleeding into co-occurring background, such as the water under a boat, use area penalties that are measured in absolute pixels. Those penalties are cheapest to satisfy by erasing tiny objects entirely.
The proposal has two parts. First, it replaces the backbone's uniform downsampling with learned importance-weighted pooling. Second, it replaces the absolute-area penalty used against background bleeding with a scale-free one, which charges an object for growing relative to its own confident core rather than for its pixel count. Both pieces are justified by a diagnostic that measures, stage by stage, where small objects disappear. The experiments are designed as a two-factor study so that each piece can fail on its own. The study is also run against the obvious cheaper fix of simply running the network at higher resolution.
Core Claim
Under image-level supervision, small-object loss at the heatmap stage has two causes that can be separated. One is resolution: evidence is diluted during pooling. The other is the regularizer: the anti-bleeding area penalty is scale-biased. Fixing only one leaves most of the loss in place. Concretely, we predict the following. On a size-balanced benchmark, a backbone with learned pooling plus a scale-free background penalty will raise small-instance recall in the raw heatmaps (before any downstream segmentation training) more than either change alone. It will also beat a compute-matched baseline that simply removes downsampling or upsamples the input. We further predict that importance maps trained only from tags do not collapse onto a single discriminative peak. The pooling normalizes within small windows, so emphasizing a bird's beak in one window cannot suppress its wing in another. This prediction is checked directly rather than assumed.
Research Program
The method supplier for the first ingredient is LIP: Local Importance-Based Pooling. LIP replaces each strided downsampling step with a weighted average whose weights come from a small learned convolutional module. It raised Faster R-CNN small-object detection precision on COCO from 21.9% to 24.0%, but only under full supervision with box labels. Whether its importance maps stay informative when the only signal is a classification loss on tags is untested. That question is the "Gap 1" tension of whether learned pooling collapses to peaks. We use the LIP formulation unchanged at every stride of a ResNet-50.
The second ingredient modifies background suppression. Background Activation Suppression for Weakly Supervised Object Localization and Semantic Segmentation (BAS) trains a foreground map by minimizing the fraction of activation that falls outside it. It needs a global area constraint with a fixed weight to keep the map from expanding everywhere. Geometry Constrained Weakly Supervised Object Localization (GC-Net) similarly penalizes shape area while making the classifier maximally confused on the masked-out background. In both, the area term charges a 64-pixel object and a 64,000-pixel object by the same per-pixel rule. So, relative to its size, the penalty pushes a small object's mask toward zero much harder.
We keep BAS's background-ratio loss and replace its area term as follows. At each training step, the current foreground map $M$ is split, without gradient, into connected regions $R_k$. Each region has a high-confidence core $C_k$, the pixels above a fixed fraction of that region's peak. The area penalty is then
$$\mathcal{L}_{\text{area}} = \frac{1}{K}\sum_{k=1}^{K} \log \frac{\sum_{p \in R_k} M(p)}{\sum_{p \in C_k} M(p)},$$
where $K$ is the number of regions and $M(p)$ is the soft foreground value at pixel $p$. Because the penalty depends on the ratio of a region's extent to its own core, doubling the extent of a tiny object costs exactly as much as doubling a large one. Each region is also averaged with equal weight. This moves the per-instance weighting idea of Small Objects Matters in Weakly-supervised Semantic Segmentation out of downstream segmentation and into heatmap generation, which is the step its authors left open.
The diagnostic runs alongside both ingredients and is a contribution in itself. For each backbone stage, we train a frozen-feature linear probe to detect the presence of each ground-truth instance from features pooled inside its mask. Instances are binned by size using PASCAL-B annotations, and the probe is used for evaluation only. We also measure how peaked the importance maps are, as entropy within each pooling window and across the image. This shows at which stage small-instance evidence drops to chance under uniform versus learned pooling. It also shows whether tag-only training makes importance maps collapse.
Baselines
The naive reference point is the original class activation map method under the corrected protocol of Evaluating Weakly Supervised Object Localization Methods Right. That paper shows that once threshold and hyperparameter tuning is restricted to a small held-out annotated set, plain heatmaps match many later methods. It reports the best weakly supervised method at 70.8% on CUB-200-2011 under that protocol. Every method here, including ours, gets that same fixed tuning set and the same number of tuning trials.
The strongest direct competitor on heatmap-stage background separation is BAS. It reports 95.41% GT-known localization accuracy on CUB-200-2011 and 69.6% mIoU on PASCAL VOC 2012 val. We rerun it from its description with the same ResNet-50, schedule, and tuning budget, sweeping its global area weight within that budget. Beating it means higher small-instance recall in raw heatmaps on PASCAL-B without losing more than about one point of pixel mIoU on VOC.
Background-aware Classification Activation Map for Weakly Supervised Object Localization (B-CAM) learns an explicit pure-background feature. It is included because it is the other strong internal background model and is known to degrade when secondary objects go untagged. We expect it to suffer on small objects for the same reason. The file reports no B-CAM number on PASCAL VOC 2012 or PASCAL-B.
For final segmentation, the reference numbers are Progressive Feature Self-Reinforcement for Weakly Supervised Semantic Segmentation at 75.7% VOC val mIoU and 45.4% COCO 2014 mIoU, Knowledge Transfer with Simulated Inter-image Erasing for Weakly Supervised Semantic Segmentation at 73.0% VOC val, and EPS at 71.0% VOC val. The KTSE paper also reports initial pseudo-mask quality of 67.0% mIoU on VOC, which is the closest published number to our primary heatmap-stage target. These are context rather than the decisive comparison, because their downstream pipelines differ.
The ablations form the core two-factor design. The four cells are uniform versus learned pooling, crossed with absolute versus scale-free area penalty. A fifth cell feeds our heatmaps into the size-weighted downstream training of the small-objects paper, to test whether heatmap-stage and downstream fixes add up. That paper reports gains of up to 8.9 points in small-instance accuracy across baselines, and absolute small-instance scores for its baselines on PASCAL-B are not available on Lacuna.
Why not just…?
Why not just run the network at higher resolution, by upsampling the input or removing the last strides with dilated convolutions? This is the real threat. If small objects die only from resolution, a resolution change at matched FLOPs should do as well as learned pooling. We run both variants (input ×2, and an output stride of 8) compute-matched against our model. If either matches the learned-pooling cell, we will report that the pooling ingredient is unnecessary and that the contribution reduces to the scale-free penalty. The penalty's effect can still be measured on top of the high-resolution backbone.
Why not just tune BAS's area weight per dataset? A single global weight cannot be correct for both a tiny and a large object in the same image, which is exactly PASCAL-B's composition. The sweep inside the BAS baseline tests this directly. If some setting of the weight closes the gap on small instances without hurting large ones, the scale-free term is not needed.
Why not just annotate a few boxes? The corrected-evaluation paper reports 92.0% on CUB from 5 to 10 fully labeled images per class, far above weak methods. We take this seriously and include a few-shot fine-tuning baseline on PASCAL-B under the same annotation budget as our tuning set. Our claim is limited: we do not claim to beat few-shot supervision overall. We claim that the small-instance failure belongs to the weak pipeline and can be fixed at the heatmap stage. Whether that fix remains worthwhile next to few-shot boxes is reported, not assumed.
Why not just use an off-the-shelf saliency detector to mark background, as EPS does? Railroad Is Not a Train: Saliency As Pseudo-Pixel Supervision for Weakly Supervised Semantic Segmentation does exactly this, using the inverse of a saliency map to supervise a background class. Its own analysis notes failure on very small objects, where saliency maps are blank. It also relies on a pretrained saliency model that may not exist in specialized domains. We run EPS from its released code at halbielee/EPS and report its PASCAL-B small-instance numbers next to ours.
Proposed Experiments
All models use ResNet-50 trained on PASCAL VOC 2012 with the augmented training set and image-level tags only. The primary metric is instance-aware mIoU and its small-instance component on PASCAL-B, computed on raw heatmaps before any downstream segmentation network. Pixel mIoU on VOC 2012 val is reported as a check that large objects are not sacrificed. COCO 2014 serves as a harder clutter test. CUB-200-2011 is evaluated under the corrected protocol (maximum box accuracy with a fixed tuning set) to confirm the method does not regress on the single-object case. The centered-Gaussian heuristic is reported there too, because that paper showed it beats several published methods on CUB.
The first decisive check is the two-factor table on PASCAL-B. Our claim requires the combined cell to exceed both single-ingredient cells and the compute-matched high-resolution baselines on small-instance recall. The second check is the stage-wise probe. It should show small-instance evidence surviving deeper into the network under learned pooling, with importance-map entropy staying well above the single-peak level. A collapse would refute the window-normalization argument and would be reported as a negative result about learned pooling under weak supervision. The third check feeds each cell's heatmaps into one fixed downstream segmentation recipe, the size-weighted training of the small-objects paper. This tests whether heatmap-stage recovery carries through to final masks or is undone downstream.
References
- Small Objects Matters in Weakly-supervised Semantic Segmentation
- LIP: Local Importance-Based Pooling
- Background Activation Suppression for Weakly Supervised Object Localization and Semantic Segmentation
- Geometry Constrained Weakly Supervised Object Localization
- Evaluating Weakly Supervised Object Localization Methods Right
- Background-Aware Classification Activation Map for Weakly Supervised Object Localization
- Progressive Feature Self-Reinforcement for Weakly Supervised Semantic Segmentation
- Knowledge Transfer with Simulated Inter-image Erasing for Weakly Supervised Semantic Segmentation
- Railroad Is Not a Train: Saliency As Pseudo-Pixel Supervision for Weakly Supervised Semantic Segmentation