Where Is the Tail Class's Evidence? Placement Consistency for Long-Tailed Recognition

Image classifiers trained on long-tailed data, where a few common classes have thousands of photos and many rare classes have a handful, tend to recognize rare classes by their surroundings rather than the object itself. A rare "speedboat" class whose few photos all show open water ends up being detected by the water. We propose a training signal that targets this failure directly. We paste a patch from one image into another, so we know exactly which pixels came from which class. Then we require that the model's spatial evidence for each class, its class activation map (a per-pixel score of how much each location supports a class), sits only on the pixels that came from that class. If speedboat evidence lights up on water that came from a shark photo, the model is penalized.

The reason this matters is a point the existing attention-consistency literature misses. The standard tool asks that attention maps move with the image under flips and shifts. A background shortcut satisfies that test perfectly well, because the water flips along with the boat. To separate object from shortcut, we need a transformation that moves the object relative to its background. Compositing two images is that transformation, and its coordinate map is known exactly. We weight the penalty most heavily for the rarest classes, where shortcuts are worst.

Core Claim

Requiring class evidence to appear only on pixels from that class's source image, inside mixed images, removes the background shortcuts that tail classes pick up. It does so without borrowing attention shapes from other classes and without penalizing large or spread-out objects. Its dividing line between shortcut and valid context is principled. Context shared with other training classes, such as water that also surrounds sharks and ferries, gets suppressed. Context specific to the rare class, such as a host plant that appears only with one insect, survives. Under compositing this is a stronger and more appropriate constraint than flip or rotation equivariance, which cannot tell a tracked object from a tracked background.

For a mixed image $\tilde I$ built from source images $I_a$ and $I_b$, let $M_a$ be the binary mask of pixels taken from $I_a$ and $\bar M_a = 1 - M_a$ its complement. Let $A_c(\cdot)$ be the activation map for class $c$, and let $\mathcal{T}_a$ be the known crop-and-place map that sends coordinates of $I_a$ into $\tilde I$. The placement loss for the class $y_a$ of $I_a$ is

$$\ell_{\text{place}}(y_a) = w_{y_a}\Big( \big\| \bar M_a \odot A_{y_a}(\tilde I) \big\|_1 + \beta \,\big\| M_a \odot \big(A_{y_a}(\tilde I) - \mathcal{T}_a(A_{y_a}(I_a))\big) \big\|_2^2 \Big),$$

and the same term is applied symmetrically for $y_b$. The two parts do different jobs:

  • The first term is the new ingredient. It drives class evidence to zero on foreign pixels.
  • The second term is the equivariance requirement of Visual Attention Consistency, generalized from flips to compositing. Weighted by $\beta$, it asks the evidence inside the pasted region to match the source image's own map, moved into place.

The class weight is $w_c = (n_{\max}/n_c)^{\gamma}$, where $n_c$ is the training count of class $c$, $n_{\max}$ the largest count, and $\gamma \ge 0$ a single exponent. This applies the strongest pressure to the rarest classes.

The activation maps come from the classifier weights applied to the last feature map. That costs one extra forward pass for the source image, not gradient-based saliency.

Relation to Prior Work

Work that motivates the proposal. - Shortcut-Resistant CAM Distillation (SRCD) establishes that shortcut learning is a central failure in long-tailed recognition.

  • It repairs tail activation maps by distilling a teacher map built from the top-K similar head classes.
  • It filters those maps with energy scores that reward compact, concentrated attention.
  • It switches distillation on only late in training.
  • Its authors concede two problems: the compactness scores penalize sprawling objects and erase valid biological context, and the "white shark" proxy for "speedboat" is a semantic mismatch. - Calibrating Class Activation Maps (CAMC) likewise observes rare-class maps highlighting background. It reweights features with cached class prototypes.

Both of these import attention structure from elsewhere. Our constraint uses only the known pixel provenance of each training image, imposes no shape prior, and can run from the first epoch.

Work that supplies the method. - Visual Attention Consistency (VAC) gives us the inverse-transform map-matching loss. VAC applied it uniformly with flips and scaling on multi-label data, never on long-tailed data. - Erasing Attention Consistency (EAC) shows that flip consistency plus random erasing stops attention from collapsing onto one facial fragment under label noise.

EAC also constrains our design. It reports that rotations and scaling did not help, so we do not expect a richer set of geometric transforms to rescue plain equivariance.

Work closest in mechanics. - GLMC already mixes images (MixUp and CutMix) inside a one-stage long-tailed method. - It enforces agreement only between global feature vectors. - It assigns labels by patch area, so a tail patch pasted on a head background can be "explained" by the background.

We keep GLMC's mixing and change only where evidence is allowed to live.

Work that sets the bar. RIDE and Balanced Contrastive Learning (BCL) are the strong backbones we plug into. RIDE also supplies the warning that reweighting trades tail bias for variance. Our class weight $w_c$ must therefore be checked for head-class damage, not assumed harmless.

Research Program

Mixing schedule. The method has three pieces. The first is a sampler that deliberately composites tail-class patches onto head-class images, and head patches onto tail images. Shortcut backgrounds are usually shared with head classes, so this is where the off-region term has the most to say.

Patch selection. Random boxes may miss the object. We therefore take tail patches as large jittered boxes (at least half the image area) centered on the peak of a slowly updated averaged model's map. This setup produces a useful conflict when the model attends to water:

  • The patch is mostly water, and its label still demands speedboat evidence.
  • The head image's water now forbids that evidence.
  • The only consistent solution is to find evidence that is not water, which large boxes make available.

The source map in the inside term comes from this averaged model with gradients stopped, which avoids a collapse in which both maps trivially go to zero.

Labels. The classification target uses area-proportional labels as in GLMC. As an ablation, we also use labels proportional to the source map's mass inside the patch.

Analysis program. We will test three things:

  • Whether tail-class evidence on head-image backgrounds actually falls.
  • Whether the gain concentrates in few-shot classes.
  • Whether valid class-specific context is preserved where SRCD's compactness prior removes it.

Baselines

Naive reference. Plain cross-entropy training of the same ResNet backbone with standard flip-and-crop augmentation. Every method shares one recipe: identical data splits, epochs, augmentation pipeline, and backbone. Each new hyperparameter ($\gamma$, $\beta$, loss weight) gets the same 12-configuration validation sweep that we grant to the baselines' own key hyperparameters.

Uniform flip equivariance (the obvious intersection). VAC-style flip equivariance with a fixed weight. A frequency-weighted variant using our $w_c$ is the most likely thing a reviewer would propose, so it is the comparison that tests our central argument that equivariance under flips cannot see background shortcuts. To count as beating it, we must win on few-shot accuracy and on the shortcut benchmark below, not merely tie on overall accuracy.

GLMC. The strongest mixing-based one-stage method, run from its official code. It is the natural host for our loss, because adding $\ell_{\text{place}}$ to GLMC changes exactly one thing.

Reported numbers:

  • CIFAR-100-LT, imbalance factor 100: 55.88% accuracy.
  • ImageNet-LT: 56.3% top-1, with 70.1% many-shot and 45.5% few-shot (with MaxNorm).

Beating it means a gain over our own GLMC reproduction, averaged over three seeds, with no loss in many-shot accuracy beyond seed noise.

SRCD. The strongest existing attention-based competitor. On ImageNet-LT it reports +1.33% overall and +3.29% on tail classes when added to BCL, where BCL reports 57.1% top-1. On iNaturalist 2018 it reports +1.82% over GCL. We will add SRCD and our loss each to the same BCL reproduction under identical schedules.

Beating SRCD means at least matching its overall gain on BCL. On top of that we must exceed it either on iNaturalist few-shot accuracy or on the context-preservation check. If we only tie it on ImageNet-LT, the proposal's claim narrows to "same gain, no proxy classes, no shape prior, one stage." That is a weaker but still reportable outcome.

Other context numbers. On iNaturalist 2018, CAMC reports 73.0% and RIDE reports 72.6%. We will report RIDE plus our loss using the RIDE code.

Key ablation. We drop the off-region term and keep only the inside-patch equivariance term, then separately set $\gamma = 0$. If the off-region term is not responsible for most of the few-shot gain, the core claim is wrong.

Why not just…?

Why not just add frequency-weighted flip equivariance, as in VAC? A flip moves the water with the boat, so a water-attending map already satisfies the loss. EAC found only flips helped and other geometric transforms did not, which suggests richer geometric sets will not fix this. The risk is real: equivariance might still help indirectly by regularizing tail features. The uniform and weighted flip baselines, scored on the controlled shortcut benchmark, settle whether it does. We predict near-zero recovery there, whereas we expect clear recovery from the off-region term.

Why not just use CutMix with area labels, as GLMC already does? Area labels only constrain the total amount of evidence, not its location. A tail label can be satisfied by evidence on the head background. GLMC's own design leaves this open by enforcing agreement only on pooled vectors. Our method is GLMC plus one term, so any gain is attributable to that term alone.

Why not just rely on class-balanced cross-entropy or logit adjustment, which already push the speedboat score down on shark images? These losses lower the tail score on head images, often through a per-class offset, without removing the water evidence. Raising tail gradients can even entrench the shortcut, as SRCD argues. Class-Balanced Loss is our representative here. We measure directly how much evidence the speedboat class places on head-image backgrounds under each loss. If balanced losses already drive this to zero, our premise fails and we will say so.

Why not just distill head-class maps as SRCD does? SRCD needs semantically mismatched proxies and a late training stage, and its compactness prior penalizes spread-out objects and valid context. Our constraint has no shape prior. The context-preservation check below is designed to show the difference, or its absence.

Proposed Experiments

Main benchmarks. We use CIFAR-100-LT at imbalance factors 10, 50, and 100, plus ImageNet-LT, iNaturalist 2018, and Places-LT. We report overall, many-shot, medium-shot, and few-shot top-1 over three seeds. For the localization side, we measure the foreground-to-background attention ratio on Pascal VOC 2012 masks, where SRCD reports roughly doubling this ratio.

Decisive check 1: controlled shortcut benchmark. We build a long-tailed analogue of the localized-shortcut setups in Localized Shortcut Removal, which reports clean CIFAR-10 accuracy recovered from 28.5% to 70.5% in a balanced color-dot setting.

  • Train on CIFAR-100-LT where each tail class's training images carry a class-tied background tint or patch.
  • Copies of the same tints also appear on a random subset of head-class images.
  • Test on clean images and on images with tints reassigned.

The prediction: the off-region term recovers most of the clean few-shot accuracy, while flip equivariance and reweighting recover little.

Decisive check 2: context preservation. We use the same construction, but tie a context cue to exactly one tail class and never show it with any head class. The cue is now genuinely predictive. Our method should keep using it, so reassigning the cue at test time should hurt it. SRCD's compactness prior should discard the cue.

If both methods behave the same in both checks, our proposed distinction between shared shortcuts and class-specific context is not real.

Cost. We report wall-clock and memory relative to GLMC, since only one extra no-gradient forward pass is needed.