Text-Guided Background Swapping Inside the Network for Few-Shot Recognition

When a classifier sees only one or five pictures of a new category, it cannot tell whether the grass behind a dog or the dog itself defines "dog". The class summary it builds absorbs the background, and it then fails on test images taken in new settings. This proposal fixes that without new images. We use the class name to find which image patches belong to the object. Partway through a frozen vision-language model, we replace the remaining background patches with background patches taken from unrelated images. The later layers then process the swapped scene, and a small adapter is trained to give the same answer regardless of what surrounds the object. In short, the text tells the model where the object is, and swapping backgrounds in the model's internal representation teaches it to ignore everything else. This costs a fraction of a forward pass, needs no segmentation masks, and never deletes features, so it avoids the failure where aggressive noise removal also strips out the object.

The opportunity sits between three lines of work that have not been joined. One line shows that background leaks into few-shot class summaries. The second shows that augmenting a network's internal features is cheap and effective. The third shows that attention from text to image regions can localize the object. Existing work either suppresses background without augmenting, augments without knowing what is background, or aligns text and image as whole vectors with no spatial decomposition.

Core Claim

Background invariance in few-shot adaptation can be taught directly. We train on internal token sequences in which the object stays fixed and the context varies. The object tokens are chosen by attention from the class-name text embedding to image patches. The swap happens at an intermediate transformer layer, so the remaining layers must re-integrate the new context. Concretely, let $z_i$ be the token for patch $i$ at layer $L$ of a frozen CLIP image encoder (CLIP is a model trained to match images with captions). Let $s_i$ be the cosine similarity between $z_i$, projected into the joint space, and the class text embedding, minus its similarity to a small set of generic "background" prompts. Let $b_{\pi(i)}$ be a patch token at the same position from a different image drawn from a pool of base-class images. The augmented token is

$$\tilde z_i = m_i z_i + (1 - m_i)\, b_{\pi(i)}, \qquad m_i = \sigma\!\left(\frac{s_i - \tau}{T}\right),$$

where $\sigma$ is the logistic function, $\tau$ is a threshold, and $T$ is a temperature. A lightweight adapter, meaning low-rank updates to the last few blocks plus the classifier head, is trained with cross-entropy on both original and swapped sequences. It also gets a consistency term that pulls the two predictions together. We expect the largest gains where background is most predictive of the label in the support set. We therefore predict gains that shrink with shot count and grow on scene-heavy or cluttered datasets.

Relation to Prior Work

Several papers establish that the gap is real. Eliminating Feature Ambiguity for Few-Shot Segmentation and Spot and Learn document that averaging support features pulls background into the prototype. A prototype here is the average feature vector used to represent a class.

Two families of methods attack this leakage by suppression. SEGA turns word embeddings into a channel mask on the prototype. CAN uses support–query correlation maps to highlight shared regions. TBS and CobNet mine and subtract background prototypes for segmentation. All of them filter background at inference but never train the model on varied backgrounds. The visual-only ones also break when object and background look alike.

NAPP is the closest combination of suppression and feature augmentation. It learns noise prompts that subtract task-irrelevant patterns and mixes same-class support features. It is unimodal, its mixing only interpolates observed features, and it reports that cancellation in too many blocks removes real signal. This negative result constrains our design: we replace background rather than subtract it.

Three papers supply methods we reuse. ISDA shows that feature-space augmentation is cheap, at 5–7% training overhead. It applies only at the penultimate layer, so earlier layers never see augmented states, and it perturbs along class covariances that are unreliable with five examples. AFA perturbs feature statistics adversarially but without any notion of which regions are background. Causality-inspired Latent Feature Augmentation splits features into object and style parts and shuffles the style part. This is the most similar mechanism, but its split is learned from image transformations alone, and it targets domain generalization rather than episodic few-shot learning.

On the cross-modal side, CCA unmixes CLIP features with a fixed ICA matrix and repairs the damaged text alignment with cross-attention, but performs no augmentation. FMA injects Gaussian noise along image-to-text flow trajectories but treats each image as one global vector with no spatial background model. Our proposal differs from both. It keeps CLIP's alignment intact because the backbone is frozen and tokens are only recombined. It also acts at the patch level, where background actually lives. Adaptive Cross-Modal Few-shot Learning and the fake-label findings around SEGA warn that text guidance misfires when the label is ambiguous. We borrow the gating idea of Gated Cross-Attention to address that risk, as described below.

Research Program

The first component is the mask. For each support image we compute $m_i$ from text-to-patch similarity at layer $L$. Mask errors are asymmetric: labeling object patches as background destroys signal, while the reverse only wastes augmentation. We therefore gate the text mask with a visual agreement score, which is the patch's correlation with the other support images of the same class in the style of CAN. The final mask is a learned convex combination of the two, and the gate can shift weight toward the visual cue when text similarity is flat. In 1-shot, where no other support images exist, the gate falls back to text alone. We also set $\tau$ conservatively and study the resulting precision–coverage tradeoff.

The second component is the background pool. Background tokens come from base-class training images, or in the pure few-shot setting from generic unlabeled images. They are drawn at matching spatial positions so positional structure survives. We compare three pool sources: same-dataset, cross-dataset, and support images of other classes in the episode. The last one directly breaks the correlation between each class and its own background.

The third component is the injection depth $L$. We sweep it across the ViT blocks. Very early swaps may be undone by later mixing. Very late swaps reduce to ISDA-like penultimate augmentation and should lose the benefit of re-integration. Finding where this curve peaks is itself a result about where background is encoded.

The fourth component is analysis. Using the same masks, we measure how much of the adapted model's prototype energy lies on background patches before and after training. We also measure the drop in accuracy when test backgrounds are replaced by other-class backgrounds.

Baselines

The naive reference points are zero-shot CLIP and a linear probe on frozen CLIP features. They show how much any adaptation helps and how background-sensitive the untouched model already is.

The strongest existing method on the multi-dataset CLIP suite is FMA (Exploring Cross-Modal Flows for Few-Shot Learning). It is the strongest competitor because it already combines cross-modal adaptation with feature-space noise augmentation, so it is exactly the "augment globally without spatial decomposition" alternative. It reports average accuracy on the Difficult Set (FGVC-Aircraft, DTD, EuroSAT, SUN397, StanfordCars) rising from 48.9% for standard CLIP to 68.9%. It also reports 16-shot FGVC-Aircraft on CLIP-LoRA improving from 54.7% to 57.8%. We will rerun FMA from its released configuration on our backbone and splits and treat our reproduction as the reference. Beating it means a higher Difficult Set average at matched shots and matched adapter parameter count. The gap must exceed the seed-to-seed standard deviation over at least three seeds. We also require a clearly smaller background-conflict gap.

CLIP-LoRA alone is the plain parameter-efficient fine-tuning baseline. The only reported number in the file is the 54.7% 16-shot FGVC-Aircraft figure quoted by FMA. CCA (Causal Disentanglement and Cross-Modal Alignment) is the disentangle-then-realign competitor. No per-dataset reported result was found on Lacuna for CCA, so we will run it ourselves. On episodic miniImageNet and CUB we include NAPP (NAPP), which reports 94.1% 5-way 1-shot miniImageNet accuracy and 85.23% 5-shot CropDiseases accuracy. We include SEGA, which reports 69.04% 1-shot miniImageNet and 84.57% on CUB, noting that it uses weaker backbones than ours. All methods get the same backbone, the same episodes (via Torchmeta for episodic benchmarks), and an equal hyperparameter search budget of 20 trials per method on validation splits.

The key ablations isolate each ingredient. Random-mask swapping replaces the same fraction of patches chosen uniformly at random, which tests whether text guidance matters or whether generic token noise does the work. Inverted-mask swapping replaces object patches and keeps background, as a sanity check that should hurt. Penultimate-only swapping sets $L$ to the last block and tests whether re-integration by later layers matters. Gaussian token noise on background patches uses the same mask with noise instead of swapping, and tests whether real background content beats synthetic perturbation. Text-only and visual-only masks test the gate.

Why not just…?

Why not just paste objects onto new backgrounds in pixel space? This is the most serious threat. Copy-paste with ground-truth masks directly teaches background invariance. It needs masks or an external segmenter, a full backbone pass per augmented view, and it introduces compositing artifacts the model can learn to detect. We will run it with the best available masks, from dataset annotations where they exist and an off-the-shelf segmenter otherwise, at an equal number of augmented views. If it matches our accuracy and background-conflict gap, the contribution shrinks to a mask-free, cheaper version of the same effect, and we will report it that way. The evidence from Causality-inspired Latent Feature Augmentation suggests that image-level transformations fail to break background–label associations. That paper's transformations were rotations and jitter, not compositing, so it does not settle the question.

Why not just use global feature noise, as FMA or ISDA do? Global noise perturbs object and background dimensions alike. With few shots, a covariance estimate is dominated by whatever background the support images share, so global perturbation keeps that dominant background direction in the prototype rather than suppressing it. The Gaussian-noise and random-mask ablations measure exactly this. If they match text-guided swapping, the spatial decomposition is unnecessary and we would say so.

Why not just learn the object/background split without text, as the causal augmentation work does? A split learned from a handful of images has no signal for which factor is the object when object and background co-vary perfectly in the support set, and in 1-shot they always do. Text supplies that missing signal. The visual-only mask ablation tests this. We expect it to close the gap at 16 shots and lose clearly at 1–2 shots. If it does not lose at low shots, the text component is not earning its place.

Why not just subtract the background, as NAPP or TBS do? NAPP itself reports that subtraction across many blocks strips real signal. Subtraction also leaves the model untrained on novel contexts. Swapping keeps the token distribution natural and exposes the adapter to context variation. A direct comparison against NAPP on miniImageNet and CropDiseases, plus our prototype-energy analysis, will show whether swapping retains more object signal.

Proposed Experiments

The main evaluation uses the 11-dataset CLIP few-shot suite from CCA and FMA at 1, 2, 4, 8, and 16 shots. We report the Difficult Set average separately, since FMA's reported numbers make it the sharpest comparison. For distribution shift we train on ImageNet few-shot and test on ImageNetV2 and ImageNet-Sketch, following CCA. For episodic transfer we train on miniImageNet and test on CUB, CropDiseases, EuroSAT, ISIC, and ChestX in 5-way 1- and 5-shot. We expect ChestX and ISIC to be hardest for text guidance because class names align poorly with image content there. Reporting those cases honestly is part of the test.

To measure background reliance directly, we build a background-conflict diagnostic from the evaluation sets. Test objects are recomposited onto backgrounds from other classes using masks, and we report the accuracy gap between original and recomposited images.

Four checks decide whether the idea works. First, text-guided swapping must beat random-mask swapping at 1 and 2 shots on the Difficult Set; otherwise the attention adds nothing. Second, the injection-depth sweep must show an interior optimum above the penultimate layer; otherwise re-integration is not the mechanism. Third, the background-conflict gap must shrink relative to FMA and CLIP-LoRA even at matched clean accuracy. Fourth, under a fake-label test, where class names are replaced by wrong or rare words, the gated mask must degrade gracefully toward the visual-only variant rather than collapse. The failure mode reported for SEGA- and AM3-style text guidance would then be contained. If pixel copy-paste matches us on all four, we will present the method as a cheaper, mask-free substitute and report training cost as the headline comparison.