Partitioned Sparse Dictionaries for Auditable Nuisance Isolation

Spurious correlations are a common way image models fail. A bird classifier that has only seen waterbirds on water learns "water" as much as "bird." Interpretability tools can now find such shortcuts inside a pretrained model. The current fix is to delete the shortcut. One line of work trains a sparse autoencoder on the model's embeddings and then projects the offending features out. A sparse autoencoder is a small network that rewrites a dense embedding as a sparse sum of mostly single-meaning features. The other line pretrains the model so that it ignores backgrounds altogether. Both have a blind spot. A sparse autoencoder trained only to reconstruct embeddings has no reason to split its features along the line between "object" and "background." A feature can quietly mean "waterbird-on-water." Deleting the "water" features then leaves background information hidden inside the "bird" features. No one can check how much remains, because the deleted part is gone.

We propose to train the sparse dictionary itself in two halves, signal and nuisance, and to keep both. Training uses image pairs that differ only in background or only in foreground. The signal half must not change when the background is swapped. The nuisance half must not change when the object is swapped. Recombining the signal code of one image with the nuisance code of another must reproduce the embedding of the rendered composite. The product is a classifier input built only from named object concepts. Next to it sits a named, inspectable inventory of what was routed away. Because nothing is thrown out, we can measure how much nuisance information leaked into the signal half, and scene recognition stays available from the nuisance half.

Core Claim

The claim is narrow and testable. When an ordinary sparse autoencoder is used for debiasing, it leaks because of feature absorption, meaning single features that mix object and context. Erasing features after training cannot repair this, because the mixing is inside individual features. Training with a partition and paired swaps resolves the mixing at the right boundary. This should show up in three ways. First, a nonlinear probe should recover the background from the signal half far less often than from the output of post-hoc projection. Second, worst-group accuracy should be at least as good as the post-hoc approach. Third, scene recognition should survive, which background-invariant pretraining loses.

If the same dictionary trained without the partition, then split afterwards, matches us on leakage, the claim is false.

Positioning Against Prior Work

The motivating evidence comes from three papers.

DIAL/DIAL+ finds spurious features in CLIP through a sparse autoencoder and removes them by orthogonal projection, without labels. It reports 69.18% worst-group accuracy on Waterbirds with ViT-L/14, compared with 31.93% zero-shot. It never checks what the projection left behind or what weak signal it destroyed.

ConceptScope sorts sparse-autoencoder features into target, context, and bias concepts using necessity and sufficiency scores. It remains an offline audit; it does not build an invariant representation. Its released code and pretrained autoencoders (jjho-choi/ConceptScope) will serve as our post-hoc pipeline.

BAP shows that in vision-language models a scene embedding is roughly the sum of a foreground embedding and a background embedding. It exploits this by pasting objects onto many backgrounds, reaching over 90% worst-group accuracy on Waterbirds. The cost is a collapse in scene recognition.

We borrow BAP's compositing as our source of paired supervision. Unlike BAP, we keep the background in its own partition instead of averaging it away.

The partitioning lineage supplies the structure.

  • AENIB splits a representation into a task code and a nuisance code. It requires the pair to reconstruct the input, makes the two codes independent, and demonstrates nuisance swapping. Both of its codes are dense and unnamed, and it relies on adversarial training, which is unstable.
  • DIVA uses class, domain, and residual subspaces. Its residual subspace acts as an uninterpretable catch-all.

We keep their insistence on sufficiency, meaning nothing is silently discarded. We replace their dense codes with sparse, nameable dictionary features, and we replace adversarial independence with explicit swap targets.

SpLiCE and EXPLAIN-R constrain the design. SpLiCE's text-word dictionary cannot name nuisances that have no short phrase. EXPLAIN-R shows that reconstructing embeddings by plain squared error hurts retrieval: COCO image-to-text R@1 is 38.12 for SpLiCE against 61.99 for EXPLAIN-R. We therefore learn the dictionary rather than fixing it, and we add EXPLAIN-R's similarity-matching loss.

Linear Adversarial Concept Erasure warns that linear erasure can still be undone by nonlinear probes. That is exactly why we measure leakage with nonlinear probes.

Research Program

We work on frozen CLIP image embeddings (ViT-B/16 and ViT-L/14). Each embedding $e$ is encoded into a sparse signal code $a_s$ and a sparse nuisance code $a_n$. These are decoded as $\hat e = D_s a_s + D_n a_n$, where $D_s$ and $D_n$ are separate dictionaries and sparsity uses a BatchTopK budget per half (BatchTopK Sparse Autoencoders).

Paired data come from compositing segmented foregrounds onto Places365 backgrounds. Suppose $x$ has foreground $f$ and background $b$, and $x'$ has foreground $f'$ and background $b'$. Let $x_{f b'}$ denote the rendered image with foreground $f$ on background $b'$. The training objective is

$$ \mathcal{L} = \mathcal{R}(e_x, \hat e_x) + \lambda_{\text{swap}}\,\mathcal{R}\big(e_{x_{fb'}},\, D_s a_s(x) + D_n a_n(x')\big) + \lambda_{\text{inv}}\big(\|a_s(x) - a_s(x_{fb'})\|_1 + \|a_n(x) - a_n(x_{f'b})\|_1\big). $$

Here $\mathcal{R}$ combines squared error with EXPLAIN-R's KL divergence between similarity distributions over a bank of reference captions. The second term is the cross-composite term. It is possible only because the true composite embedding can be rendered, and it is what forces each half to carry its own factor. The third term makes the signal code invariant to background swaps and the nuisance code invariant to object swaps.

Features in each half are named automatically. We do this by their highest-activating images and by their nearest caption concepts, and we score labels with ConceptScope's concept-prediction protocol. Downstream object classifiers read only $a_s$. Scene classifiers read only $a_n$.

The second phase handles nuisances that are not spatially separable, such as ISIC rulers and ink. These cannot be composited. Here we seed the nuisance half from ConceptScope's bias-concept scores and use DIAL+'s flip-based flagging to form weak swap pairs. This phase is explicitly exploratory: the paired-swap supervision is what the core claim rests on.

Baselines

The naive reference point is zero-shot CLIP with class prompts, at 31.93% Waterbirds worst-group accuracy and 42.21% on ISIC, as reported by DIAL. A linear probe on raw embeddings trained by ordinary empirical risk minimization serves as the trained naive point.

The strongest label-free competitor on the interpretability side is DIAL+. It reports 69.18% worst-group accuracy on Waterbirds with ViT-L/14, and 68.42% on ISIC for DIAL. We will rerun it on our backbones using a sparse autoencoder of the same width and sparsity budget as ours.

The strongest robustness number is BAP, reported at over 90% worst-group accuracy on Waterbirds under 100% spurious training correlation. We will rerun BAP using the same composited image pool and the same student compute. We found no reported scene-recognition number for BAP on Lacuna; the paper describes the collapse only qualitatively, so we will measure it ourselves.

The ablation that isolates our key ingredient is called post-hoc partition. It uses the identical autoencoder architecture and the identical composited data, but with only the reconstruction and sparsity terms. Its features are then split into signal and nuisance by ConceptScope scores, and DIAL-style projection is applied.

A dense two-head variant tests whether sparsity matters at all. It keeps our swap losses but drops the sparsity constraint, making it AENIB-like without adversaries.

All methods get the same embeddings, the same composite pool, and the same hyperparameter search budget of 20 configurations, with selection on a validation split that contains no minority-group images.

Beating the field means three things at once:

  • worst-group accuracy within one point of BAP or above it;
  • background recovery from the signal half by a two-layer probe at least 15 points closer to chance than post-hoc partition;
  • Places365 accuracy from the nuisance half within five points of raw CLIP.

Why not just…?

Why not just keep the raw embedding for scene tasks and a DIAL-projected copy for object tasks? This recovers scene recognition trivially, and we concede that point. It cannot address leakage. If absorbed features carry background inside "bird" features, the projected copy still leaks. The comparison against post-hoc partition, measured with nonlinear probes, settles whether that leakage is real. If post-hoc leakage turns out to be negligible, our method's advantage reduces to auditing convenience, and we would report it that way.

Why not just train a linear probe, or run group-balanced training, on frozen CLIP features using the composited images? This is a genuine threat on Waterbirds-style backgrounds, and it may match our worst-group accuracy. We run it as a baseline under the same budget. Our claimed advantages do not depend on beating it there. They are the named nuisance inventory and the transfer to CounterAnimal and NICO++. Those benchmarks contain real atypical contexts, not pasted ones, so a probe fit to composites may overfit compositing artifacts. If the probe also wins on those, the robustness half of our claim fails.

Why not just use BAP? BAP discards the background by design, which causes the scene-recognition collapse its authors report. BAP also gives no account of what was removed. The Places365 evaluation from the nuisance half is the direct test of the first problem. The concept-labeling evaluation addresses the second.

Why not just a dense partitioned autoencoder like AENIB? The dense two-head ablation answers this. If it matches us on robustness and leakage, sparsity buys only nameability. We would then test whether the names are useful, by checking whether they let annotators predict which test images a model will misclassify, rather than claiming a robustness benefit.

Proposed Experiments

The robustness evaluation uses Waterbirds, the ImageNet-9 Background Challenge, CounterAnimal, and NICO++. For each we report worst-group or atypical-context accuracy from $a_s$ alone.

We also report context retention, measured as Places365 scene accuracy from $a_n$, and fidelity, measured as ImageNet zero-shot accuracy and COCO R@1 from the recombined $\hat e$. The fidelity numbers are compared against SpLiCE's 51.33% ImageNet accuracy and 38.12 R@1 and EXPLAIN-R's 53.38% and 61.99, as reported by EXPLAIN-R.

The decisive check is leakage. For a held-out set of real (non-composited) images, a two-layer probe trained to predict background category is applied to three representations:

  • our signal half;
  • the output of post-hoc partition;
  • the output of DIAL+.

We also count absorbed features: features whose activation shifts by more than a threshold under background swaps while their top label names an object. If the joint objective does not reduce both counts relative to post-hoc partition, the core claim is rejected.

A swap audit then decodes $a_s$ from one image with $a_n$ from another and checks, by CLIP zero-shot classification, that the object and scene predictions follow their respective codes. Finally, the ISIC and CelebA extension tests whether weakly seeded swap pairs preserve the leakage gains when compositing is impossible. We report it separately so that a failure there does not obscure the main result.