When a pre-trained image generator is used to audit a classifier, for example by synthesizing counterfactual images to find which attributes the classifier secretly relies on, the audit can only test attributes the generator is able to draw. If the generator never learned to render eyeglasses, the audit concludes nothing about eyeglasses, but nothing in the output says so. Silence about an attribute looks the same as a clean bill of health. This note proposes to make that silence visible.
The proposal uses a generator built from a mixture of experts: a network whose layers contain many specialist sub-networks, plus a small gating module (the "router") that sends each image patch to the few specialists that match it best. We argue that the router's internal signals form a cheap, per-patch map of where the generator is competent. Two signals matter. The first is how strongly a patch matches any specialist. The second is whether the specialist assignment flips abruptly as one moves through the generator's input space. Together they let an auditor separate three findings: "the classifier depends on this attribute," "the classifier does not depend on it," and "this attribute cannot be tested with this generator." The same router also says where to repair the generator, by adding one specialist that only flagged patches use.
Core Claim
In a diffusion transformer whose router assigns patches by cosine similarity to learned prototype vectors, as in ProMoE, the maximum prototype similarity of a patch's hidden state acts as a built-in coverage score. We expect it to fall when sampling is steered toward content outside the training support. The rate at which top-$k$ expert assignments change along a latent path acts as a scalable proxy for the "conceptual phase boundaries" that Hessian Geometry of Latent Space in Generative Models found. In that work, outputs change abruptly and the local Lipschitz constant diverges at these boundaries. Its method computes them from a free-energy Hessian and only on low-dimensional slices.
The new element is not a new architecture. It is the use of an existing routing mechanism as an instrument, together with a causal test that the instrument points at the right place. The causal test asks whether inserting one new expert, used only by low-coverage patches, repairs the gap more cheaply and with less collateral damage than adapting the whole generator.
Where This Sits in Prior Work
Three papers motivate the problem by documenting that generator-based diagnosis inherits the generator's blind spots, and none of them detects when it is happening. Discover the Unknown Biased Attribute of an Image Classifier optimizes a hyperplane in a GAN latent space to surface classifier biases. Counterfactual Explanations on Robust Perceptual Geodesics walks StyleGAN latent spaces along a metric pulled back from a robust classifier. Probing Perceptual Priors of MLLMs via Gibbs Sampling with Interpretable Generative Controls samples FLUX slider coordinates to read out a multimodal model's priors. All three rely on static, dense generators and give no signal for unreachable attributes. Editing work such as SeFFeC, StyleMC and LACE constrains the design. These papers show that no amount of steering or energy guidance produces off-manifold attributes, so the proposal must diagnose and then add capacity, not steer harder.
Two papers supply methods. ProMoE supplies the prototype router and its Routing Contrastive Loss, which pushes prototypes of different experts apart. We reuse both unchanged and ask a question ProMoE did not ask, namely what the routing reveals about coverage. The Hessian geometry paper supplies ground-truth phase boundaries on low-dimensional slices, which we use to calibrate the cheap routing signal.
Two further papers supply the repair side. CARE-Edit routes diffusion tokens among fixed, hand-designed adapter experts. Anomaly-Preference Image Generation varies LoRA rank only as a function of timestep to avoid overfitting backgrounds. We instead let the router's own coverage score decide which patches get new capacity.
Research Program
We first build controlled support holes, so that ground truth about unreachability exists. We train ProMoE-style class- or attribute-conditional diffusion transformers at base scale on FFHQ, filtered with an attribute classifier to remove one attribute entirely (for example eyeglasses or heavy beards). We do the same on ImageNet-1K with a few classes removed. Matched dense transformers and unfiltered MoE models serve as controls.
Second, we define the two signals. For patch $i$ at layer $\ell$ and timestep $t$, the coverage score is
$$c_{i,\ell,t} = \max_{e} \cos\left(h_{i,\ell,t},\, p_{e,\ell}\right),$$
where $h_{i,\ell,t}$ is the patch's hidden state and $p_{e,\ell}$ is the prototype of expert $e$. The boundary score along a latent path is the fraction of patches whose top-$k$ expert set changes between adjacent path points, normalized by step length. We aggregate both scores over layers and timesteps with a small set of fixed rules, chosen on held-in data only.
Third, we calibrate the boundary score against Hessian free-energy boundaries on two-dimensional latent slices, where that method is tractable. If they agree there, we use the routing score where the Hessian is not computable.
Fourth, we wire both signals into the hyperplane-search audit and the geodesic counterfactual audit. Each audit then outputs a three-way verdict per attribute: dependent, not dependent, or untestable. Fifth, for attributes flagged untestable, we add one expert and one prototype to selected MoE layers. We train them on a few dozen images of the missing attribute, with the router allowed to send only low-coverage patches to the new expert.
Baselines
The naive reference is the audit as published, with no reachability check. Every attribute the search fails to move the classifier with is reported as "not dependent." Beating it means correctly relabeling planted, unreachable dependencies as untestable while leaving correct verdicts intact.
For detecting unreachable targets, the strongest simple competitor is a nearest-neighbor distance from generated images to the training set in a frozen self-supervised feature space. It targets exactly the question of whether an image lies near the training support, and it needs no special architecture. A second competitor is a text-image similarity check of whether the target attribute actually appeared in the output. A third is post-hoc k-means prototypes fitted to the hidden states of the matched dense transformer. This third one is the key ablation, since it isolates whether training-time routing matters or any clustering of activations would do.
All detectors receive the same generated samples, the same held-in data for choosing thresholds, and the same number of tuning trials. They are scored by area under the ROC curve for flagging steering targets inside the held-out region. Beating them means a higher area under the curve with at least comparable false-flag rates on reachable attributes.
For phase boundaries, the competitor is finite-difference perceptual distance (LPIPS) between consecutive path images, which estimates the local Lipschitz constant directly. Ground truth comes from the Hessian method; the file gives no reported numerical boundary-recovery result for it, only that it achieved lower RMSE than PCA-VAE on Ising and TASEP, with no number found on Lacuna. Beating finite differences means matching their localization while also attributing boundaries to image regions and layers at no extra sampling cost.
For generation quality, the reference model is ProMoE. It reports an FID of 2.79 on ImageNet for ProMoE-L-Flow, against 3.56 for dense DiT-L ProMoE. Our base-scale models will not match this, but our MoE must stay within the paper's reported gap over its matched dense model, or the instrument comes from a weakened generator.
For counterfactual quality inside the audit, the reference is PCG's R-FID of 9.1 on AFHQ versus 37.8 for RSGD PCG. We rerun PCG on our generators instead of comparing across backbones.
For repair, the competitors are a LoRA adapter trained on the same few images and the same step budget, and that LoRA with timestep-dependent rank as in Anomaly-Preference Image Generation. Beating them means equal or better attribute success on the target together with smaller FID drift on unrelated FFHQ samples.
Why not just…?
Why not just check the output with a text-image model? A similarity check can confirm an attribute appeared, but it has its own vocabulary limits. It cannot say whether a failure came from the generator's support or from a weak steering run. It also gives no location inside the generator to repair. It remains a real competitor for the binary flag, and the detection experiment compares the two directly.
Why not just measure distance to training data in a frozen feature space? This is the most serious threat for detection alone. If it matches routing coverage in area under the curve, we will report that, and the remaining contribution shrinks to the repair step. Nearest-neighbor distance is image-level and says nothing about which patches or layers lack capacity. Routing coverage is per patch, which is what targeted expert insertion needs. The repair experiment tests whether that localization actually pays off.
Why not just cluster a dense model's activations after training? If post-hoc prototypes on a dense transformer give the same coverage and boundary signals, then mixtures of experts are unnecessary and the honest conclusion is "use activation clustering." We expect the Routing Contrastive Loss and the fact that routing is causal in the forward pass to produce sharper drops in coverage. The matched-dense ablation decides this.
Why not just fine-tune the generator on the missing attribute? Fine-tuning fixes the gap only once someone already knows the gap exists, which is the problem being solved. Anomaly-Preference Image Generation also documents background overfitting from fixed adapters on scarce data. The repair comparison measures whether routing-gated insertion avoids this.
Decisive Checks
The program succeeds if four things hold. Coverage scores separate held-out from held-in targets better than nearest-neighbor distance and the dense-clustering ablation. Routing-flip locations agree with Hessian boundaries on two-dimensional slices. The augmented audits recover planted classifier dependence on a removed attribute as "untestable" rather than "absent." Router-gated expert insertion turns those verdicts into correct "dependent" findings with less FID drift than LoRA.
The program fails cleanly in either of two cases. If dense activation clustering matches routing on every check, conditional computation adds nothing here. If coverage does not drop for the removed FFHQ attribute, the router is not tracking support at all.
References
- Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance
- Hessian Geometry of Latent Space in Generative Models
- Discover the Unknown Biased Attribute of an Image Classifier
- Counterfactual Explanations on Robust Perceptual Geodesics
- Probing Perceptual Priors of MLLMs via Gibbs Sampling with Interpretable Generative Controls
- UP-FacE: User-predictable Fine-grained Face Shape Editing
- StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulation
- Controllable and Compositional Generation with Latent-Space Energy-Based Models
- CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing
- Anomaly-Preference Image Generation