Learning Where the Generator Lies: A Held-Out-Class Critic for Budgeted Zero-Shot Feature Synthesis

A common way to recognize categories a model has never seen is to train a generator on the seen categories. The generator learns to turn a class description, such as a list of attributes ("has wings", "is striped"), into plausible image features. It then invents features for the unseen categories, and an ordinary classifier is trained on them. The weak point is that the generator was fitted only to seen classes. Its errors on unseen classes are exactly the "domain shift" that zero-shot learning suffers from, and nothing in standard training measures those errors. Current practice compensates by generating large piles of synthetic features. Generating more mostly adds redundant and misleading samples: they slow training and can lower accuracy (Synthetic Sample Selection for Generalized Zero-Shot Learning).

We propose to learn a model of the generator's own failures on classes it was not trained on, and then spend a fixed synthesis budget only where that model says the samples are trustworthy and useful. We repeatedly hide a subset of the seen classes, train the generator without them, and train an adversarial critic to tell the generator's features for the hidden classes apart from their real features. A critic trained this way learns what "generator output for a class it never saw" looks like when it goes wrong. At deployment, it filters and allocates synthetic features for the true unseen classes without looking at any test data.

Core Claim

Our claim is that a critic trained on simulated unseen-class episodes gives a better per-sample signal of synthetic-feature quality than the signals used today. The signals used today are the generator's own critic, which only ever compared real and fake seen-class features, and seen-class validation accuracy, which is SPOT's reinforcement-learning reward. With a better signal, a generalized zero-shot classifier can match the accuracy of a full synthetic set with a small fraction of the samples, or beat it at equal budget.

The novelty is specific. The discriminator is trained on a controlled proxy of the seen-to-unseen shift. Its job is to curate and budget synthetic data inductively, meaning with no access to unlabeled test images. It is not used to realign the generator to test data, which is the transductive approach.

Formally, write $G_{-k}$ for the generator trained on all seen classes except a held-out set $H_k$, $a_c$ for the attribute vector of class $c$, and $x$ for a real visual feature. The shift critic $C$ is trained across episodes $k$ with a standard adversarial objective:

$$\max_C \; \sum_k \Big( \mathbb{E}_{c \in H_k,\, x \sim P(x \mid c)} \log C(x, a_c) + \mathbb{E}_{c \in H_k,\, \hat x \sim G_{-k}(a_c)} \log\big(1 - C(\hat x, a_c)\big) \Big)$$

At deployment, $C(\hat x, a_u)$ scores features produced for each unseen class $u$ by the full generator. The average score per class tells us how much to trust the generator on that class.

Research Program

The program has three parts.

Building the critic. We build episodic hold-outs over the seen classes of standard attribute benchmarks. We cheapen them by warm-starting each $G_{-k}$ from a shared generator and fine-tuning briefly, so that critic training costs a small multiple of ordinary generator training. The base generators are feature generators of the kind used in Gradient Matching Generative Networks for Zero-Shot Learning and Free: Feature refinement for generalized zero-shot learning, with f-VAEGAN as the reference generator.

Using the critic. It is used in two ways. First, per-sample filtering keeps high-scoring synthetic features. A diversity constraint prevents the filter from collapsing onto a few "safe" points near the class mean. Second, per-class budget allocation gives more samples to unseen classes where the critic's scores are spread widely, and fewer where the generator is either confidently good or hopelessly off.

Analysis. We test whether critic scores computed without test data predict the real per-class failure of the generator. This is the scientific question underneath the method.

Prior work plays three roles here.

Work that motivates the proposal. Synthetic Sample Selection for Generalized Zero-Shot Learning (SPOT) shows that pruning synthetic features helps. It scores features with a Transformer and trains a Proximal Policy Optimization (PPO) agent rewarded by seen-class validation accuracy. That reward never observes an unseen-like class, which our episodic critic does. A Generative Framework for Zero-Shot Learning with Adversarial Domain Adaptation shows that adversarially aligning synthetic unseen features to real ones helps a lot. It reaches 78.6% on AWA2, but it requires unlabeled unseen test features and a CycleGAN mapping. We keep the adversarial signal and drop the test-data dependency by simulating the shift within the seen classes.

Work that supplies the method. Adversarial Learning for Zero-Shot Domain Adaptation treats domain shift as something learned on one task and transferred to another. Its limitation is that it needs a paired "irrelevant task" with matching shift. Our held-out seen classes serve as a naturally matched source of the same kind of shift, namely seen-to-unseen generator error.

Work that constrains the design. Addressing target shift in zero-shot learning using grouped adversarial learning reports that adversarial losses are sensitive to their weighting. Our critic is therefore never backpropagated into the generator in the main method; it only scores. Evolving Semantic Prototype Improves Generative Zero-Shot Learning (DSP) argues that noisy attributes are the root of the shift. That is complementary to our approach and a real confound, so we test the two in combination.

The overhead argument sharpens in the pixel regime. There, each sample is expensive: 62 hours on 8 GPUs in Zero-shot Generation of Training Data with Denoising Diffusion Probabilistic Model for Handwritten Chinese Character Recognition, and diffusion-generated corpora in Can Synthetic Images Serve as Effective and Efficient Class Prototypes? and Attributed Synthetic Data Generation for Zero-shot Image Classification. Per-sample filtering in that regime saves only classifier training. The per-class allocation half of the method saves generation itself. We therefore include a secondary study that allocates a Stable Diffusion image budget across CUB unseen classes using critic scores computed on a small pilot batch.

Baselines

Naive reference. The do-nothing baseline is the f-VAEGAN generator with a generalized zero-shot classifier trained on the full default synthetic set. The file reports its harmonic mean on CUB as 52.9% (Evolving Semantic Prototype Improves Generative Zero-Shot Learning). The harmonic mean of seen and unseen accuracy is the standard generalized zero-shot metric.

Random subsampling at matched budget. This is the same generator, keeping a uniformly random subset of $B$ features per unseen class. It tests whether any selection signal matters at all.

Strongest selection competitor: SPOT. SPOT answers the same question with a different signal: which synthetic features to keep. We run it on the identical generator checkpoints, the same features, and the same final classifier. Its tuning budget in GPU-hours is matched to our episodic critic's training cost. The file reports gains of up to 4.1% top-1 on CUB for WGAN and f-VAEGAN plus SPOT; no absolute SPOT harmonic-mean figure was found on Lacuna. Beating SPOT means a higher harmonic mean at every matched budget on at least three of AWA2, CUB, SUN and FLO, with lower or equal total compute.

Strongest inductive accuracy reference: f-VAEGAN plus DSP. The file reports it at 67.4% harmonic mean on CUB. We treat it as a combinable method rather than a rival and report DSP with and without our critic.

Absolute reference points. We cite CoAR-ZSL's reported 74.0% generalized harmonic mean on CUB with ResNet-101 (Boosting Zero-Shot Learning via Contrastive Optimization of Attribute Representations) as the non-generative ceiling. We cite the transductive method of A Generative Framework for Zero-Shot Learning with Adversarial Domain Adaptation as an upper bound that sees test data. We do not claim to beat either in absolute terms.

Key ablation. The ablation that isolates our ingredient replaces the episodic shift critic with the generator's own seen-class adversarial critic. Everything else stays the same: filtering rule, diversity constraint and budget. If this ablation matches our method, the held-out episodes add nothing and the proposal fails.

Why not just…?

Why not just generate fewer samples at random? If synthetic features are redundant, random thinning might recover most of the benefit. SPOT's reported gains over unfiltered sets suggest the choice of which samples to keep matters, but they do not show that it beats random thinning at equal count. Our accuracy-versus-budget curves settle this directly. If random subsampling tracks our curve within noise, we will report that selection is unnecessary in feature space and restrict the claim to per-class allocation.

Why not just use the generator's existing critic, or keep samples near the class mean? These are the cheapest selection rules and a real threat. The generator's critic only ever compared real and fake seen-class features, so there is no reason its scores reflect errors that appear only under the seen-to-unseen shift. Mean-proximity filtering is known to collapse intra-class variance. Can Synthetic Images Serve as Effective and Efficient Class Prototypes? shows that averaging to a single prototype gives only modest ImageNet gains of +1.8% to +3.1%. Both rules are run as ablations at matched budgets. The decisive number is whether the episodic critic beats the seen critic on the unseen accuracy component specifically.

Why not just use SPOT's reinforcement-learning selector? SPOT is the closest existing method. Its reward is seen-class validation accuracy, which is a proxy for generalization that cannot see unseen-like errors, and PPO adds training instability. Our critic replaces the reward with direct evidence from simulated unseen classes. The head-to-head at matched compute will show whether that evidence is worth more than a learned policy. The two can also be combined by using critic scores as SPOT's reward, and we will test that combination if the head-to-head is close.

Why not just go transductive and align to test features? When unlabeled test data are available, CycleGAN alignment is strong (A Generative Framework for Zero-Shot Learning with Adversarial Domain Adaptation). The inductive setting forbids it, and transductive realignment prevents single-instance inference (Universal Prototype Transport for Zero-Shot Action Recognition and Localization). We report the transductive number only as an upper bound. The gap we close toward it is the quantity of interest.

Proposed Experiments

Main benchmarks. We use AWA2, CUB, SUN and FLO with standard generalized zero-shot splits and ResNet-101 features, plus the correlation-shifted AWA2-CS split from Addressing target shift in zero-shot learning using grouped adversarial learning as a stress test. On AWA2-CS, ALE reports 25.3% and gAL 42.5%. For every method we plot harmonic mean, seen accuracy and unseen accuracy against synthetic budgets from 1% to 100% of the default set size. We also report wall-clock time for generation, critic or selector training, and classifier training, over five seeds.

First decisive check: does the critic see the true shift? Using real unseen features only for evaluation, we compute each unseen class's accuracy drop relative to seen classes and correlate it with the critic's per-class score, which was computed without any test data. If the episodic critic's rank correlation is not clearly higher than the seen critic's, the central claim is false.

Second decisive check: the budget claim. Our method should reach the full-set harmonic mean at no more than a quarter of the samples on at least three datasets, and beat SPOT at equal compute.

Third check: does the gain survive DSP? We apply our method on top of DSP. If our gain vanishes, we conclude that the critic was only detecting attribute noise, which DSP already fixes.

Pixel-regime study. On CUB we allocate a fixed Stable Diffusion image budget across unseen classes, either uniformly or by critic score. The images use attribute-based prompts in the style of Attributed Synthetic Data Generation for Zero-shot Image Classification. The student is trained with L2 feature distillation to avoid the spurious-shortcut failure reported in Feature Distillation Improves Zero-Shot Transfer from Synthetic Images. We measure accuracy per generated image. This tests whether the critic reduces actual generation overhead rather than only downstream training cost.