Refining Attribute Programs on Seen Classes to Recognize Unseen Ones

Zero-shot classification asks a model to recognize categories it has never seen labeled examples of, using only a description of each new category, such as "a small bird with a yellow crown and black wing bars." Current systems compare an image embedding with a text embedding and pick the closest class. When that comparison goes wrong, nobody can say which part of the description failed, and nothing can be fixed. This proposal makes each class description an executable program instead. The program is built from small Python functions, and each function checks one visual attribute ("has wing bars", "crown is yellow") by calling a vision-language model on image regions. A language model then repeatedly edits these attribute functions, as in code-debugging loops, using classification errors on classes that do have labels. An edit is kept only if it improves accuracy on a separate set of labeled classes that the editing process never saw. That acceptance rule mimics transfer to unseen classes.

The bet is that the attribute library sits at the right level for repair. A class can be unseen while its attributes are not: a wing-bar detector repaired on seen warblers is reused, unchanged, inside the program for an unseen vireo. Code-refinement loops have so far been judged on the same task they were repaired on. Here we ask whether execution-grounded repair made on one set of classes improves recognition of a disjoint set. That is the question zero-shot learning cares about, and it is also where such loops are most likely to overfit.

Core Claim

Iterative refinement of a shared library of attribute-checking functions, using seen-class labels as the execution signal, can raise unseen-class accuracy on attribute benchmarks. This requires one condition: edits are accepted by measuring them on held-out "pseudo-unseen" seen classes, not on the classes that produced the error. Without that condition, we predict the loop will raise seen-class accuracy and worsen the generalized zero-shot setting, where seen and unseen classes compete at test time and seen-class bias already dominates. We also claim that editing code, which can express conjunctions, region-specific checks, and conditional fallbacks, beats editing the plain-text attribute descriptions under the same budget. If the second claim fails, the proposal reduces to "refine descriptions with cross-class validation," and we will report that outcome.

Relation to Prior Work

The closest precursor is ViperGPT. It has a code model compose vision modules (GLIP detection, BLIP-2 region queries) into Python programs and reports strong zero-shot results, such as 72.0% IoU on RefCOCO. It generates each program once, with no feedback, and was never applied to zero-shot classification benchmarks such as CUB or AWA2 or to the seen/unseen trade-off. We reuse its idea of composing perception calls in code. We differ in three ways: our library persists across classes, it is repaired iteratively, and it is scored under zero-shot class splits.

The repair mechanism comes from three sources. LearnAct shows that agents can rewrite a persistent library of Python functions from failure cases. It reaches 72.2% success on AlfWorld versus 54.7% for ReAct+Reflexion, and the agent preferred code edits over natural-language notes 90% of the time. CAAFE keeps LLM-written feature code only if cross-validated performance improves. POLO uses a separate decision agent that rejects edits whose measured benefit does not hold up. All three accept edits on the task distribution they will be tested on. Our acceptance set is deliberately disjoint at the class level. GAZP is the nearest precedent for using execution agreement to adapt to an unseen domain (new SQL schemas). It does this offline, and only for SQL.

Two papers constrain the design. CodeTransOcean finds that self-debugging gains flatten quickly: the debugging success rate rises from 48.57% to only 52.57% after three rounds. We therefore budget few rounds and report curves over rounds. Perry et al. show that loops closed only by the model's own judgment reinforce errors while users stay confident in them. For that reason, every accept or reject decision in our loop is made by labeled images, never by the language model.

On the zero-shot side, three papers improve the matching between image regions and descriptions with fixed rules over embeddings. LaZSL matches image patches to GPT-written attributes with optimal transport, a minimum-cost assignment between the two sets. BiFTA prunes redundant image crops and redundant descriptions. GRAIN pretrains region-level alignment. None of them inspects errors and rewrites the matching procedure. BiFTA's drop on Oxford Pets also warns that a fixed multi-crop rule can hurt on single-subject images, which is a case our programs can branch on. Trained methods such as BFM fight seen-class bias with a calibration loss. That is the effect our cross-class acceptance rule is meant to provide without any training.

Research Program

Each class $c$ gets a program $P_c$, initialized from the dataset's class-attribute table. CUB, AWA2, and SUN all ship one. The program combines calls to library functions $f_1,\dots,f_K$, one per attribute. Each $f_k$ is initially a one-line call that scores a CLIP text prompt against the whole image. Its body can later be edited to crop a region, query a detector for a part, combine scores, or fall back conditionally. The class score is the program's output, and prediction takes the highest-scoring class.

Seen classes are split into a repair set $R$ and an acceptance set $A$, rotated across folds. In each round, the system collects confusions on images from $R$ and finds the attribute functions whose outputs disagree with the class-attribute table on those images. It then asks a language model to rewrite one function, sampling several candidate variants as LearnAct does, and including runtime errors in the context when they occur. A candidate replaces the current function only if accuracy over classes in $A$ improves by more than a noise threshold estimated from bootstrap resampling. If not, the library rolls back. Unseen-class programs are never edited directly. They benefit only through shared functions.

We will study three questions. First, does cross-class acceptance prevent the loop from inflating seen-class bias? We measure this through the harmonic mean of seen and unseen accuracy. Second, which kinds of edits transfer: region crops, conjunctions, or recalibrated thresholds? Third, how quickly do gains flatten across rounds, compared with the plateau CodeTransOcean reports for code?

Baselines

The naive reference point is plain CLIP zero-shot classification with "a photo of a {class}" prompts on the same backbone. Next to it we run CLIP scoring with the initial, unedited attribute programs. That isolates what writing classes as programs contributes before any refinement.

The strongest training-free competitor is LaZSL. It uses the same kind of LLM-written attributes and local regions, and its optimal-transport matching is a strong fixed answer to the problem our edited functions try to solve. The file reports only aggregate results: +2.0% average over DCLIP across five datasets with ViT-B/32, and 60.9% average on ImageNet shift variants. No reported per-dataset CUB result was found on Lacuna for LaZSL. We will therefore rerun it with the same backbone, crops, and attribute sets.

The strongest reported trained numbers come from BFM. It reports 71.9% top-1 accuracy on CUB, 62.8% on SUN, and 69.3% on AWA2 in classic zero-shot learning, and harmonic means of 61.6% (CUB), 33.2% (SUN), and 66.6% (AWA2) in the generalized setting. It uses ResNet-101 features rather than CLIP, so the comparison mixes backbones. We will report it as a reference, not as a matched comparison. Beating BFM's CUB harmonic mean of 61.6% under the Xian et al. proposed splits would be a strong result for a training-free method.

The decisive baseline runs the identical loop but lets the language model rewrite only the text description of each attribute, not code. It uses the same LLM, the same number of candidate edits and image evaluations, and the same cross-class acceptance rule. The key ablation keeps code editing but accepts edits on the repair classes themselves, which is the usual practice in CAAFE- or POLO-style loops. A traceback-only variant fixes runtime errors and nothing else, as a minimal debugging addition to ViperGPT, and shows how much of the gain comes from visual feedback rather than error handling.

The proposal counts as successful if it beats the description-editing loop and the same-class-acceptance ablation on unseen accuracy and harmonic mean on at least two of the three benchmarks. Differences must exceed the variation across class-split folds.

Why not just…?

Why not just refine the text descriptions instead of code? This is the real threat. LearnAct's 90% preference for code edits is suggestive, but it comes from planning tasks, not perception. The matched description-editing loop settles the question. If it matches code editing, we conclude that cross-class acceptance is the contribution and that code adds nothing. We will report that result.

Why not just use LaZSL or BiFTA with better attributes? Both apply one fixed matching rule to every class, and the BiFTA paper itself reports cases (Oxford Pets) where its fixed crop rule hurts. Our per-attribute breakdown of errors will show whether LaZSL's mistakes concentrate on attributes that our edited functions handle with region- or condition-specific logic. If they do not, the extra machinery is unjustified.

Why not just fine-tune CLIP on seen classes with a calibration loss, as BFM does? Fine-tuning is a strong option and may win on raw harmonic mean. It changes the backbone and gives no record of what changed, whereas our edits are readable and attributable to specific attributes. We will run a seen-class linear probe with calibrated seen/unseen scores at the same label budget. If it dominates on the generalized metric, our claim narrows to interpretability at comparable accuracy.

Proposed Experiments

We will run CUB, AWA2, and SUN under the proposed splits of Xian et al. and report per-class top-1 accuracy for classic zero-shot learning plus seen accuracy, unseen accuracy, and their harmonic mean for the generalized setting. Results are averaged over three rotations of the repair and acceptance split.

CLIP has likely seen these bird and animal names during pretraining. We therefore also evaluate on GRAIN's Products-2023 (27 classes released after 2023), using LLM-generated attribute tables in place of curated ones. This tests whether repaired functions transfer when the class names carry no memorized signal.

The decisive check is the gap in unseen accuracy between cross-class acceptance and same-class acceptance. If refinement helps only when the acceptance classes look like the test classes, the transfer claim fails. A second check plots accuracy against refinement rounds, to see whether gains flatten as early as they do in CodeTransOcean. A third check examines the 20 most-edited functions and confirms that each edit's gain holds up on unseen classes, attribute by attribute.