Spending Human Judgments Where Architecture Search Is Blind

Image restoration networks, such as super-resolution models that upscale photos or denoisers trained to produce realistic texture, are now judged by whether people like the output, not by pixel error. Automatic architecture search, the practice of letting an algorithm pick a network's layers and connections, has become cheap for these tasks. Evaluation has not. Search methods rank candidates with pixel error (PSNR) or a learned perceptual metric, both of which a human panel is meant to stand in for. On outputs with generated texture those stand-ins are weak. PSNR and SSIM are anti-correlated with human judgment on generative super-resolution, and the best learned metric reaches only about 0.51 rank correlation with humans (PIPAL). A search that ranks thousands of candidates with such a metric may pick the metric's favorite rather than the one people prefer.

We propose to treat human judgment as the most expensive fidelity level in a multi-fidelity architecture search. Cheap estimators do the coarse work. Zero-cost scores at initialization and short-training perceptual metrics throw out bad candidates. A small, adaptively allocated budget of human paired comparisons then settles the ranking among the survivors. The intended result has two parts: an estimate of how much human effort a perceptual architecture search actually needs, and an allocation rule that recovers the human-preferred architecture with a small fraction of a dense panel's comparisons.

Core Claim

Both literatures report the same failure. In architecture search, zero-cost proxies are good at filtering out bad models but fail to discriminate among the top 10%. Their biases also depend on architecture: synflow favors large models, and snip and fisher overvalue networks without skip connections (Evaluating Efficient Performance Estimators). In quality assessment, learned metrics track humans on classical distortions but break on generated texture, partly because common pooling layers are not shift-invariant (PIPAL).

Our claim is that these failures stack. On a perceptual restoration search space, the errors of automated estimators should grow toward the top of the ranking. They should also correlate with specific architectural choices, such as the upsampling operator, downsampling, and receptive field. If so, three things follow. First, the errors can be corrected with a few human comparisons. Second, those comparisons should go to architectures, not individual images. Third, the comparisons should be concentrated at the frontier the cheap estimators cannot resolve.

What is new is the unit of subjective testing: architectures rather than images. The metric's prediction serves as a prior that human data debiases, and the objective is identifying the best architecture rather than calibrating a metric.

Research Program

Search space. We build a cell-based search space of super-resolution and denoising generators trained with a perceptual plus adversarial loss. Search choices cover upsampling operator, downsampling type (strided, max-pool, or blur-pooled), kernel sizes following the sliced superkernel idea of Superkernel NAS, and hierarchical multi-scale cells in the style of IR-NAS/HiNAS. Both of those works make search on dense high-resolution inputs affordable, but both optimize only PSNR. We reuse their memory-saving mechanics and change the objective. We sample roughly 120 architectures and train each with three seeds on a short schedule.

Human ground truth, collected once. To obtain human ground truth, we select the 40 architectures ranked highest by any automated estimator plus 10 random ones. Their outputs on PIPAL reference content are compared densely by crowd raters and converted to Elo scores, following PIPAL's protocol. With this dense table in hand, any allocation policy can be evaluated offline by replaying subsamples of it. Many policies can therefore be compared without new human studies.

Estimator stack. The stack has three tiers:

  1. Tier 0 (at initialization): zero-cost scores (synflow, jacob_cov, snip, and their vote from Zero-Cost Proxies for Lightweight NAS), plus parameter count and FLOPs.
  2. Tier 1 (after short training): automated scores from LPIPS with and without BlurPool, MUSIQ as a reference-free judge, HaarPSI, and PSNR.
  3. Tier 2 (humans): paired comparisons between two architectures' outputs on the same image.

We fit a Bradley–Terry model, which predicts the probability that one item is preferred over another from a latent score per item. Each architecture's latent score has a prior mean given by a Tier-1 metric ensemble plus a learned correction for each architectural operation. The correction captures how much the metric over- or under-rates, say, transposed convolutions. The next pair shown to a rater is the one whose outcome most reduces the posterior probability of misidentifying the best architecture. This adapts the particle-filter sampling of Adaptive Paired-Comparison, which applied it to videos, to a best-arm identification objective over architectures.

Analysis. For each tier we report rank correlation with human Elo overall and within the top decile. We also report how much of the metric–human disagreement the per-operation correction explains.

Baselines

Naive reference. Rank architectures by parameter count, then by PSNR after short training. Evaluating Efficient Performance Estimators shows parameter count often beats sophisticated zero-shot scores. PSNR is what IR-NAS and Superkernel NAS actually optimize. It uses the same trained candidates at zero human cost.

Strongest metric-only method. Rank every candidate by LPIPS with BlurPool. It is the best learned full-reference score reported on generative distortions, at 0.5135 SRCC on PIPAL GAN outputs versus 0.4862 without BlurPool (PIPAL). We also include an ensemble of all Tier-1 metrics, so the method is compared with the best automated combination rather than one weak metric. On the zero-cost side the strongest reported figure is the voting ensemble's 0.82 Spearman on NAS-Bench-201 (Zero-Cost Proxies). That number is for classification accuracy. There is no reported result on Lacuna for zero-cost proxies ranking perceptual restoration architectures, or for any metric ranking architectures against human preference; our dense table supplies the first.

Uniform human allocation. Spend the same human budget spread evenly over randomly chosen pairs of candidates. This is the direct competitor for the allocation rule.

Ablations. Two ablations isolate the key ingredients: adaptive allocation without the metric prior, and the metric prior without the per-operation correction.

Fairness and the bar to clear. All methods see identical trained candidates and seeds, and all human-using methods get identical budgets. The method wins if it reaches lower top-1 regret in human Elo at equal budget, or the same regret with at most half the comparisons, averaged over 50 replays.

Why not just…?

Why not just trust the best perceptual metric and skip humans entirely? At roughly 0.51 correlation on generated texture (PIPAL), a metric can rank far-apart candidates correctly and still pick the wrong winner among close ones. This alternative is a real threat, though. If Tier-1 top-decile correlation with human Elo turns out high in our space, humans add little, and we will report that as the finding.

Why not just take a known architecture and retrain it with a perceptual loss? This is the sharpest objection, and it echoes ProxIQA. ProxIQA gains 17.7% in MOS BD-rate (bitrate saved at equal human-rated quality) by changing the training signal of a fixed compression network. If architectures barely differ in human preference once the loss is perceptual, search is pointless. Our first check therefore compares human Elo variance across architectures with variance across seeds. It also includes HiNAS's published cell, retrained with our loss, as a candidate.

Why not just use zero-cost proxies or parameter count as the filter and stop there? The literature says they fail precisely at the top (Evaluating Efficient Performance Estimators). They have also never been measured on perceptual regression targets. Tier-0 correlations in our table test directly whether they keep or lose their classification behavior.

Why not just ask a vision-language model to judge outputs instead of people? Work on vision-language models scoring AR scenes documents where such judges stumble, and CLIPVQA shows CLIP features can score quality. We add a CLIP-based scorer as an extra Tier-1 metric, so it competes on the same terms as the others.

Proposed Experiments

Does architecture matter to humans? The first decisive check tests whether across-architecture variance in human Elo exceeds seed variance by a clear margin in the dense table. If it does not, the project reports a negative result about perceptual architecture search and stops.

Where do automated estimators fail? The second check measures whether rank correlation between each automated tier and human Elo drops from the full set to the top decile. It also asks whether the per-operation correction, fit on half the architectures, predicts metric–human disagreement on the held-out half. That test establishes whether metric bias is structured by architecture or is just noise.

Does adaptive allocation pay off? The third check plots top-1 regret and top-5 recall against human budget for all baselines, replaying from the dense table. Budgets run from 1% to 30% of the dense comparisons.

Does it hold up live? We run one fresh search with the chosen policy on Rain800 de-raining content (IR-NAS). A new rater pool confirms whether the selected architecture beats the one LPIPS with BlurPool selects. This guards against overfitting the policy to the collected table.