The Map as Instrument: A Metrology for Expert Judgments on Embedding Projections

When researchers want to understand what a protein language model or a large generative model has actually learned, they usually do the same thing: they compress the model's internal vector representations of thousands or millions of items into a two-dimensional picture and look at it. These pictures — made by algorithms like t-SNE or UMAP, which place similar items near each other and dissimilar ones far apart — are treated as a free, informal window into the model. But two bodies of evidence say this window is anything but reliable. First, the pictures themselves are unstable: the same underlying vectors yield visibly different layouts depending on hyperparameters and random initialization, and distances between clusters can be optimization artifacts rather than real semantic structure. Second, the human evaluation literature has documented that unstandardized visual judgment is among the least reproducible measurement procedures we have: fewer than 5% of human evaluations in generative NLP are consistently repeatable, and a single rating change can collapse a standard agreement statistic from 0.7 to 0.24 (ConSiDERS-The-Human Evaluation Framework).

This proposal treats the embedding map not as a picture but as a measurement instrument, and asks a question nobody has asked: when a domain expert looks at a projected embedding space and makes a diagnostic judgment, how much of the resulting measurement is about the data, how much is about the expert, and how much is an artifact of arbitrary projection choices? We answer this with a designed experiment that crosses experts with projection settings against tasks that have gold answers, quantifies each variance component with metrological tools, and uses the findings to build the first reproducibility protocol for visual latent-space inspection.

Core Claim

The claim is specific and falsifiable: a substantial, isolable fraction of the disagreement in expert judgments made from embedding maps is attributable to the projection pipeline (algorithm choice, hyperparameters, random seed) rather than to the data or the experts; this fraction can be measured, and a calibration protocol — frozen layouts, planted reference points, and placebo controls — can reduce it enough that expert visual inspection becomes a defensible, reportable measurement rather than an anecdote. Nothing in the existing literature does this. Semantic Concept Spaces built an interactive interface where experts refine topic models by manipulating t-SNE maps of word embeddings, and reported a 331% gain in topic distinctiveness — but with a handful of experts, no inter-rater reliability, and no variance tracking, so we cannot know whether another expert panel would have refined anything similarly. Protein language models expose viral mimicry and immune escape used t-SNE projections of protein language model embeddings to visually audit classification errors, uncovering a striking asymmetry (9.48% of viral proteins misclassified as human versus 1.87% of human proteins misclassified as viral) — but the visualization step was purely qualitative and post-hoc. And Quantified Reproducibility Assessment of NLP Results built exactly the metrological machinery we need (a corrected coefficient of variation, $CV^*$) and even showed that embedding-based automated systems are less reproducible than syntactic ones ($CV^* \approx 18.2$ versus $\approx 3.8$) — but never opened the black box to ask why, and never touched visualization. Our proposal is the intersection those papers leave vacant: apply formal metrology to the act of looking at a map.

Research Program

The program has three components. First, we construct a task suite in which expert visual judgments have verifiable gold answers, drawn from two benchmarks in the file. The SwissProt host–virus corpus of Ofer and Linial provides one: protein language model embeddings of human and viral proteins, with known misclassification labels, where the gold diagnostic finding is the asymmetric error structure itself. The Cao wet-lab dataset behind ProtDBench provides the other: over 478,000 experimentally confirmed non-binders and 1,385 confirmed binders across ten targets, where the gold answer for any triage judgment is the wet-lab label. Second, we run an instrument-variation experiment: a crossed design in which each expert performs the same diagnostic task on multiple renderings of the same underlying embeddings, varying the projection algorithm (t-SNE, UMAP, MDS, PCA), its hyperparameters (perplexity, number of neighbors), and the random seed. Third, we decompose the variance in expert judgments and design a stabilization protocol.

The metrological core reuses the Quantified Reproducibility Assessment framework of Belz et al., which measures reproducibility as a small-sample corrected coefficient of variation,

$$CV^* = \left( \frac{s^*}{\mu} \right) \left( 1 + \frac{1}{4n} \right),$$

where $s^*$ is the unbiased sample standard deviation of replicated measurements, $\mu$ is their mean, and $n$ is the number of replications. Because $CV^*$ requires a non-negative ratio scale — a known limitation when applied to Likert ratings — we follow ConSiDERS and decompose expert judgments into binary atomic decisions (e.g., "this marked region contains model errors: yes/no"), which sit cleanly on a ratio scale. The novel step is the variance decomposition. We model each binary judgment as arising from four sources and estimate them with crossed random effects:

$$\sigma^2_{\text{total}} = \sigma^2_{\text{projection}} + \sigma^2_{\text{expert}} + \sigma^2_{\text{item}} + \sigma^2_{\text{residual}}.$$

The quantity $\sigma^2_{\text{projection}}$ — the share of judgment variance that disappears when you fix the map and re-run the expert — is the paper's central number. If it is near zero, the field's informal practice is vindicated and the maps can be trusted as stable instruments. If it is large, every qualitative t-SNE audit in the literature inherits an unreported error bar.

The stabilization protocol adapts two ideas from standardized human evaluation. From GENIE, which maintains standard errors below 1.77% on sentence-level tasks by planting 5% positive and 5% permuted negative controls among crowdworker items, we borrow planted calibration points: known-error and known-clean items embedded in the map at fixed positions, so experts can be scored and filtered. From the reproducibility audit of Non-Repeatable Experiments, which showed most published human evaluations omit the exact stimuli and instructions, we borrow full artifact release: the projection code, seeds, screenshots, instructions, and raw judgments ship with every result.

Baselines

The naive reference point is current practice: a single default t-SNE map rendered once, inspected qualitatively, and reported in prose — exactly how Ofer and Linial used their projection to audit misclassified proteins. No reported result found on Lacuna for the accuracy or repeatability of that inspection step itself; their quantitative results (ROC AUC of 99.7%, accuracy of about 97%) describe the classifier, not the visual audit. We will run this condition formally — same experts, same items, one arbitrary map, no protocol — and any competing protocol must beat it on both gold-label detection and reproducibility to count as progress.

The strongest existing competitor is the fully automated verifier pipeline of ProtDBench: candidate binders scored by structure predictors (AlphaFold2, Protenix, Boltz-1), filtered by deterministic thresholds, and ranked by success criteria, with throughput-aware yield ($N_{24h}$) as the operational metric. This is the strongest competitor because it is the incumbent: it needs no humans, and on the Cao dataset it already demonstrated that AF3-style verifiers enrich true wet-lab binders better than AF2-style ones (no reported number found on Lacuna for that enrichment value). We will run it on exactly our Cao subsample with the same candidate budget, and our expert-with-map pipeline beats it only if it achieves higher enrichment of the 1,385 confirmed binders at matched labeling cost per selected candidate — a deliberately demanding bar, since human time is the expensive resource (GENIE's runs cost $350–$600 each).

Two ablations isolate our key ingredient. The fixed-map ablation applies the full QRA machinery of Belz et al. — replication runs, $CV^*$, full documentation — but holds the projection frozen; the gap between this and our varied-map condition is precisely $\sigma^2_{\text{projection}}$, the thing we claim matters. The no-map ablation gives experts the same candidates as a ranked list with verifier scores but no spatial layout; the gap between it and the map condition measures whether the map adds any diagnostic signal at all, rather than merely noise.

Why not just…?

Why not just rely on automated benchmark metrics and skip the humans? This is the most reasonable objection, and the file gives three concrete reasons it fails in this domain. ProteinGym showed that models with high overall rank correlation fail at the design task of picking top variants, so a single automated number misleads about downstream use. ProtDBench showed generative models slide into "adversarial ridges" — candidates that score near-perfectly under a verifier while lacking physical viability — and that different verifiers accept non-overlapping regions of candidate space, so there is no single automated number to rely on. And Belz et al. showed automated metrics themselves wobble: BLEU script and tokenizer variations produced $CV^*$ up to 4.176 on identical outputs. That said, if our experiments show verifier pipelines dominate expert inspection on binder enrichment, the honest conclusion will be that expert visual triage is not worth standardizing for that task — and our no-map and clustering baselines are designed to detect exactly that.

Why not just compute clustering statistics in the native high-dimensional space instead of drawing a map? A reviewer will correctly note that silhouette scores, nearest-neighbor purity, or FoldSeek structural clustering (as ProtDBench uses) are deterministic and need no human eyes. This is a real threat, and we test it head-on: a high-dimensional clustering detector run on the same embeddings against the same gold labels, with the same tuning budget. The evidence tilting against it is that flat summary statistics already failed once in this exact setting — Ofer and Linial's headline metrics (99.7% ROC AUC) concealed the asymmetric 9.48% versus 1.87% error structure that spatial inspection exposed, and Semantic Concept Spaces found statistically "coherent" topics that violated domain logic only experts could see. But if the clustering detector matches expert-plus-map performance on our gold tasks, we will report that the map is unnecessary — that would itself be a useful, publishable negative result about a widespread practice.

Why not just replace the human judges with LLM-as-judge or verifier networks? The file documents two failure modes that transfer directly. Verifier bias: ProtDBench found candidate designs accepted by one verifier completely rejected by another, so an automated judge is not a stable reference either. Shared blind spots: Ofer and Linial showed protein language models replicate the same failure modes as the human immune system on molecular mimicry — the automated judge inherits the very biases you are trying to audit. And GENIE found automated metrics badly underestimated the human–machine gap on reasoning tasks (UnifiedQA scored 80.8 on ARC-DA yet lagged human quality). Still, we include an LLM-judge condition in the triage study; if it proves as accurate and as stable as calibrated experts, that result would redirect the protocol toward auditing the judge instead.

Proposed Experiments

Study 1 (instrument variation) renders the SwissProt host–virus embeddings of Ofer and Linial under 30 projection settings (t-SNE, UMAP, MDS, PCA; three hyperparameter settings each; multiple seeds). Twelve bioinformatics experts mark regions they judge as error-prone and classify marked clusters. Outcomes: Krippendorff's alpha within and across maps, $CV^*$ of detection rates over replications, and the estimated $\sigma^2_{\text{projection}}$. The decisive check is whether experts' region judgments track the gold asymmetric error structure (the 9.48% viral-mimicry error class) or merely chase whichever clusters the current seed happened to tear apart.

Study 2 (triage against wet-lab truth) samples a stratified set from the Cao dataset (all 1,385 confirmed binders is infeasible for human rating; we sample binders and matched non-binders at a rate set by a GENIE-style cost model) and asks experts to triage candidates via map, via ranked verifier-score list, and via LLM judge. The metric is enrichment of true binders among the expert-selected subset, with confidence intervals over experts and re-renderings, compared against the ProtDBench verifier pipeline on the identical sample.

Study 3 (anchoring and pareidolia) re-renders the same embeddings under a new seed after each expert's first pass and asks for the same judgments; the drop in test-retest agreement is the visual-anchoring error rate the literature suspects but has never measured. A placebo condition — maps of randomly shuffled embeddings — sets the false-positive floor: if experts report meaningful structure there, that is the instrument's noise floor, and every reported finding must clear it.

Study 4 (protocol) combines frozen layouts, planted calibration points adapted from GENIE, and binary atomic judgments, and tests whether $CV^*$ of expert measurements falls below the unstructured baseline by a pre-registered margin, with the human-stability benchmark of Belz et al. ($CV^* \approx 6.1$ for well-specified human judgments) as the target regime. Success across the four studies yields something the field currently lacks: an error bar on the sentence "as can be seen from the t-SNE plot."