Using a large language model as a grader ("LLM-as-a-judge") has become the default way to score open-ended outputs, but every verdict costs a full model call, and real evaluation workloads repeat themselves heavily. Reinforcement-learning pipelines sample dozens of similar responses per prompt. Best-of-N reranking scores near-copies of the same solution. Successive model checkpoints are re-graded on the same fixed benchmark prompts. We propose a judgment memory. Each (prompt, response, rubric) triple that has already been graded is stored as a short binary code, meaning a fingerprint of a few dozen bits. When a new item arrives, the system finds every stored item within a fixed number of differing bits. If those neighbors agree, it returns their verdict; otherwise it calls the judge. Retrieval uses multi-index hashing, a classic method for searching binary codes. It splits each code into pieces and relies on the fact that two codes differing in at most $r$ bits must match exactly on at least one piece.
The central bet is that the expensive and interesting part is not fast lookup. It is learning a code in which "few bits apart" means "the judge would say the same thing," and then certifying how often reuse is wrong. Generic semantic similarity is the wrong notion for this purpose. A correct proof and the same proof with one wrong step are near-identical as text, yet they deserve opposite verdicts. A judgment cache built on ordinary text embeddings should fail badly on exactly those cases.
Core Claim
A binary code trained to preserve judge verdicts, rather than textual meaning, can be paired with exact radius search. The radius and the agreement rule can then be calibrated on held-out judge calls. The resulting cache should answer a large fraction of judge queries in repetitive workloads while disagreeing with a fresh judge call no more often than the judge disagrees with itself. Semantic caches calibrated to the same error budget should achieve much lower reuse, and the gap should be largest on adversarial near-duplicate pairs. Multi-index hashing plays two specific roles. First, it returns the complete set of stored items within radius $r$, which makes the reuse rule and its calibration exact rather than dependent on an approximate index's misses. Second, it keeps a memory of hundreds of millions of graded items in compact bit storage. We do not claim that it speeds up the dominant cost, which remains the judge call.
Relation to Prior Work
The retrieval machinery comes directly from prior work. Fast search in Hamming space with multi-index hashing and Fast Exact Search in Hamming Space With Multi-Index Hashing. introduce the pigeonhole construction and show exact search over a billion codes more than 1000 times faster than a linear scan. Their codes come from generic image descriptors, whereas ours must encode evaluation-relevant structure.
Learned codes for this kind of index are known to clump into overloaded buckets. Improved Search in Hamming Space Using Deep Multi-Index Hashing counters this with balancing penalties across pieces and buckets, which we reuse in our training objective. SECRET shows that segmented deep hashing works for text-to-code search, cutting retrieval time by at least 95% while keeping over 97% of accuracy. SECRET's target is relevance retrieval, whereas ours is verdict preservation.
The closest system is MVR-cache. It caches LLM prompts using learned segmentation, dense multi-vector matching over an approximate graph index, and an explicit error budget, and raises hit rates by up to 37%. It differs from our proposal in three ways. It caches responses to prompts, not judgments of prompt–response pairs. Its similarity is semantic, not trained on judge outcomes. Its index is approximate and dense.
Two judge-side papers motivate the work. LLM-as-a-Judge on a Budget shows that repeated judge calls have heterogeneous variance and allocates them adaptively, but each evaluation campaign starts with no memory of past ones. JETTS documents the cost of judges inside test-time search, with pairwise reranking needing $O(N^2)$ calls. It also documents a constraint that we take seriously: small 8B judges can have negative helpfulness on math and code. Any cheap surrogate for the judge, our cache included, must therefore be held to a strict error budget rather than to average agreement.
Research Program
We first build the code. A small text encoder reads the prompt, the response and the rubric text, and outputs $B$ bits (64 or 128) split into $m$ pieces for the index. Training uses logged judge calls. Pairs with matching verdicts, or with scores within one point on HelpSteer2-style 1–5 scales, are pulled together in Hamming distance. Pairs with differing verdicts are pushed apart beyond a margin, and hard negatives are mined from textually similar pairs. The two balancing penalties from deep multi-index hashing keep bucket occupancy flat. HelpSteer2 reports a 0.943 correlation between helpfulness and correctness. We therefore do not assign one rubric dimension to each piece; the rubric enters as conditioning text instead. We will report whether per-dimension pieces collapse, as that correlation predicts.
The second component is the reuse rule and its calibration. For a new item $x$, let $\mathcal{N}_r(x)$ be the full set of stored items within Hamming radius $r$. The system returns the stored verdict $v$ if at least $k$ neighbors exist and a fraction of at least $\tau$ of them share $v$. Otherwise it calls the judge and inserts the result. We choose $(r, k, \tau)$ on a held-out stream so that
$$\Pr\big[\hat v(x) \neq J(x)\big] \le \delta,$$
where $\hat v(x)$ is the cache's answer, $J(x)$ is a fresh judge call, and $\delta$ is the error budget. Because the judge is itself stochastic, we also measure its test-retest disagreement $\delta_J$ by calling it twice on the same item. We set $\delta = \delta_J$ as the natural target: the cache should be no less consistent with the judge than the judge is with itself.
The third component is the workloads. We construct streaming workloads with realistic repetition. The first samples 16–64 responses per prompt on the JETTS datasets for Best-of-N. The second re-grades MT-Bench and HelpSteer2 prompts across successive checkpoints of an open model. The third uses RewardBench chosen/rejected trios as an adversarial near-duplicate stress test.
Baselines
The naive reference point is calling the judge on every item. It defines the cost to beat and gives zero cache error by construction. Its quality ceiling is set by the judge itself, which on MT-Bench was GPT-4 with 80–85% agreement with humans in Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. A second trivial reference is an exact-match cache keyed on the hash of the full text. It is error-free but reuses only verbatim repeats, so it measures how much repetition is literal.
The strongest existing caching method is MVR-cache-style semantic caching. We will run a faithful reimplementation, with dense multi-vector matching, an approximate graph index retrieving 20 candidates, and reranking. We will also run a simpler single-vector dense cache with exact flat search and a tuned threshold. Both apply to the same (prompt, response, rubric) strings, are calibrated to the same $\delta$ on the same held-out split, and get the same hyperparameter search budget as our method. The reported reference figure is MVR-cache's hit-rate gain of up to 37% on search queries (MVR-cache). No reported result was found on Lacuna for semantic caching of judge verdicts. Beating this baseline means a higher fraction of avoided judge calls at equal measured disagreement $\delta$, with the gain concentrated on hard near-duplicate items.
The strongest parametric competitor is a reward model distilled from the same logged judge calls and given abstention: it answers only when its confidence clears a threshold calibrated to the same $\delta$. For context, ArmoRM-Llama3-8B scores 89.0 on RewardBench (RewardBench), and a reward model trained on HelpSteer2 reaches 92.0% (HelpSteer2). We will initialize from a comparable open reward model and fine-tune on identical judge logs with matched compute. Beating it means more calls avoided at equal $\delta$. Alternatively, it means matching its savings while absorbing new judge calls with no retraining, measured as savings over a stream whose prompt distribution shifts midway.
For the cost of repeated stochastic grading, the strongest method is ROBIN-HOOD. It reaches a worst-case absolute error on HelpSteer2 correctness of 0.300 with GPT-4.1 nano at 50k queries, against 0.451 for uniform allocation (LLM-as-a-Judge on a Budget). We treat it as complementary. In that same setting, we will compare ROBIN-HOOD alone against ROBIN-HOOD with cache-served items removed from the pool and with neighbor variances used as its initial estimates. Beating it means lower worst-case error at 50k total judge calls.
Two ablations isolate the key ingredient. The first replaces judge-trained codes with codes obtained by binarizing a generic sentence embedding; this tests whether verdict-aware training, rather than hashing, is the source of any gain. The second keeps judge-trained training but uses dense codes with flat search; this measures what binarization costs in reuse rate and what it saves in memory.
Why not just…?
Why not just use a dense vector database with a similarity threshold? At the scale of a single benchmark, a flat dense search is exact and its latency is negligible next to a judge call. So the speed argument for multi-index hashing is weak, and we do not make it. The case rests on two other points. The first is the learned metric, which the dense judge-trained ablation tests directly. If dense judge-trained codes match binary ones on reuse at equal $\delta$, the contribution narrows to memory: 64 bits per item instead of kilobytes of floats. We will report it that way. The second point is that approximate graph indexes miss neighbors, so they undercount disagreeing neighbors and can make reuse look safer than it is. Exact radius search removes this failure mode, and we will measure how often approximate retrieval drops a dissenting neighbor.
Why not just distill a small reward model and use it as the cheap judge? This is the most serious threat, because a neighbor-vote cache is essentially a nonparametric reward model with abstention. JETTS shows that small evaluators can actively mislead on math and code (JETTS), but a calibrated abstaining model sidesteps much of that problem. The case for the memory is that it updates the moment a new judgment is logged. It also abstains naturally in unseen regions because those regions have no neighbors, and every reused verdict points to the concrete prior judgment behind it. The distribution-shift stream is designed to settle this question. If the abstaining reward model matches the cache there, the honest conclusion is that a judgment memory adds only auditability.
Why not just reduce calls with adaptive repetition like ROBIN-HOOD? ROBIN-HOOD cuts repeated calls on a fixed set of items by about half, but it still spends tens of thousands of calls on 1,000 pairs. It also cannot avoid the first call on an item it has effectively seen before. The two methods address different redundancy: repetition of the same item versus near-repetition across items. The combined experiment shows whether the savings add up.
Proposed Experiments
The first decisive check runs on RewardBench Chat Hard and Reasoning. We measure, at matched $\delta$, how often each cache serves the chosen response's verdict to the rejected response. We expect semantic caches calibrated to $\delta = \delta_J$ to be forced into near-zero reuse or to violate the budget on these subsets, and judge-trained codes to keep chosen and rejected apart. If judge-trained codes show the same collapse, the core claim fails.
The second check is end to end. On GSM8k, MATH, HumanEval+, MBPP+ and IFEval from JETTS, we run Best-of-N with N up to 64 and report normalized helpfulness of the selected answer against judge calls spent, comparing full judging, each cache, and the abstaining reward model. No reported normalized-helpfulness values were found on Lacuna for these exact configurations, so the full-judge run serves as our reference. We also include pairwise judging, caching ordered pair codes so that position bias stays consistent, to test whether reuse dents the $O(N^2)$ cost without degrading selection.
The third check is the checkpoint-regrading stream on MT-Bench and HelpSteer2. We report the fraction of calls avoided across ten successive checkpoints, mean absolute score deviation per attribute against fresh judging, and stability of the model ranking (rank correlation against full judging).
Systems measurements cover memory per stored item, bucket occupancy with and without balancing, and encoder versus lookup versus judge latency. The encoder pass is expected to dominate the cache's own overhead. A negative result on any of the first two checks would be informative in its own right, since it would tell practitioners that judge verdicts are not locally stable in any compact learned space.
References
- Fast search in Hamming space with multi-index hashing
- Fast Exact Search in Hamming Space with Multi-Index Hashing
- Improved Search in Hamming Space Using Deep Multi-Index Hashing
- SECRET: Towards Scalable and Efficient Code Retrieval via Segmented Deep Hashing
- MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation
- LLM-as-a-Judge on a Budget
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
- HelpSteer 2: Open-source dataset for training top-performing reward models
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
- RewardBench: Evaluating Reward Models for Language Modeling