Contrastive Precedents: Retrieving Explained Near-Misses for In-Context Classification

When a large language model classifies text from a few examples placed in its prompt, the usual practice is to fetch the stored examples that look most like the new input. For judgment-heavy tasks such as spotting coded hate speech or checking a message against a custom safety policy, that practice has a quiet failure mode. The most similar stored examples tend to share the new input's surface features and often its label, so the model can simply copy the label of its lookalikes without learning the rule that separates the classes. We propose retrieving contrastive precedents instead. A contrastive precedent is a pair of stored examples that sit close to the query but carry opposite labels, and each member comes with a one-sentence explanation naming the cue that decided its label ("this is a violation because it targets a named individual; its neighbor is not, because it criticizes an institution"). The prompt then shows the model where the decision boundary runs near this query, rather than only which side most neighbors fall on.

The idea matters for two reasons. First, the literature shows that even 70-billion-parameter models lean on the surface meaning of in-context examples rather than their demonstrated input-to-label mapping, so bigger models do not make the problem go away. Second, explanations help classification, but nobody has tested whether retrieved explanations help or instead hand the model a new shortcut: echoing explanation phrasing while still guessing from surface cues. Our core claim can be stated in two sentences. For explanation-augmented classification, the retrieval unit should be a pair of near-identical examples with opposite labels and cue-naming explanations, not a list of nearest neighbors. This should cut label-copying errors and hold its advantage from 7B to 70B+ models, and a set of controlled ablations will show whether the explanations or the contrast carries the gain.

Core Claim

Three bodies of work meet here without overlapping. The first line of evidence says that fetching demonstrations helps classification. Making Pre-trained Language Models Better Few-shot Learners adds sentence-embedding-retrieved demonstrations to prompt-based fine-tuning and reports an 11% average absolute gain over 15 tasks. STAND-Guard fetches 10 embedding-nearest moderation examples and lets a 7B model match GPT-4-Turbo on unseen moderation tasks. Selective Demonstration Retrieval for Improved Implicit Hate Speech Detection (ARIIHA) finds that keyword-based BM25 retrieval beats dense retrieval for implicit hate and adds a filter that rejects candidates likely to induce shortcuts. All three show only inputs and labels, with no explanations. The second line says explanations help. Explain Yourself! generates explanations before answering on CommonsenseQA and gains about ten points, and DynaGuard trains guard models to reason before emitting PASS/FAIL against user-written policies. Neither retrieves examples at all. The third line says similarity-based retrieval misleads when a task requires reasoning. Many-Shot CoT-ICL documents a "similarity trap" in math, where the most-similar examples sometimes lose to the most-dissimilar ones. Reasoning Graph Enhanced Exemplars Retrieval (RGER) and the chain-of-thought-aware retrieval in Dr.ICL respond by matching on the structure or text of reasoning chains, but only for math and formal logic, and RGER needs an extra generation pass for every query.

We claim that explained classification sits on the boundary these works leave unmapped, and that the similarity trap takes a specific form there. The fetched neighbors are not misleading because they need a different procedure. They are misleading because they share a label for the wrong reason. Rectifying Demonstration Shortcut in In-Context Learning constrains the design. It shows that models up to Llama2-70B fall back on the semantic priors of demonstration words, and it addresses this with probability calibration on direct-label prompts. Our proposal targets the same shortcut at the prompt-construction level and in the explain-then-label setting, which that paper does not study. What would be new is the retrieval unit (opposite-label near-miss pairs), the role of the explanation (naming the deciding cue, not narrating a solution), and a direct measurement of whether retrieved explanations fix label copying or replace it with explanation copying, across scale.

Research Program

We will build three components and one set of diagnostics.

The first component is an explained example pool. Some benchmarks lack human explanations. For those, we generate a one-sentence cue-naming explanation for each pool example with a mid-sized model that is told the gold label and the task guideline. We keep an explanation only if a separate model, shown the explanation alone, recovers the gold label. This filter matters because Explain Yourself! shows that explanation quality caps downstream accuracy: its generated explanations reach 64.7% on CommonsenseQA against roughly 90% for human ones. It also shows that generated explanations can carry dataset biases. For CommonsenseQA we use the human CoS-E explanations directly, which gives a clean control on the effect of explanation quality.

The second component is the precedent retriever. For a query, a cheap first-stage retriever (BM25, following ARIIHA's finding for implicit hate, and a dense encoder as an alternative) returns candidates. For each top candidate, we find its nearest pool neighbor with the opposite label, measured by the same similarity score. We keep pairs whose two members are close to each other and to the query. The prompt contains $k/2$ such pairs in a fixed pair-internal order that is randomized across pairs, each example followed by its explanation and label. The query is then answered with a short explanation and a label. No extra model call is spent on retrieval, which keeps latency close to ordinary nearest-neighbor retrieval and distinguishes the method from RGER and from the inference-time gradient scoring of MDR.

The third component is a scale ladder of open-weight instruction-tuned models from about 7B through 70B, extended to GPT-OSS-120B if compute allows. All models are run with identical prompts. This choice follows the observation that findings on 7B models often fail to predict 70B+ behavior.

The diagnostics are what make the study informative rather than a leaderboard entry. The first is neighbor echo: among test items whose gold label differs from the majority label of the shown examples, the fraction predicted with that majority label. This is our operational measure of label copying. The second is explanation-swap sensitivity: we replace each shown explanation with one that names a plausible but wrong cue while keeping the label unchanged, and measure how much predictions and the model's own explanations move. If nothing moves, the model is not reading the explanations. If the model's explanation copies the swapped cue but its label stays driven by surface overlap, that is an explanation shortcut. The third is a label-shuffled "task learning" condition borrowed from Rectifying Demonstration Shortcut, with arbitrary string labels, which tests whether pairs teach the mapping when surface meaning of label words cannot help.

Baselines

The naive reference points are zero-shot classification with the task guideline, run in both direct-label and explain-then-label modes on every model. The latter matters most at 70B+, where a reviewer will expect a strong model to need no examples. For zero-shot on the Implicit Hate Corpus, ARIIHA reports 69.52 macro-F1 with Qwen2.5-7B-Instruct. For the 70B models, no reported result found on Lacuna for zero-shot explain-then-label on these benchmarks, so we will measure it.

The standard competitor is nearest-neighbor retrieval with inputs and labels only, as in STAND-Guard's retrieval step and ARIIHA's RetICL baseline (73.76 macro-F1 on the Implicit Hate Corpus, 8-shot). Its explained twin attaches the same filtered explanations to the same nearest neighbors. That twin is the fairest test of whether the pairing matters once explanations are held fixed.

The strongest existing retrieval method on our headline benchmark is ARIIHA itself. It reports 76.79 macro-F1 on the Implicit Hate Corpus at 8 shots with Qwen2.5-7B-Instruct. That figure is the number to beat. We will reimplement ARIIHA on our pool and models, confirm we reproduce roughly its reported figure at 7B, and then run it at 70B, where no reported result found on Lacuna. As the reasoning-aware retrieval competitor we run an RGER-style variant: the model drafts an explanation for the query, and retrieval matches on explanation similarity. This is the approach of RGER and Dr.ICL's chain-aware retrieval, adapted to classification. It pays one extra generation per query, and we report that cost. For calibration, In-Context Calibration is applied on top of direct-label nearest-neighbor prompts, since it attacks the same shortcut by a different route.

On DynaBench the reference point is a trained system rather than a prompting one. DynaGuard-8B reports 73.1% F1, against 13.1% for LlamaGuard3. On STAND-Guard's out-of-distribution moderation suite, STAND-Guard reports 0.577 average F1 and GPT-4-Turbo 0.588. We treat these as external anchors, not same-budget comparisons, because they involve task-specific fine-tuning.

Fairness is enforced by holding constant, within each benchmark, the example pool, the explanation generator and filter, the number of shown examples $k$, the prompt template apart from example content, and the decoding settings. Each method gets the same small development-set budget for tuning $k$ and first-stage retriever choice. Beating the field means a macro-F1 gain over ARIIHA and over explained nearest-neighbor retrieval that holds across three random seeds for example order. It must also come with a lower neighbor-echo rate, and it must appear at both the 7B and 70B ends of the ladder.

The key ablation separates the two ingredients by running contrastive pairs without explanations and explanations without contrast (the explained twin above). A further ablation replaces the cue-naming explanations with generic chain-of-thought narrations, to test whether the specific job of naming the deciding cue matters.

Why not just…?

Why not just prompt a 70B model zero-shot with the guideline and let it reason? Existing evidence says this falls short but is not conclusive at current scales. On in-distribution moderation, STAND-Guard reports zero-shot GPT-4-Turbo at 0.458 F1 against 0.680 for a retrieval-augmented 7B model. DynaGuard reports GPT-4o-mini at 75.8% average F1 against 79.7% for its trained 8B guard. Neither tested open 70B+ models reasoning zero-shot, however, so this is a real threat. Our scale ladder settles it. If the zero-shot explain-then-label run at 70B+ comes within noise of contrastive precedents, we will report that retrieval is a small-model crutch for these tasks.

Why not just retrieve label-balanced nearest neighbors and skip the explanations? This is the most dangerous alternative, because much of the benefit could come from simply showing both labels near the boundary. The ablation of pairs without explanations answers it directly. If that ablation matches the full method on macro-F1 and neighbor echo, we will narrow the claim to "contrast, not explanation," which is still a finding the literature lacks. The explanation-swap test tells us whether the models read explanations at all, and whether this changes with size.

Why not just calibrate output probabilities instead of changing which examples are shown? Rectifying Demonstration Shortcut reports a 23% average F1 gain on Llama2 from calibration and 9% in the label-shuffled setting at 70B. So calibration is effective, and we run it as a baseline. It corrects a prior over label words in direct-label prompts, however, and has no obvious counterpart once the model writes an explanation before the label. We also run it stacked on contrastive precedents to test whether the two fixes are complementary.

Why not just retrieve by a drafted explanation, as RGER does for math? It is a reasonable transfer and we run it. It costs a full extra generation per query, which RGER itself flags as a latency problem. A drafted explanation that picks the wrong cue will also fetch examples that confirm the wrong cue. If it beats contrastive precedents at equal $k$, the cost-accuracy comparison we report decides which one a practitioner should use.

Proposed Experiments

The headline experiment runs all methods on the Implicit Hate Corpus, DynaBench, and the English binary tasks of STAND-Guard's out-of-distribution suite, at $k=8$ and $k=16$, across the model ladder. We report macro-F1 (F1 for DynaBench), neighbor echo, and tokens per query. CommonsenseQA with human CoS-E explanations versus filtered generated explanations serves as the explanation-quality control. It asks whether the pairing gain shrinks when explanations are worse, which Explain Yourself! suggests it should.

Three checks would decide the question. First, on the subset of test items where the gold label disagrees with the majority label of the nearest neighbors, contrastive precedents should beat explained nearest-neighbor retrieval by a clear margin even if the overall macro-F1 gap is small. That subset is where the claimed mechanism acts, and a null result there falsifies the core claim. Second, explanation-swap sensitivity should rise with model size if larger models truly use the stated cue. If it stays flat while accuracy rises, larger models are ignoring the explanations, and the gain must come from the contrast alone. Third, in the label-shuffled setting at 70B, contrastive pairs should recover more of the arbitrary mapping than nearest neighbors. This would show that the method weakens reliance on semantic priors rather than riding them.

Together these results tell a practitioner which retrieval to use for explained classification at a given model size, and whether the explanations in the prompt are doing work or just decorating it.