Large language models still fail at professional judgment, even when they know the relevant rules. In Chinese civil appeals, every one of twenty mainstream models, including reasoning models, scores under 50% F1 at predicting whether an appellate court will reverse a lower-court ruling. Yet the legal system already produces, at scale, exactly the teaching material those models lack. Every appellate judgment explains, in the court's own words, why it overturned or upheld the decision below. This proposal treats those explanations as free, automatically harvested worked examples. We pair each first-instance case with the reasoning written by the court that reviewed it, and show a frozen model a few such pairs before it judges a new case. Because automatic pairing is error-prone, we select which pairs to show by testing each candidate's effect on roughly one hundred hand-checked cases.
The bet is that this fixes the failure two simpler fixes cannot. Telling the model the rule does not help: when TheoremQA prepended theorem statements to prompts, accuracy barely moved. Letting the model write its own worked examples does not help either, because a model that cannot reason about appeals produces confident wrong examples. Externally written, application-level reasoning is the missing ingredient. The open technical question is whether label and rationale noise in an automatically built example pool helps or hurts. Prior work on example selection says nothing about this, because it assumes the pool is clean.
Core Claim
Our claim has two parts. The first is that showing a frozen model distantly supervised demonstrations, meaning examples built by programmatically aligning text with external records, beats three alternatives. Those alternatives are bare rule retrieval, self-generated pseudo-examples, and a small clean example set. The demonstrations must carry a rationale written by a real expert (the appellate court), not just a label. The second is that this advantage only survives realistic alignment noise if demonstrations are chosen by their measured effect on a small clean seed set, not by text similarity alone.
Both parts can fail independently. In-context learning may be robust enough to noisy labels that no denoising is needed. Alternatively, a hundred clean examples may cover the task well enough that the large noisy pool adds nothing. The experiments are built to detect either outcome.
Where This Sits
Three lines of prior work motivate the gap. AppealCase builds 10,000 matched first- and second-instance civil judgments across 91 causes of action. It aligns them with regular expressions, case identifiers, and a rules-plus-LLM-plus-expert labeling pipeline. It shows the reversal-prediction barrier, but it uses the pairs only as a benchmark, never as teaching material. TheoremQA shows that open models score below 15% on theorem-driven STEM questions and that injecting theorem statements yields negligible gains. Its conclusion is that knowing a rule is decoupled from applying it. Aligning Unstructured Paris Agreement Climate Plans with Sustainable Development Goals finds that up to 20 concrete demonstrations substantially help on a specialist taxonomy, while expert personas barely help. Its demonstrations, however, came from expert gold annotations. We keep the "concrete examples beat abstract instructions" lesson and remove the dependence on gold data.
Two papers constrain the design. PAPO builds demonstrations from the model's own high-confidence answers and refines prompts with textual critiques. It reports GPQA accuracy moving from 37.9% to 39.9%. By construction, though, it cannot escape a cold start where the model's majority answer is wrong. We replace its self-generated supervision with externally written court reasoning. KMQA converts medical knowledge-graph triples into pseudo-sentences for a fine-tuned reader. It attributes 32% of its errors to retrieval and conversion noise. That warns us that distantly aligned knowledge is noisy enough to hurt, and that a noise-handling step is not optional.
Two papers supply methods. Few Clean Instances Help Denoising Distant Supervision shows that about 100 clean instances can guide influence-based filtering of distant-supervision corpora with up to 90% noise. It does so only for fine-tuning small encoders, and with expensive inverse-Hessian computations. Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation, called ICL-GRADEX, approximates a demonstration set's effect on a frozen model with a first-order expansion around an anchor prompt. Its gradient-guided retrieval variant beats the strongest embedding baselines by 11% F1 on average, but on clean pools only. We reuse its cheap estimator and point it at a different target: the clean seed rather than the test query. The clean-seed idea itself comes from the influence-function paper, transplanted from weight updates to prompt construction.
Research Program
Building the pool. For each matched pair, we take the first-instance record, meaning the claims, the facts, the court's view, and the judgment. From the second-instance judgment we extract a short rationale with rule-based section splitting. This is the passage where the appellate court states what the lower court got right or wrong and which provisions apply. We also extract a distant label: affirmed, reversed for factual error, or reversed for legal error. We deliberately use only regular expressions and keyword patterns, without the expert pass that AppealCase applied. The result is a genuinely distant pool whose noise rate we can measure against AppealCase's curated labels.
The clean seed. We draw about 100 cases, stratified by cause of action, from the curated labels. These are the only human-checked items the method may touch, and they never appear as test items.
Scoring candidates. Let $q$ be a test case. Let $d$ be a candidate demonstration from the pool, and $S_q$ the clean seed cases nearest to $q$ in embedding space. Let $A$ be a fixed anchor prompt, and $\ell(y \mid x, P)$ the frozen model's log-probability of the correct label $y$ for case $x$ under prompt $P$. We score
$$u(d \mid q) = \frac{1}{|S_q|} \sum_{(x,y) \in S_q} \big[\ell(y \mid x, A \cup \{d\}) - \ell(y \mid x, A)\big].$$
In words, $u(d \mid q)$ is the average change in the model's confidence in the correct answer on nearby verified cases when $d$ is added to the prompt. A mislabeled pair or a garbled rationale should push $u$ negative. We compute $u$ with the ICL-GRADEX linear approximation, using low-dimensional random projections, so that scoring the full pool costs a linear number of forward and backward passes rather than one pass per candidate. The prompt for $q$ consists of the top $k$ candidates under a product of embedding relevance and $u$, with a cap of two per cause of action to keep the selected examples diverse.
What we analyze. Beyond headline accuracy, we compute the correlation between $u$ and true label correctness on the pool, using the curated labels as ground truth. This tests whether the seed-based score is in fact detecting noise. We also separate gains from labels and gains from rationales by running the same selected cases with rationales removed.
Baselines
The naive reference point is zero-shot prompting of the same frozen model on first-instance records. AppealCase reports that all 20 models it evaluated score under 50% F1 on first-instance reversal prediction. The file gives only this bound, not per-model figures, so we will rerun our chosen open model ourselves to fix the zero-shot number.
The second reference is seed-only prompting. Here the 100 clean cases, with their curated rationales, are the entire demonstration pool, retrieved by embedding similarity. This is the baseline a careful practitioner would actually build. Beating it is necessary for the claim that the large noisy pool earns its keep.
The third baseline is label-blind top-$k$ retrieval over the raw distant pool, with no seed scoring. Comparing our method against it isolates the denoising step.
The strongest existing selector is ICL-GRADEX in its original form, which estimates utility with respect to the query anchor and ignores label noise. On clean benchmarks its authors report an average 11% F1 gain over the strongest embedding baselines. No reported result was found on Lacuna for ICL-GRADEX on AppealCase or on any noisy pool.
The strongest self-supervised competitor is PAPO, run with its self-consistency pseudo-labels on the unlabeled training cases. It gets the same number of model calls and is allowed to use the 100 seed cases for its critique step.
The strongest parameter-updating competitor applies the influence-based denoising of Few Clean Instances to the same distant pool and fine-tunes an encoder classifier on the result. It shows whether frozen in-context use is even the right deployment mode. No reported result was found on Lacuna for that method on AppealCase.
The key ablations run our selector with labels only (no rationales), and our selector on the curated-label pool as an oracle-clean upper bound.
All prompted methods use the same frozen model, the same number of shots $k$, the same token budget (truncating case facts identically), and the same 100 seed cases. Hyperparameters such as $k$, the anchor, and the neighborhood size are tuned only on a held-out split of the seed set, with an equal number of configurations per method. Beating the baselines means a statistically reliable gain in reversal F1 over both seed-only prompting and raw top-$k$ retrieval, measured by bootstrap over test cases with at least three prompt-order seeds. A gain over only one of the two does not support the full claim.
Why not just…?
Why not just use a bigger or reasoning-specialized model? AppealCase already tested GPT-4, DeepSeek-V3, and DeepSeek-R1, and none passed 50% reversal F1. Scale alone therefore does not close this gap. We still run every prompting condition on two model sizes, to check whether our gains shrink as the base model gets stronger.
Why not just retrieve the relevant statutes? The TheoremQA augmentation result is direct evidence that handing a model the rule does not teach it to apply the rule. KMQA adds that retrieved structured knowledge is itself a major error source. We include statute retrieval drawn from the provisions cited in similar cases as an explicit condition. If it matches rationale demonstrations, our central claim about application-level examples is wrong.
Why not just use the hundred clean cases as demonstrations? This is the most serious threat. If 100 cases cover the 91 causes of action well enough, the distant pool is pointless. Our argument is coverage: most test cases will have no seed neighbor from the same cause of action and fact pattern. The seed-only baseline settles this directly. We additionally break results down by whether the test case's cause of action appears among the seed cases' causes of action. The proposal predicts the largest gains exactly where the seed is silent.
Why not just use raw distant labels and skip denoising? In-context learning may tolerate noisy demonstrations, in which case the scoring step is unnecessary overhead. Raw top-$k$ retrieval is a baseline for this reason. The controlled-noise study below measures where, if anywhere, the two diverge. If they never diverge, we will report that as the finding and drop the denoising claim.
Why not just fine-tune a small model on the denoised pool? Few Clean Instances shows this works for relation extraction. Appellate reasoning may simply be learnable by a fine-tuned encoder. We run it as a baseline under equal data. If it wins, the right conclusion is that distant appellate supervision should be used for training rather than prompting, and the paper reports that.
Proposed Experiments
The main experiment runs on AppealCase with a split by case identifier, so that no case or its counterpart appears in both the pool and the test set. Where dates permit, we also use a temporal split to limit memorization. Metrics are reversal F1 and macro-F1 over the three-way reversal-reason label. We also report pool noise rate, the correlation between $u$ and label correctness, and token cost per query.
A controlled-noise study runs on ACE05-N, the corrupted version of ACE05 relation extraction with label noise from 10% to 90%, using in-context relation classification with 100 clean seed instances. We plot accuracy against noise rate for seed-scored selection, raw top-$k$, and original ICL-GRADEX. No reported in-context result was found on Lacuna for ACE05-N; Few Clean Instances reports only that its fine-tuned method maintains high F1 up to 90% noise. ACE05-N lacks expert rationales, so this study tests only the denoising half of the claim.
Three outcomes decide the proposal. The first is whether rationale-bearing distant demonstrations beat both statute retrieval and label-only demonstrations on AppealCase. That would show that externally written application reasoning, not added facts, carries the gain. The second is whether seed-scored selection beats raw top-$k$ once pool noise passes some threshold on ACE05-N, and whether AppealCase's measured noise rate lies above that threshold. The third is whether the full method beats seed-only prompting chiefly on causes of action the seed does not cover.
Positive answers to all three would establish distant supervision as a practical source of in-context demonstrations for expert domains. A negative answer to any one tells us precisely which ingredient is unnecessary.
References
- AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios
- TheoremQA: A Theorem-driven Question Answering Dataset
- Aligning Unstructured Paris Agreement Climate Plans with Sustainable Development Goals
- In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement
- Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
- Few Clean Instances Help Denoising Distant Supervision
- Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation