Precedent Labels and Facts Shift LLM Approval Decisions in Opposite Directions

Junwei Yao, Zike Yang, Yongxi Zhou, Yanxian Chen, Jianan Liu, Jiaxi Wen, Zhijun WangarXiv 2026 · arXiv

When building Large Language Model (LLM) systems for administrative tasks—like deciding whether to grant a refund or approve an insurance claim—developers often use Retrieval-Augmented Generation (RAG). The idea is simple: show the model the current request alongside a few "precedents" (similar past cases) to ensure consistency.

However, a new paper titled "Precedent Labels and Facts Shift LLM Approval Decisions in Opposite Directions" reveals a fascinating and counter-intuitive tug-of-war happening inside the model’s "mind." It turns out that the labels of past decisions and the facts of those cases pull the LLM in completely opposite directions.

TL;DR

When an LLM evaluates a borderline request, seeing past cases with "APPROVE" labels makes it much more likely to approve. Conversely, seeing the facts of those same approvable cases (without the labels) actually makes the model less likely to approve, as the current request looks weaker by comparison—a phenomenon known as the "contrast effect."

What Problem Does This Paper Address?

As we move toward "AI-as-a-judge" systems, consistency is the gold standard. If a customer gets a refund for a broken screen today, a similar customer should get one tomorrow. RAG is the primary tool used to achieve this "case law" style of grounding.

The authors wanted to know: what exactly is the model learning from these precedents? Is it following the logic of the past facts, or is it just being swayed by the labels (the final "APPROVE" or "DENY" verdict)? Understanding this is critical for building fair systems that don't just blindly anchor to past mistakes.

Methods

The researchers tested three major models: DeepSeek V4.1 Flash, GPT-5.6 Luna, and Claude Haiku 5.5. They focused on "borderline" cases—requests where the model, when left to its own devices, was roughly 50/50 on whether to approve or deny.

The experimental design was clever in its simplicity. For a fixed "target" request, they provided three precedents and systematically varied two things:

  1. The Label Effect: They kept the facts of the precedents the same but flipped the recorded decisions between APPROVE and DENY.
  2. The Facts Contrast: They removed the decision labels entirely and changed the facts of the precedents to be either clearly "approve-side" (meeting all criteria) or "deny-side" (failing criteria).
This example shows how changing just the decision label on a Steam refund request can flip the model's verdict.

Datasets

The study utilized two distinct sets of materials:

  • Public Policies: 9 real-world policies (e.g., Steam refunds, IRS penalty relief, Airbnb rebooking) paraphrased into 5–6 clear rules. They generated 65 borderline target requests for these.
  • Fictional Policies: 6 entirely new organizations (retailers and insurers) to ensure the models hadn't seen the data during training.
  • Human Benchmarks: Three human annotators provided a "ground truth" consensus on 40 of the targets to see if the model's shifts were corrections toward the "right" answer or merely biases.

Key Insights and Ideas

  • The Tug-of-War: Labels and facts are in a constant battle. Labels act as an anchor: if the model sees "APPROVE" on similar cases, it feels social pressure to do the same. Facts, however, trigger a contrast effect: if the precedents are "perfect" examples of a valid claim, the borderline target request looks "shabby" or "weak" by comparison.
  • Similarity Amplifies Labels: The "pull" of the label is strongest when the precedents are near-copies of the target request. When the precedents are from a different type of request (even under the same policy), the label effect shrinks significantly.
  • The Contrast Effect is a Bias, Not a Correction: One might hope the "facts push" helps the model find the right answer. However, human annotation showed the push happened even on requests that humans judged as clearly approvable. The model wasn't getting "smarter"; it was just being biased by the comparison.

Key Results

The quantitative shifts were massive, especially for DeepSeek:

  • Label Power: Switching precedent labels from DENY to APPROVE raised approval rates by 30 to 55 percentage points across the three models.
  • The Facts Push: In the absence of labels, showing "approve-side" facts (which should theoretically encourage approval) actually lowered DeepSeek's approval rate by 26.2 points.

As shown in the figure below, the label effect ($L$) is consistently positive (pulling toward the label), while the facts contrast ($F$) is consistently negative (pushing away from the "strong" facts).

The label effect (filled circles) and facts contrast (open squares) move in opposite directions across all nine public policies.

Limitations and Future Work

The authors acknowledge several constraints:

  • Single-Step Logic: The study used relatively short fact sheets and single-call decisions. Real-world claims are often much more complex.
  • Borderline Focus: The effects are most pronounced on "close calls." Clear-cut cases (e.g., a refund request 100 days past the limit) are much harder to sway.
  • Retrieval Dynamics: The study used "near-copy" variants. In a real production system, the retrieval tool might not always find such close matches, which could change the balance between the label and fact effects.

The takeaway for engineers? If you're building a RAG system for decision-making, test your labels and facts separately. Don't assume that showing "good" precedents will lead to "good" decisions; you might accidentally be setting a bar so high that your model starts denying perfectly valid requests.

Summary updated

Abstract

When a language model decides refund or claim requests under written policies, the system can show it similar past cases. Each past case has two parts: its facts and the decision recorded for it. We separate the two. A model sees a policy, a request and three past cases, and we change either the recorded decisions or the facts, keeping the request fixed. We select requests that are close calls: deciding without past cases, DeepSeek V4.1 Flash approved each in some of four tries and denied it in others. When the past cases are near-copies of the request (its facts with a few details changed), switching their recorded decisions from DENY to APPROVE raises approval by about 30 to 55 percentage points in DeepSeek V4.1 Flash, GPT-5.6 Luna and Claude Haiku 5.5 on policies written for the study, and DeepSeek and Luna show the same on nine public policies paraphrased from real ones. The facts push the other way: shown without their recorded decisions, past cases whose facts support approval make these two models approve less often than past cases whose facts support denial. One reading is that the request looks weaker next to them. This push is not just a correction toward the right answer: in an additional DeepSeek analysis, it was of similar estimated size on requests that human annotators judged approvable and on those they judged deniable. In a separate DeepSeek run on the public policies, when the past cases are near-copies of a different request, switching their decisions moves approval by 9 points instead of 43, while facts that support approval still lower it by 22 points against facts that support denial. Systems that retrieve past decisions should test the two parts separately.
BibTeX
@misc{yao2026precedent,
  author = {Junwei Yao and Zike Yang and Yongxi Zhou and Yanxian Chen and Jianan Liu and Jiaxi Wen and Zhijun Wang},
  title = {{Precedent Labels and Facts Shift LLM Approval Decisions in Opposite Directions}},
  year = {2026},
  eprint = {2611.00672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00672},
}
Figure