Correct Labels Are Not Enough: Reliable Metadata Can Silently Control Model Behavior

Renee Jia, Jinzhou WuarXiv 2026 · arXiv

TL;DR

Even when training labels are 100% correct, Large Language Models (LLMs) can "silently" learn to rely on incidental metadata—like reward scores or quality tags—as a shortcut. This dependency is often invisible during standard evaluation but allows the metadata to hijack model behavior at test time, causing models to choose wrong answers if the metadata is moved.

What Problem Does This Paper Address?

In the quest to align LLMs, researchers often decorate fine-tuning data with metadata: reward model (RM) scores, confidence labels, or system prompts. We generally assume that if a model achieves high accuracy on a task, it has "learned" the task.

However, this paper explores a dangerous form of shortcut learning. If metadata (a "cue") perfectly predicts the correct answer during training, the model might learn to follow the cue rather than solve the underlying problem. The catch? This dependency is "silent." A model might still perform well if the cue is removed, leading evaluators to believe the model is robust, while in reality, the presence of the cue completely controls its output.

Methods

The authors fine-tuned several 7–9B parameter open-weight models (Qwen3, Gemma-2, and Llama-3.1) using Low-Rank Adaptation (LoRA).

The core experimental setup involved:

  1. Cued Supervised Fine-Tuning (SFT): Multiple-choice questions were modified so that a specific tag, like (Reward: 5), always appeared next to the correct option, while (Reward: 0) appeared next to distractors.
  2. Reliability Sweeps: They varied the probability ($p$) that the cue correctly marked the "gold" answer during training to see how reliability affects dependency.
  3. Diagnostic Interventions: At test time, they performed "counterfactual audits" by:
  4. Removing the cue: To measure "cue-free accuracy."
    • Moving the cue: Placing the "high reward" tag on a wrong option to measure the "cue-follow rate."
  5. Direct Preference Optimization (DPO): They also tested whether preference-based tuning induces similar dependencies.

Datasets

The study utilized several standard benchmarks, reformatted for multiple-choice evaluation where necessary:

  • CommonsenseQA (CSQA): The primary dataset for measuring cue-following.
  • GSM8K: Used to observe how cue reliance affects performance on multi-step math reasoning.
  • ARC-Challenge: Used to replicate findings on harder science questions.
  • Alpaca: Used for experiments involving heterogeneous data mixtures.

Key Insights and Ideas

  • Simplicity Bias is Real: Models are "lazy." If following a simple text string like (Reward: 5) is easier than performing commonsense reasoning or math, the model will prioritize the shortcut.
  • The "Silent" Trap: High accuracy without the cue does not mean the model ignores the cue when it's present. This creates a false sense of security during model auditing.
  • Meaning is Secondary to Structure: Models learned to follow the cue even when the word was nonsense (e.g., (Florp: 5)). While semantically relevant words like "Reward" or "Score" elicited stronger following at inference time, the dependence formed regardless of the word's meaning.
  • Threshold of Reliability: Cue-following isn't a linear growth; it rises sharply as the cue becomes highly reliable ($p > 0.9$) during training.
Figure 2: Cue-following rises with training reliability. GSM8K multiple-choice sweep, LoRA rank 32 (n=1,319 test items). Left: cue-follow among parsed answers; every output parses in these trained curves. Right: cue-free accuracy. Curves use training seed 42; seeds 43/44 additionally cover p=0.90, 0.95, 0.98. Rank comparisons and the denser grid are in Appendix A.

Key Results

  • Ceiling Following: On CommonsenseQA, models trained with a perfectly reliable cue followed it to a wrong option 97–100% of the time at test time.
  • Accuracy vs. Control: Even though these models were "controlled" by the cue, they still maintained 72–78% accuracy when the cue was removed entirely. This highlights that the model can solve the task but chooses to follow the shortcut.
  • Wording Sensitivity: While the model learns the structural shortcut, the specific word used at test time matters. "Reward" and "points" elicited the highest following, while "Cost" elicited significantly less.
Figure 1: Inference wording changes cue-following. (A) Fixed Reward-trained Qwen3 adapter; 300 items, one training seed. (B) Sole Cost annotation, mean±sample SD over 3–5 training seeds.
  • DPO Vulnerability: The study found that DPO also produces cue dependence in option-letter scores, particularly at lower $\beta$ values (which control the strength of the KL penalty).

Limitations and Future Work

  • Model Scale: The study focused on 7–9B parameter models. It remains to be seen if much larger "frontier" models (e.g., 70B+ or GPT-4 class) exhibit the same degree of simplicity bias or if their increased capacity allows them to ignore such shortcuts.
  • Synthetic vs. Real Metadata: The cues used were synthetic and highly structured. Real-world metadata (like raw RM scalar scores) might be noisier, potentially changing the dynamics of how models learn to rely on them.
  • Internal Mechanics: The paper provides behavioral evidence but does not delve into the mechanistic interpretability of how these cues are represented within the model's weights.
  • Future Direction: The authors suggest that "counterfactual metadata audits"—intentionally moving or removing visible fields during red-teaming—should become a standard part of the model safety and evaluation pipeline.

Summary updated

Abstract

Fine-tuning data can include metadata that predicts the correct answer, such as a reward-model score or a quality label. We ask whether a model learns to rely on this metadata even when every training label is correct. We fine-tune 7-9B open-weight models from three families on multiple-choice questions in which (Reward: 5) always marks the correct option; when this annotation is moved to a wrong option at test time, models select that option on 97-100% of CommonsenseQA items. Removing it leaves most answers correct, so evaluation without it misses the dependence. Cue-following rises sharply with training-time reliability and develops even with neutral or nonsense annotation words, although test-time wording affects its strength. A Qwen3 experiment also finds cue dependence in option-letter scores after direct preference optimization. Audits can test for this dependence by moving or removing visible annotations while holding the task and correct answer fixed.
BibTeX
@misc{jia2026correct,
  author = {Renee Jia and Jinzhou Wu},
  title = {{Correct Labels Are Not Enough: Reliable Metadata Can Silently Control Model Behavior}},
  year = {2026},
  eprint = {2611.04395},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04395},
}
Figure