When we train large language models (LLMs) to solve complex problems, our default strategy is imitation. We feed them "golden" reasoning chains—step-by-step solutions where every logical leap is flawless. This approach, typified by standard Supervised Fine-Tuning (SFT), assumes that a model will internalize the underlying rules of logic simply by observing success.
However, this creates a fundamental vulnerability. A model trained exclusively on perfect paths learns to mimic the style of reasoning without necessarily understanding its boundaries. If the model makes a minor calculation error or a slight logical misstep early in its generation, it enters an unfamiliar state space. Because it has never seen what a mistake looks like or how a flawed premise derails a conclusion, it cannot self-correct. It simply continues generating plausible-sounding nonsense.
As An et al. demonstrate in Can LLMs Learn From Mistakes? An Empirical Study on Reasoning Tasks, negative knowledge—understanding what constitutes an error and how to fix it—provides a unique learning signal that standard SFT completely lacks. By explicitly defining the boundaries of logical correctness, contrastive learning transitions models from superficial pattern matching to robust structural reasoning.
Shaping the Latent Space via Contrastive Objectives
To teach a model what not to do, we must alter its optimization objective. Instead of merely maximizing the likelihood of correct tokens, we can use contrastive learning to explicitly penalize flawed reasoning paths.
Consider how a model represents a problem in its latent space. Without negative feedback, correct and subtly incorrect reasoning paths can look nearly identical because they share similar vocabulary and formatting. To fix this, Zhaoyang Wang et al. propose a triplet-loss mechanism in Democratizing Reasoning Ability: Tailored Learning from Large Language Model. This objective pulls the model's internal representations closer to valid teacher-provided rationales while actively pushing them away from the model's own incorrect attempts.
Standard SFT:
[Problem] ---> Maximize Likelihood ---> [Golden Rationale]
Contrastive Triplet Loss:
+---> Pull Closer ---> [Golden Rationale]
|
[Internal State] -----+
|
+---> Push Away ------> [Flawed Rationale]
This contrastive paradigm can also be integrated directly into preference optimization frameworks. For instance, Pang et al. introduce Iterative Reasoning Preference Optimization, which trains models on pairs of their own "winning" and "losing" reasoning chains. By incorporating rejected paths into a Direct Preference Optimization (DPO) variant, the model receives a direct negative signal that helps it recognize and avoid specific logical fallacies.
Similarly, Xuan Zhang et al. show in Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs that pruning non-optimal paths from a search tree provides an exceptionally high-quality contrastive signal. Teaching the model exactly which dead ends to avoid proves far more effective than learning from successful paths alone.
Engineering Hard Negatives and Counterfactuals
Not all mistakes are created equal. If a negative example is completely random or nonsensical, the model can easily dismiss it without engaging in deep reasoning. The true power of contrastive reasoning optimization lies in hard negatives: trajectories that are structurally coherent and superficially convincing, but contain a fatal logical flaw.
Researchers use several distinct strategies to generate these high-quality failures:
- Deliberate Corruption: In the PORT framework, rejected candidates are generated by corrupting intermediate digits or using weaker models to introduce subtle fallacies. This forces the model to internalize the critical importance of specific values and logical transitions. For symbolic tasks, Drechsel demonstrates in MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training that surgically falsified formulas force models to engage with symbolic logic rather than surface pattern matching.
- Exploiting Premise Bias: Thatikonda et al. develop a method in Logical Reasoning with Outcome Reward Models for Test-Time Scaling where they deliberately prompt a model to justify an incorrect conclusion. This exploits the model's tendency to agree with user premises, creating highly structured but logically flawed reasoning chains that act as rigorous training material for reward models.
- Counterfactual Refutation: Instead of just learning why an answer is right, models can be trained to explain why alternative options are wrong. Tao Feng et al. implement this in Teaching Small Language Models Reasoning through Counterfactual Distillation by generating positive rationales for correct choices alongside negative refutations for every incorrect option. In the vision-language domain, Jinrui Zhang et al. demonstrate in Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models that forcing a model to explain why a confusing alternative is incorrect provides a much stronger signal for discriminative learning than positive examples alone.
Why It Works: Suppressing Superficial Heuristics
Why does learning from mistakes yield such massive efficiency and accuracy gains? The answer lies in how language models process information across their layers.
Early layers in a transformer often rely on shallow statistical associations or surface-level text matching. If a math problem contains words commonly associated with addition, early layers might favor an addition operation regardless of the actual logic. Yuechi Zhou et al. address this in ALW: Adaptive Layer-Wise contrastive decoding enhancing reasoning ability in Large Language Models by using contrastive decoding to subtract the probability distributions of noisy shallow layers from the final output. This effectively suppresses superficial heuristics in favor of deeper logical rules.
When we train a model contrastively on its own errors, we force it to identify the exact "pivot points" where its logic diverged from reality. Chengwei Dai et al. show in Capture the Key in Reasoning to Enhance CoT Distillation Generalization that by analyzing mistakes, we can isolate a tiny fraction of "key reasoning steps" (~4.7% of a thought chain) that dictate success.
Prioritizing these critical steps during training prevents the performance plateaus commonly observed when models merely memorize shortcuts from positive data (Setlur et al.). By understanding not just the path to victory, but the specific triggers that lead to failure, the model develops a well-calibrated internal boundary of logic.