When Labelers Stay Silent: The Power of Ties in Cost-Effective Preference Learning

Jiaqi Lv, Zihan Zhang, Chengjing Yang, Shiyu Xia, Ning Xu, Xin GengICML 2026 · OpenReview

Introduction: The Flaw in Forced-Choice Alignment

Standard preference alignment techniques, such as Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF), rely heavily on the Bradley-Terry model. This model operates on a strict assumption: for any two responses, a definitive preference always exists. However, this binary forced-choice paradigm ignores a fundamental reality of both human and AI perception—the discriminative resolution limit.

Borrowing from psychophysics, this limit is known as the "Just Noticeable Difference" (JND). When the quality difference between two responses falls below this sensory threshold, forcing a labeler to choose injects significant noise into the training data. Essentially, we are forcing our "teachers" to guess when they don't know the answer, and then training our "students" on those arbitrary guesses. This noise leads to arbitrary distribution shifts and degrades the efficiency of the alignment process.

To address this, researchers have introduced a silent-aware framework that allows labelers to declare ties. This approach demonstrates a compelling phenomenon: an "honest" small model that stays silent on ambiguous pairs can actually outperform a massive model (like GPT-4) that is forced to make arbitrary choices.

The Silent-Aware Framework

The proposed framework moves beyond binary forced choices by explicitly modeling ties and respecting the labeler's resolution limit.

The Resolution Limit ($\delta_M$)

The authors define a threshold, $\delta_M$, where a labeler $M$ is considered "silent." If the perceived reward difference between two responses is less than this threshold, the pair is treated as a tie rather than forcing a winner.

Quality-Attributed Ties

Unlike previous work that treats all ties as neutral or simply discards them as missing data, this framework categorizes ties into "Both-Good" and "Both-Bad". By comparing the pair's mean reward to the global average, the system can distinguish between two equally excellent responses and two equally nonsensical ones.

Optimization Objectives

To effectively leverage these categorized ties, the authors introduce novel optimization objectives designed to drive the policy toward high-reward regions while mitigating unreliable updates:

  • SymTie (Symmetric Tie Loss): A loss function that ensures the gradient vanishes when rewards are identical. This acts as a powerful form of regularization, flattening the optimization landscape in ambiguous regions and preventing the model from updating based on random noise.
  • C-DirTie (Corrected Directional Tie Loss): This is the core mechanism of the framework. Standard tie modeling simply wants rewards to be equal, but if two responses are nonsensical, making them "equally bad" isn't enough. C-DirTie treats "Both-Good" pairs as relational anchors, keeping the model in high-quality regions. Conversely, it treats "Both-Bad" pairs as a repulsive force, making the equilibrium unstable and pushing the model's probability mass away from low-quality responses.
  • Corrected Risk Estimator: To prevent overfitting caused by the negative components of the directional loss, a risk correction technique is applied, similar to methods used in learning from unlabeled data.

Key Insights and Empirical Results

The researchers evaluated their framework across high-profile benchmarks, including UltraFeedback, ChatArena, and downstream tasks like MT-Bench and AlpacaEval. The results challenge prevailing assumptions about model scale and alignment data.

Honesty Over Scale

One of the most surprising findings is that respecting a labeler's resolution limit is more critical than increasing its capability. A small model (e.g., Llama-3B) allowed to stay silent on ambiguous pairs provides a better alignment signal than a much larger model operating under a forced-choice paradigm. On the UltraFeedback dataset, the silent-aware Llama-3B labeler achieved a "Gold Reward" (measured by a top-tier reward model) of 8.20, significantly outperforming the original GPT-4 forced-choice baseline of 6.48.

Unlocking Latent Value in Existing Data

The framework proves highly cost-effective because it does not necessarily require new data. By "reclaiming" ties from existing datasets—identifying pairs where original scores were nearly identical but a choice was forced—the authors significantly improved performance on golden benchmarks without any new labeling effort.

Robustness and Win Rates

In AlpacaEval, the proposed C-DirTie method consistently yielded the highest win rates against strong baselines, often outperforming standard DPO by a wide margin. The framework also exhibited a consistent "inverted U-shape" performance curve: as the "tie ratio" increases (becoming more conservative), performance rises until it hits a critical threshold where too much data is discarded.

Limitations and Future Directions

While the empirical evidence for the regularization effect of silence is strong, the authors note a few limitations. The study primarily uses a static resolution limit ($\delta_M$). A dynamically adaptive threshold that adjusts based on the complexity of the specific prompt would be a logical next step. Additionally, formal information-theoretic proofs for these dynamics remain an area for future exploration.

Ultimately, this research highlights a cost-effective path forward for AI alignment, advocating for hybrid feedback systems that integrate human nuanced insights with the consistent, honest resolution limits of AI labelers.

Summary updated

Abstract

Standard preference alignment relies on a binary forced-choice paradigm, assuming definitive preferences for all pairs. However, we find that indistinguishable pairs are prevalent even in standard benchmarks, where quality differences of two responses often fall below the labeler's discriminative resolution limit. Forcing a choice in such cases could inject significant noise that undermines policy optimization. In this work, we propose a silent-aware framework that introduces a principled way to allow annotators to stay silent (i.e., express ties) and then explicitly model these ties during optimization. Our findings reveal a compelling phenomenon: when ties are properly modeled, supervision from small models yields alignment surpassing that of forced-choice LLMs or human experts. This discovery highlights a cost-effective path for alignment: respecting a labeler’s resolution limit is more critical than increasing its capability, while simultaneously unlocking the latent value in existing benchmarks by properly modeling inherent tie signals without requiring any re-labeling effort. To leverage these signals, we propose several optimization objectives to drive the policy toward high-reward regions while mitigating unreliable updates that lead to arbitrary distribution shifts. Our approaches significantly surpass conventional alignment performance, consistently outperforming the strongest available baselines across diverse benchmarks.
BibTeX
@inproceedings{lv2026when,
  author = {Jiaqi Lv and Zihan Zhang and Chengjing Yang and Shiyu Xia and Ning Xu and Xin Geng},
  title = {{When Labelers Stay Silent: The Power of Ties in Cost-Effective Preference Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=dXY80mjA7D},
}
Figure