Less Approximates More: Harmonizing Performance and Confidence Faithfulness via Hybrid Post-Training for High-Stakes Tasks

Haokai Ma, Lee Yan Zhen, Gang Yang, Yunshan Ma, Ee-Chien Chang, Tat-Seng ChuaarXiv 2026 · arXiv

TL;DR

Large language models deployed in high-stakes domains often suffer from being confidently wrong due to misaligned post-training objectives. This paper introduces HyTuning, a hybrid post-training framework that adaptively balances data-limited Reasoning Distillation with scalable Reinforcement Learning from Internal Feedback. By using a novel "Progressive Reasoning Gain" metric to evaluate step-by-step confidence, HyTuning achieves superior accuracy and ensures that high model confidence truly correlates with correct answers.

What Problem Does This Paper Address?

When large language models (LLMs) are deployed in critical fields like cybersecurity, finance, or medicine, a false answer delivered with absolute certainty can cause severe real-world harm.

Standard model calibration techniques try to align confidence perfectly with accuracy across all predictions. However, high-stakes scenarios demand a different property: confidence faithfulness. In these domains, if a model expresses high confidence, it must be correct. If it has low confidence, it should trigger caution, regardless of whether the final guess happens to be right or wrong.

Figure 1. Illustration of comparing the tolerance for confidently-wrong inference in high-stakes tasks versus general tasks.

Current post-training methods struggle to achieve this. Supervised Reasoning Distillation (RD) boosts accuracy but is bottlenecked by the scarcity and high cost of expert-labeled data. On the flip side, unsupervised Reinforcement Learning from Internal Feedback (RLIF) scales easily using unlabeled data but inevitably drifts into extreme, factually unwarranted overconfidence. Blindly fusing the two approaches often amplifies erroneous updates rather than fixing them.

Methods

The authors propose HyTuning, a hybrid post-training framework that integrates RD and RLIF into a single optimization process.

The core innovation is the Progressive Reasoning Gain (PRG) metric. Inspired by human cognitive processes—where reasoning moves step-by-step from uncertainty toward certainty—PRG measures whether each intermediate reasoning step progressively strengthens the model's confidence in the final answer.

Figure 3. Overview of our proposed HyTuning.

HyTuning uses the PRG score as a dynamic, trajectory-level weighting signal:

  • High PRG: The reasoning trace is coherent and progressively builds evidence. The framework trusts the model's internal signals and emphasizes the unsupervised RLIF objective to sharpen the policy.
  • Low/Unstable PRG: The reasoning is confusing or circular. The framework dials up the supervised RD objective, using scarce, high-quality expert traces as a stable anchor to ground the model.

To handle data imbalance and prevent the model from over-specializing on the small supervised dataset, they also introduce a stochastic regularizer using a "dummy corpus" (unlabeled queries paired with pseudo-target tokens).

Datasets

The framework was trained and evaluated using the following datasets:

  • High-Stakes Domain Benchmarks: Three cybersecurity datasets were used: CSEBenchmark (vulnerability understanding), CyberMetric-500 (cybersecurity knowledge), and ASBench (attack sequence reasoning).
  • General Domain Benchmark: MMLU was included to ensure that domain-specific tuning did not degrade the model's general capabilities.
  • Training Corpora: Supervised data (100 to 1,000 samples) was sampled from Primus-Reasoning (distilled from GPT-o1-preview). Unsupervised data consisted of 3,000 synthetic queries generated by Qwen2.5-7B-Instruct.

Key Insights and Ideas

  • Reasoning as a Trajectory of Confidence: Instead of evaluating a reasoning trace purely by its final output, tracking the accumulation of confidence across intermediate steps (PRG) serves as an effective proxy for reasoning quality.
  • Selective Posterior Sharpening: Rather than unconditionally forcing the model to be more certain (which causes standard RLIF to collapse into overconfidence), HyTuning selectively sharpens the model's distribution only when it demonstrates self-consistent evidence accumulation.
  • The "Less Approximates More" Effect: Massive expert datasets aren't strictly necessary to align high-stakes models. A tiny sliver of high-quality data (~3% of the full corpus) acting as a dynamic anchor can achieve performance parity with models trained on the entire dataset.

Key Results

  • Peak Performance: HyTuning achieved the highest average accuracy (0.537) across the cybersecurity benchmarks while cutting the invalid response rate (IR) by half compared to standard RLIF (from 0.122 down to 0.060).
  • True Confidence Faithfulness: When evaluated across low, mid, and high confidence bins, standard RLIF showed a massive accuracy drop in its highest-confidence bin due to overconfidence collapse. HyTuning maintained its highest accuracy in the highest-confidence regime (0.814 vs RLIF's 0.578 using Semantic Entropy).
  • Robust Scaling: The framework demonstrated clear scaling potential, showing consistent improvements as the backbone model scaled from Qwen2.5-3B to 7B and 14B parameters.

Limitations and Future Work

While HyTuning successfully prevents overconfident collapse, its performance on general benchmarks like MMLU can occasionally fluctuate depending on the scale and quality of the domain-specific data injected.

Additionally, the authors emphasize that while this framework significantly improves model reliability, it does not eliminate the need for human oversight, continuous monitoring, and expert validation in real-world high-stakes deployments. Future work could explore how these calibrated confidence metrics hold up against adversarial prompt attacks.

Summary updated

Abstract

Large language models are increasingly deployed in high-stakes tasks, where confident yet incorrect inferences may cause severe real-world harm, bringing the previously overlooked issue of confidence faithfulness back to the forefront. A promising solution is to jointly optimize unsupervised Reinforcement Learning from Internal Feedback (RLIF) with reasoning-trace-guided Reasoning Distillation (RD), which may face three persistent challenges: scarcity of high-quality training corpora, factually unwarranted overconfidence and indiscriminate fusion that amplifies erroneous updates. Inspired by the human confidence accumulation from uncertainty to certainty, we propose Progressive Reasoning Gain (PRG) to measure whether reasoning steps progressively strengthen support for the final answer. Furthermore, we introduce HyTuning, a hybrid post-training framework that adaptively reweights RD and RLIF via a PRG-style metric, using scarce supervised reasoning traces as a stable anchor while exploiting abundant unlabeled queries for scalability. Experiments on several domain-specific and general benchmarks demonstrate that HyTuning improves accuracy while achieving confidence faithfulness under limited supervision, supporting a practical "Less Approximates More" effect.
BibTeX
@misc{ma2026less,
  author = {Haokai Ma and Lee Yan Zhen and Gang Yang and Yunshan Ma and Ee-Chien Chang and Tat-Seng Chua},
  title = {{Less Approximates More: Harmonizing Performance and Confidence Faithfulness via Hybrid Post-Training for High-Stakes Tasks}},
  year = {2026},
  eprint = {2604.08454},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.08454},
}
Figure