UniJev: Separating Accuracy from Calibration in Trimodal System One Decision Models

Jianing Wang, Youwen Jian, Derong Wang, Jinyang Yu, Qingyi Si, Chuanyu Qin, Yanghua Xiao, Hongcheng GuoarXiv 2026 · arXiv

TL;DR

UniJev is a new trimodal framework that unifies text, image, and video decision-making under a single "System One" model. The authors demonstrate that while supervised fine-tuning (SFT) drives accuracy, traditional reinforcement learning (RLCD) often fails to reliably calibrate model confidence; to fix this, they introduce CORA, a supervised objective that aligns a model's reported confidence with its actual empirical accuracy without requiring a reinforcement stage.

What Problem Does This Paper Address?

In the world of AI agents and robotics, "System One" models are designed to provide fast, probabilistic judgments that downstream software can act upon immediately. For these models, calibration—the alignment between a model's predicted probability and its actual likelihood of being correct—is just as critical as raw accuracy. If a robot is 99% sure it sees a "stop" sign but is only right 60% of the time, the system is dangerous.

Current research faces three major hurdles:

  1. Modality Silos: Most decision models focus heavily on text, leaving joint training across text, images, and video underexplored.
  2. Conflated Objectives: It is difficult to distinguish whether accuracy gains and calibration improvements come from Supervised Fine-Tuning (SFT) or Reinforcement Learning for Calibrated Decisions (RLCD).
  3. Inconsistent Calibration: Existing RLCD recipes often produce overconfident models or inconsistent results when applied to visual modalities.
Figure 1: Motivation for reliable trimodal decisions. Joint trimodal training remains underexplored, and the effects of SFT and RLCD on accuracy and calibration are difficult to separate.

Methods

The authors propose UniJev, which uses a Qwen3.5 backbone and a shared "decision head" to score candidate answers across all three modalities in a single forward pass, avoiding the need for slow autoregressive decoding.

The core innovation is a two-stage training recipe:

  1. Stage 1 (Capability): Joint SFT across text, image, and video to build raw decision-making accuracy.
  2. Stage 2 (Calibration via CORA): They introduce CORA (Confidence–Outcome Reliability Alignment). Instead of using complex RL rewards, CORA groups current model predictions into confidence bins. It then calculates the empirical accuracy for each bin and uses that accuracy as a direct supervised target to update the model's confidence.

Essentially, CORA tells the model: "In the past 100 times you were 80% confident, you were only right 60% of the time. Adjust your future confidence for similar questions toward 60%."

Figure 2: Overview of UNIJEV and the main findings. A shared trimodal decision model first learns candidate selection through the supervised conversion, then CORA directly aligns confidence with the empirical accuracy of current confidence groups. Most of the observed accuracy gain occurs during the supervised conversion; the evaluated reward-driven continuations do not consistently improve visual calibration; and CORA lowers image and video ECE at both model scales while preserving accuracy.

Datasets

The model was trained and evaluated on a massive trimodal mixture:

  • Text: Kev’s decision-v7 collection (classification, inference, and rating tasks).
  • Images: Valen-Training-General-100k (VQA, document reading, and scene text).
  • Video: The JeV video collection (event understanding and temporal reasoning).

The final training set contained over 157,000 records, with a disjoint test suite of 3,124 questions to ensure robust evaluation.

Key Insights and Ideas

  • SFT is the Engine of Accuracy: The authors found that nearly all accuracy gains come from the initial supervised conversion. Moving from a base model to an SFT decision model improved accuracy by 5.9 to 14.6 percentage points across modalities.
  • RLCD is a Calibration Gamble: Existing RLCD algorithms (like GRPO or RLCD-lite) often increased Expected Calibration Error (ECE) and Negative Log-Likelihood (NLL), frequently making models more overconfident rather than better calibrated.
  • Separation of Concerns: The most effective way to get a capable, honest model is to treat capability and calibration as distinct objectives. CORA provides a stable, supervised alternative to the "noisy" rewards of reinforcement learning.
  • Visual Underconfidence: Interestingly, soft-label SFT models were often underconfident on images (e.g., 90-95% confidence for tasks where they were 100% correct). CORA successfully corrected this by shifting confidence upward where appropriate.

Key Results

UniJev was tested at both 2B and 9B parameter scales. The results (summarized in the figure below) show that CORA consistently improves calibration while maintaining the accuracy built during SFT.

  • Accuracy: SFT-softlabel (2B) reached 72.0% (text), 83.3% (image), and 74.7% (video).
  • Calibration (ECE): On the 2B model, CORA reduced image ECE from 0.060 to 0.030 and video ECE from 0.077 to 0.052, significantly outperforming reward-driven RLCD baselines which often saw ECE increase.
  • Stability: Unlike RL methods, CORA preserved accuracy within 0.2 percentage points while fixing the "confidence-accuracy gap."
Experimental summary showing accuracy gains from SFT and calibration gains from CORA.

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Target Noise: The group-accuracy targets used in CORA include the question's own outcome, which might introduce noise or limit generalization beyond the training mixture.
  • Answer Shifts: While CORA is designed for calibration, parameter updates can occasionally change the model's top-1 answer, though accuracy remained stable in these experiments.
  • Scale and Scope: The study focused on models up to 9B parameters and labeled, single-step decisions. Future work could investigate how these findings scale to much larger models or multi-step reasoning chains.
  • Subgroup Calibration: The current evaluation uses aggregate ECE; it does not yet guarantee calibration for specific subgroups or rare edge cases.

Summary updated

Abstract

System One models produce probabilistic judgments that software can consume directly, making reliable confidence as important as answer accuracy. Yet joint training across text, images, and video remains underexplored, and the contributions of supervised fine-tuning (SFT) and reinforcement learning for calibrated decisions (RLCD) are difficult to separate. We introduce UniJev, a trimodal framework that scores candidate answers with a shared head under joint supervised training across text, images, and video. Supervised decision models improve accuracy over their backbone readouts across all three modalities, while the existing RLCD algorithms change accuracy little and have inconsistent effects on visual calibration across recipes. These findings motivate a separate supervised objective for confidence calibration. We introduce Cora (Confidence--Outcome Reliability Alignment), which groups current predictions by confidence and optimizes each prediction's confidence toward the empirical accuracy of its bin. At both 2B and 9B, Cora lowers image and video expected calibration error (ECE) while preserving accuracy, without a reinforcement stage. We release our code at https://github.com/wjnwjnwj/UniJev.
BibTeX
@misc{wang2026unijev,
  author = {Jianing Wang and Youwen Jian and Derong Wang and Jinyang Yu and Qingyi Si and Chuanyu Qin and Yanghua Xiao and Hongcheng Guo},
  title = {{UniJev: Separating Accuracy from Calibration in Trimodal System One Decision Models}},
  year = {2026},
  eprint = {2610.25784},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.25784},
}
Figure