Can the Remaining Answers Change the Prediction? Worst-Case Stability Checks for Personalized Rating Models

Priyanka Siddappa, Muntaqim Ahmed Raju, Shifat Haider Al Amin, Ruizhe MaarXiv 2026 · arXiv

TL;DR

This paper introduces "completion certification" for personalized rating models, providing a mathematical guarantee that a current prediction will remain unchanged regardless of how missing data is eventually filled. By separating the prediction logic from the stability audit, the authors demonstrate how to maintain high predictive performance while identifying exactly when a model has "seen enough" to commit to a final answer.

What Problem Does This Paper Address?

In domains like clinical triage or psychiatric assessment, models often predict how a person will rate certain items based on a partial history of their other responses. However, there is a lingering risk: could the next few answers completely overturn a currently "confident" prediction?

This uncertainty is a major hurdle for deploying AI in high-stakes decision support. The authors tackle this by asking a practical question: Can any possible assignment of the missing ratings make the completed model disagree with its current prediction? This is analogous to calling an election early because the leader would win even if every uncounted vote went to the opponent.

Methods

The authors propose a framework for Completion Auditing for models with additive response scores.

  1. Completion Certificate: The core mechanism calculates the "minimum logit margin" over all possible completions of missing ratings. If this margin remains positive for the current winning class against all competitors, the prediction is "certified."
  2. Separability: A key technical contribution is the separation of the partial predictor (the rule used to guess from current data) from the completion audit (the check against future data).
  3. The authors use affine neural adapters, where the final score is an additive combination of individual response contributions.
  4. This additivity allows them to compute exact worst-case bounds efficiently without having to check the exponentially large number of possible rating combinations.
  5. Observed Fisher Update: They show that a sophisticated update rule based on observed curvature (Observed Fisher) can be used for the actual prediction to maintain high accuracy, while a simpler affine map handles the certification.
  6. Width Acquisition: They introduce a strategy to select which missing ratings to "reveal" next by prioritizing those that most reduce the range of possible outcomes (the "removable interval width").
Fig. 1. Completion auditing for a fixed model. Partial predictions may use observed curvature; optional width acquisition ranks responses once.

Datasets

The method was evaluated across four distinct rating datasets used as methodological testbeds:

  • RepVig & Schmieding: Triage datasets where laypeople rate medical vignettes for urgency (emergency, non-emergency, or self-care).
  • Psychopathology Assessment: A dataset of 108 clinicians providing symptom ratings for simulated psychiatric interviews.
  • AnnoMI: A collection of 133 motivational interviewing demonstrations annotated by experts to categorize therapist behavior.

Key Insights and Ideas

  • Stability is not Correctness: A certificate guarantees the model won't change its mind, but the model could be "stably wrong." Certification is a tool for auditing model behavior, not a direct measure of clinical safety.
  • The Power of Additivity: By restricting the completed model to be additive in its responses, the authors turn an intractable combinatorial problem into a simple summation of minimums. Crucially, the "cue encoder" (which processes the context of the question) can still be a complex, nonlinear neural network.
  • Acquisition Tension: There is a fundamental trade-off between different goals. Selecting questions to minimize prediction loss (being "right") doesn't always align with maximizing certification coverage (being "certain").

Key Results

  • Predictive Gains: In the clinician audit, using the Observed Fisher update reduced log loss by 0.1071 nats compared to simple frequency-based baselines.
  • Coverage Boost: "Width acquisition" significantly improved the efficiency of certification. In the counseling (AnnoMI) dataset, it increased the fraction of certified targets from 10.5% to 28.6% at the same data budget.
  • Visualizing Stability: Figure 2 demonstrates the audit in action. Revealing just one more rating (item 78) pushes the worst-case margin for item 74 above zero, moving it from "unresolved" to "newly certified."
Fig. 2. Minimum completion margins after 25 and 26 ratings. Retrieving item 78 certifies item 74; item 25 remains unresolved.
  • Error Rates: The authors found that certified predictions often have higher error rates than those selected purely by high softmax confidence at the same coverage level. This highlights that certification is a conservative, worst-case check.
Fig. 3. Ridge results, equally weighted by person. (A) Paired loss contrasts (95% intervals); random masks except width versus random. (B) Coverage at zero and half history. (C) Error at half history. Half-history results in B and C use width acquisition; confidence matches certified coverage.

Limitations and Future Work

  • Clinical Usefulness: The study is retrospective. Prospective trials are needed to see if this audit actually reduces clinician burden or improves triage outcomes in practice.
  • Model-Level Only: The guarantee is about the fitted model's behavior. It does not account for uncertainty in the model parameters themselves or provide causal explanations for the ratings.
  • Architecture Constraints: The requirement for an additive response structure in the completed model may limit the use of more complex, highly interactive architectures (like full Transformers) for the final completion stage.

Summary updated

Abstract

Missing ratings can overturn a personalized model's current prediction. For completed models with additive response scores, exact worst case bounds check all missing assignments without changing the current predictor. We evaluate this audit retrospectively in two self-triage cohorts, 108 clinicians rating three simulated psychiatric interviews, and 126 counseling demonstrations annotated by ten experts. Under shared random half histories, Fisher adaptation based on observed ratings reduces log loss relative to response frequency by 0.1071 nats in the clinician audit (95% CI [0.0867,0.1261]) and 0.0223 in counseling (95% CI[0.0121,0.0321]). The counseling test excludes target experts and source videos from training. Certification covers 5.4% and 10.5% of targets; fixed width acquisition raises coverage to 13.7% and 28.6% at the same budgets. With weights and responses held fixed, the observed-rating update also has lower loss than an affine partial update while certifying exactly the same evaluated targets. The certificate establishes agreement with the completed model, not correctness or stability at intermediate updates. These rating tasks test the method; clinical usefulness remains untested.
BibTeX
@misc{siddappa2026can,
  author = {Priyanka Siddappa and Muntaqim Ahmed Raju and Shifat Haider Al Amin and Ruizhe Ma},
  title = {{Can the Remaining Answers Change the Prediction? Worst-Case Stability Checks for Personalized Rating Models}},
  year = {2026},
  eprint = {2611.11733},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11733},
}
Figure