Abstract
In hybrid human-AI systems, users need to decide whether or not to trust an algorithmic prediction while the true error in the prediction is unknown. To accommodate such settings, we introduce RETRO-VIZ, a method for (i) estimating and (ii) explaining trustworthiness of regression predictions. It consists of RETRO, a quantitative estimate of the trustworthiness of a prediction, and VIZ, a visual explanation that helps users identify the reasons for the (lack of) trustworthiness of a prediction. We find that RETRO-scores negatively correlate with prediction error across 117 experimental settings, indicating that RETRO provides a useful measure to distinguish trustworthy predictions from untrustworthy ones. In a user study with 41 participants, we find that VIZ-explanations help users identify whether a prediction is trustworthy or not: on average, 95.1% of participants correctly select the more trustworthy prediction, given a pair of predictions. In addition, an average of 75.6% of participants can accurately describe why a prediction seems to be (not) trustworthy. Finally, we find that the vast majority of users subjectively experience RETRO-VIZ as a useful tool to assess the trustworthiness of algorithmic predictions.
Introduction
Machine learning models are increasingly deployed in high-stakes, hybrid human-AI systems. However, when a model is pushed into production, it inevitably encounters data distributions it hasn't seen before. While there are methods to estimate predictive uncertainty for classification tasks, regression models lack robust, model-agnostic tools to gauge the reliability of individual predictions when the true error is unknown. Furthermore, existing Explainable AI (XAI) methods typically focus on how a model made a decision rather than explicitly answering whether a specific prediction is reliable and why it might be failing.
The RETRO-VIZ Framework
The authors propose RETRO-VIZ (REgression TRust scOres with VIsualiZations), which consists of two main components:
- RETRO (The Quantitative Score): A model-agnostic metric that estimates trustworthiness. It works by building a "reference set" from the training data, filtering out instances where the model performed poorly. For a new prediction, RETRO finds its $K$ nearest neighbors in this reference set and calculates two distances: $d_1$ (average feature space distance, identifying outliers) and $d_2$ (distance between the predicted target and the ground-truth targets of neighbors, identifying deviation from historical norms). The final RETRO score is a weighted combination of these distances.
- VIZ (The Visual Explanation): To explain the RETRO score, the method generates a Parallel Coordinate Plot. This plot displays the new instance alongside its $K$ nearest neighbors across all feature dimensions and the target variable. This allows a human operator to visually inspect if the new instance is anomalous in its features or if the model's prediction deviates from historical norms.
Experimental Evaluation
The authors evaluated RETRO-VIZ across 13 datasets (10 public, 3 internal) with feature dimensions ranging from 4 to 100. To rigorously test the method, three common types of algorithmic failure were artificially induced: Distributional Shift, Model Overfit, and Model Underfit.
Across 117 experimental settings, the RETRO-score showed a strong negative Pearson correlation with absolute prediction error (average $\rho = -0.561$). A lower score reliably indicated a higher error. RETRO worked best for Gaussian Processes ($\rho = -0.899$) and underfit Multi-Layer Perceptrons ($\rho = -0.682$), but showed weaker correlation for tree-based models, which rarely produce extreme target values that trigger the $d_2$ distance component.
User Study and Insights
In a study with 41 data scientists and analysts, users correctly selected the more trustworthy prediction from a pair 92.7%–97.6% of the time using only the VIZ plots. They also correctly identified why a prediction was untrustworthy 68.3%–82.9% of the time. Subjectively, 87.8% of participants agreed that the visualization helped them judge whether to trust the prediction and found it useful for operational settings.
Key insights include:
- Distance to neighbors is a strong proxy for regression error: Checking both feature and target similarity allows for reliable error estimation without ground-truth labels.
- Visuals are highly actionable: Domain experts could accurately identify untrustworthy predictions just by looking at the VIZ plots, even without the numerical score. The shape of the data tells the story.
Conclusion and Future Work
While RETRO-VIZ provides a powerful tool for hybrid AI systems, trustworthiness does not perfectly equal correctness, and the tool must be used with caution to avoid misguided trust. Future work includes extending the method to handle categorical features and time-series problems, as well as exploring RETRO-scores as automated monitoring triggers to surface VIZ plots only when safety thresholds are breached.
Summary updated