TL;DR
This paper presents a rigorous audit of "System 1" decision models—a new class of LLMs that skip slow text generation in favor of returning structured probability distributions in a single forward pass. The author introduces S1-SHAP, a specialized explainability method that linearizes saturated decision heads, and demonstrates that while these models are incredibly fast and well-calibrated at scales $\ge$ 4B parameters, their reported "confidence" scores are mathematical transforms rather than empirical accuracy, and bundling multiple questions can lead to unexpected "cross-talk" interference.
What Problem Does This Paper Address?
Standard Large Language Models (LLMs) typically operate as "System 2" engines: they are slow, deliberative, and generate text token-by-token. Recently, a new paradigm of "System 1" models (like TypeSafe AI’s Jev or Together AI’s Tev1) has emerged. These models are designed for speed, returning structured data (like a "Choice" or a "Score") directly from their internal probability distributions.
While these models are being rapidly deployed for low-latency tasks like clinical triage and security routing, we don't actually know if we can trust them. Are their probability scores accurate (calibrated)? Do they rely on shallow shortcuts? Can we explain why they made a decision? This paper fills that gap by providing the first comprehensive audit of their reliability and explainability.
Methods
The author developed a deterministic audit framework that evaluates models across three dimensions: calibration, robustness to noise, and attribution faithfulness.
A key technical contribution is S1-SHAP. Standard explainability tools like KernelSHAP often struggle with "softmax saturation"—where a model is so certain (e.g., 99.9% probability) that removing a small piece of evidence barely changes the output probability, even if it significantly changes the internal logic. S1-SHAP applies a logit link function to KernelSHAP, mapping probabilities back to a linear space. This allows the tool to "see" through the saturation and accurately credit multiple pieces of evidence.
The audit also compares two request protocols:
- Isolated: One question per request.
- Bundled: Multiple questions in one request (a common industry practice to save latency).
Datasets
The study uses five complementary datasets to stress-test different aspects of model behavior:
Notably, the Planted Rule Benchmark is a novel programmatic suite where the author injects explicit "hidden triggers" (e.g., "If the note mentions 'velvet', classify as immediate"). This provides an absolute ground truth to see if XAI methods can actually recover the model's internal logic gates (AND/OR rules).
Key Insights and Ideas
- The "Confidence" Field is a Mathematical Identity: The "confidence" score returned by these model providers isn't an empirical measure of how often the model is right. Instead, it's a closed-form calculation of Shannon Entropy (or margin). It tells you how concentrated the probability distribution is, not how much you should trust the answer.
- Bundling Induces Interference: Asking two questions at once (e.g., "What is the triage level?" and "Is the patient in immediate danger?") causes "cross-question interference." In smaller models (0.8B), this caused binary answer flips in up to 22.4% of cases.
- Logit-Space is Essential for XAI: The paper proves that evaluating feature importance in raw probability space fails to recover redundant logic (OR gates). By moving to logit-space, S1-SHAP improved logic gate recovery from ~57% to ~89%.
- The 4B Parameter Threshold: There is a significant jump in reliability at the 4B parameter mark. Models at this scale were substantially more robust to adversarial shortcuts (like those in the HANS dataset) than sub-billion parameter models.
Key Results
- Extreme Latency Gains: System 1 models operate at 24–64 ms median latency, compared to 770–4,800 ms for generative System 2 baselines.
- Superior Calibration: Models $\ge$ 4B parameters achieved excellent calibration (Expected Calibration Error $ECE_w \approx 0.03$), significantly outperforming generative models using JSON mode ($ECE_w \approx 0.37$).
- Security Performance: For prompt-injection defense, 4B+ models reached 0.988–0.998 AUROC, detecting nearly all attacks with zero false alarms on difficult technical queries.
- Interactive XAI: Because these models are so fast, the framework can evaluate 512 different "what-if" scenarios (coalitions) in just 1.3–3.2 seconds, making rigorous SHAP explanations interactive for the first time.
Limitations and Future Work
The author notes several boundaries to this study:
- Synthetic vs. Real-World: The clinical triage vignettes were programmatically generated to ensure ground-truth accuracy. Real-world electronic health records (EHR) are messier and require further validation.
- Base Rates: The benchmarks used balanced classes (e.g., 50/50 split), but real-world deployments often deal with "needles in haystacks" (e.g., very rare security attacks), which might affect calibration.
- Quantization: The models were tested using specific quantizations (GGUF Q8_0/Q4_K_M). How different compression techniques affect these decision heads remains an open question.
Summary updated