Honest Small Decision Models for Finance: Scale, Specialization Collapse, and Mixed-Corpus Remediation

Ajitabh KumararXiv 2026 · arXiv

TL;DR

This paper introduces "System-1" decision models—small language models (0.8B to 32B parameters) fine-tuned to provide calibrated probabilities for financial decisions in a single forward pass. The authors discover that narrow fine-tuning causes a "specialization collapse" in general reasoning, but demonstrate that a mixed-corpus training approach creates "honest" models that follow rules perfectly and report uncertainty truthfully when market outcomes are unpredictable.

What Problem Does This Paper Address?

Financial institutions don't just need chatty AI; they need fast, auditable, and calibrated decisions (e.g., "Should this trade be approved?"). While large generative models are powerful, they are often too slow, expensive, and prone to "hallucinating" confidence.

The authors tackle three main challenges:

  1. Specialization Collapse: When you fine-tune a small model on a narrow task (like reading a specific company policy), it often loses its ability to handle general logic—a phenomenon the authors quantify across different model scales.
  2. Reality vs. Labels: Most benchmarks use "clean" labels (right/wrong). Financial markets are noisy and often unpredictable. How does a model behave when the "ground truth" is essentially a coin flip?
  3. Operationalizing Probabilities: How do you turn a model's output probability (e.g., 55% chance of success) into a concrete business action like "Approve," "Reject," or "Escalate to Human Review"?

Methods

The researchers employed a "System-1" architecture, where the model is constrained to answer typed questions (choice, score, or yes/no) by emitting probabilities over specific tokens in a single forward pass. This is significantly faster and cheaper than standard text generation.

Key methodological components include:

  • Model Families: They tested nine rungs of models across the Qwen and K2-Horizon families, ranging from 0.8B to 32B parameters.
  • Frozen LoRA Recipe: To ensure a controlled study, they used a fixed Low-Rank Adaptation (LoRA) configuration ($r=16, \alpha=32$) across all experiments.
  • Temperature Scaling: A post-processing step to ensure the model's reported probabilities actually match real-world frequencies (calibration).
  • Mixed-Corpus Remediation: Instead of training on one task at a time, they concatenated general decision data with domain-specific financial data into a single training run to prevent capability erosion.

Datasets

The study utilized three distinct corpora:

  1. Public Benchmark ("kev"): A general-purpose decision suite used to measure "generalist" competence.
  2. Policy Corpus (Programmatic-Gold): A new dataset of 2,651 records where labels are derived from strict logical rules (e.g., "If RSI > 70, then Reject"). This tests the model's ability to "read the rulebook."
  3. Outcome Corpus (Reality-Labeled): A new dataset of 19,640 realized forward returns from 40 liquid US equities (2016–present). This tests if models can predict actual market moves based on technical indicators like RSI and ADX.

Key Insights and Ideas

  • The Specialization-Gap Trend: Narrow fine-tuning is destructive. Training only on market outcomes can tank a model's general decision accuracy by up to 45 percentage points. Interestingly, larger models are more resilient to this "collapse," but even at 32B, a significant gap remains.
  • Honest Uncertainty: In the market outcome task, no model—regardless of size—beat the simple baseline of "betting on the most frequent historical result." However, the models were "honest": they produced flat, well-calibrated probability distributions. They essentially learned to say, "I don't know, and here is exactly how much I don't know."
  • Band Dispatch: The authors propose a deployment strategy based on these probabilities. For rule-based tasks, the model is highly confident (bimodal distribution). For market predictions, it stays in the "middle." By setting thresholds, firms can automate high-confidence rule-following while routing "honest uncertainty" to human experts.
Figure 1: Calibrated 𝑃(yes) by realized label for the Q-2B mixed adapter, post domain temperature (policy 𝑇=1.54, outcome 𝑇=1.28), common axis. Left: policy development noul questions (n=129) separate into sharp extremes (21% at 𝑃≥0.90, 79% at 𝑃≤0.25). Right: outcome development 2025 noul questions (n=695) overlap near the middle; the shaded [0.40, 0.60) region is the proposed review band, covering 88% of outcome mass.

Key Results

  • Rule Following: Trained adapters achieved near-perfect accuracy (0.99–1.00) on the policy corpus, proving that SLMs are excellent at learning and applying explicit logical structures.
  • Market Prediction: Models landed at the "marginal ceiling" (accuracy ~0.38), matching the prior baseline. They found no "alpha" or predictive edge using standard technical features, but maintained excellent calibration (ECE as low as 0.026).
  • Remediation Success: The mixed-corpus adapter successfully recovered generalist performance. As shown in Figure 2, the "mixed" approach (purple) maintains high accuracy on both the public benchmark and domain-specific tasks, whereas "policy-only" (pink) or "generalist" (blue) models fail on at least one axis.
Figure 2: The specialization trade-off and its remediation at Qwen-2B (three seeds; mean with min–max whiskers, Table 6). Left: policy development accuracy against public-benchmark accuracy; right: outcome development Brier (lower is better) against public-benchmark accuracy. Narrow training buys one finance axis at a large cost in general competence (and leaves the untrained finance axis at chance); mixed training recovers the generalist position on the public benchmark while holding both finance axes near their specialists.

Limitations and Future Work

The authors acknowledge several constraints:

  • Feature Set: The lack of predictive "edge" in market outcomes might be due to the limited feature set (only seven technical indicators). Future work could involve richer states like news sentiment or fundamental data.
  • Universe Size: The study focused on 40 large-cap US stocks; results might differ in less efficient or more volatile markets.
  • Single Runs: Most scale rungs were evaluated via single training runs rather than multi-seed averages (though a 3-seed replication at the 2B scale confirmed the main trends).
  • Sealed Test: The 2026 test set remains unevaluated to prevent data leakage, leaving the model's performance in a truly "future" regime as an open question.

Summary updated

Abstract

Financial workflows need fast, calibrated, auditable decisions, not just generated text. We train System-1 decision models, small language models fine-tuned to answer typed questions (choice/score/noul) with a probability in a single forward pass, across two model families (K2-Horizon and Qwen3.5/3.8) and nine rungs from 0.8B to 32B parameters, using one frozen LoRA recipe on a public decision benchmark and two new finance corpora: a programmatic-gold policy corpus (rule-reading with exact labels) and a reality-labeled outcome corpus (19,640 realized forward-return questions in walk-forward splits, plus a sealed 2026 test of 1,360 questions). Narrow fine-tuning costs general decision competence 9-45 percentage points, a data-mixture effect that shrinks with model scale and is associated with narrow-gradient volume. On reality-labeled questions, every trained rung from 0.8B to 32B lands at the horizon-conditioned prior baseline (accuracy 0.37-0.38, no detectable conditional edge under our controls); the models respond with flat, well-calibrated probabilities that track per-horizon base rates: naturally honest uncertainty, with a mid-scale calibration sweet spot (ECE 0.026 at 2B, degrading to 0.043-0.044 at 9-32B under prior drift). A mixed-corpus remediation carries all competencies in one adapter at modest cost: it matches or slightly exceeds the generalist specialist on the public benchmark while holding the policy ceiling and near-specialist outcome calibration, yielding a deployable artifact with per-domain temperature regimes and a band-dispatch escalation analysis: sharp, dispatchable bands where rules determine answers; flat, review-routing bands where reality does not.
BibTeX
@misc{kumar2026honest,
  author = {Ajitabh Kumar},
  title = {{Honest Small Decision Models for Finance: Scale, Specialization Collapse, and Mixed-Corpus Remediation}},
  year = {2026},
  eprint = {2611.03755},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03755},
}
Figure