TL;DR
This paper introduces "System-1" decision models—small language models (0.8B to 32B parameters) fine-tuned to provide calibrated probabilities for financial decisions in a single forward pass. The authors discover that narrow fine-tuning causes a "specialization collapse" in general reasoning, but demonstrate that a mixed-corpus training approach creates "honest" models that follow rules perfectly and report uncertainty truthfully when market outcomes are unpredictable.
What Problem Does This Paper Address?
Financial institutions don't just need chatty AI; they need fast, auditable, and calibrated decisions (e.g., "Should this trade be approved?"). While large generative models are powerful, they are often too slow, expensive, and prone to "hallucinating" confidence.
The authors tackle three main challenges:
- Specialization Collapse: When you fine-tune a small model on a narrow task (like reading a specific company policy), it often loses its ability to handle general logic—a phenomenon the authors quantify across different model scales.
- Reality vs. Labels: Most benchmarks use "clean" labels (right/wrong). Financial markets are noisy and often unpredictable. How does a model behave when the "ground truth" is essentially a coin flip?
- Operationalizing Probabilities: How do you turn a model's output probability (e.g., 55% chance of success) into a concrete business action like "Approve," "Reject," or "Escalate to Human Review"?
Methods
The researchers employed a "System-1" architecture, where the model is constrained to answer typed questions (choice, score, or yes/no) by emitting probabilities over specific tokens in a single forward pass. This is significantly faster and cheaper than standard text generation.
Key methodological components include:
- Model Families: They tested nine rungs of models across the Qwen and K2-Horizon families, ranging from 0.8B to 32B parameters.
- Frozen LoRA Recipe: To ensure a controlled study, they used a fixed Low-Rank Adaptation (LoRA) configuration ($r=16, \alpha=32$) across all experiments.
- Temperature Scaling: A post-processing step to ensure the model's reported probabilities actually match real-world frequencies (calibration).
- Mixed-Corpus Remediation: Instead of training on one task at a time, they concatenated general decision data with domain-specific financial data into a single training run to prevent capability erosion.
Datasets
The study utilized three distinct corpora:
- Public Benchmark ("kev"): A general-purpose decision suite used to measure "generalist" competence.
- Policy Corpus (Programmatic-Gold): A new dataset of 2,651 records where labels are derived from strict logical rules (e.g., "If RSI > 70, then Reject"). This tests the model's ability to "read the rulebook."
- Outcome Corpus (Reality-Labeled): A new dataset of 19,640 realized forward returns from 40 liquid US equities (2016–present). This tests if models can predict actual market moves based on technical indicators like RSI and ADX.
Key Insights and Ideas
- The Specialization-Gap Trend: Narrow fine-tuning is destructive. Training only on market outcomes can tank a model's general decision accuracy by up to 45 percentage points. Interestingly, larger models are more resilient to this "collapse," but even at 32B, a significant gap remains.
- Honest Uncertainty: In the market outcome task, no model—regardless of size—beat the simple baseline of "betting on the most frequent historical result." However, the models were "honest": they produced flat, well-calibrated probability distributions. They essentially learned to say, "I don't know, and here is exactly how much I don't know."
- Band Dispatch: The authors propose a deployment strategy based on these probabilities. For rule-based tasks, the model is highly confident (bimodal distribution). For market predictions, it stays in the "middle." By setting thresholds, firms can automate high-confidence rule-following while routing "honest uncertainty" to human experts.
Key Results
- Rule Following: Trained adapters achieved near-perfect accuracy (0.99–1.00) on the policy corpus, proving that SLMs are excellent at learning and applying explicit logical structures.
- Market Prediction: Models landed at the "marginal ceiling" (accuracy ~0.38), matching the prior baseline. They found no "alpha" or predictive edge using standard technical features, but maintained excellent calibration (ECE as low as 0.026).
- Remediation Success: The mixed-corpus adapter successfully recovered generalist performance. As shown in Figure 2, the "mixed" approach (purple) maintains high accuracy on both the public benchmark and domain-specific tasks, whereas "policy-only" (pink) or "generalist" (blue) models fail on at least one axis.
Limitations and Future Work
The authors acknowledge several constraints:
- Feature Set: The lack of predictive "edge" in market outcomes might be due to the limited feature set (only seven technical indicators). Future work could involve richer states like news sentiment or fundamental data.
- Universe Size: The study focused on 40 large-cap US stocks; results might differ in less efficient or more volatile markets.
- Single Runs: Most scale rungs were evaluated via single training runs rather than multi-seed averages (though a 3-seed replication at the 2B scale confirmed the main trends).
- Sealed Test: The 2026 test set remains unevaluated to prevent data leakage, leaving the model's performance in a truly "future" regime as an open question.
Summary updated