When a language model answers a question about a company's annual report, the numbers it writes do different jobs. Some are copied from the document ("revenue in 2019 was 1,234"). Some are computed from copied values ("a 12.2% increase"). Some are labels that only look like quantities, such as the year "2019" or "Note 7." Recent work trains models to respect the number line: predicting 49 when the answer is 50 costs less than predicting 900. That helps computed values. For copied values it is the wrong lesson. Writing 1,233 when the filing says 1,234 is not a "near miss." It is a fabricated figure, and it is close enough to the truth that a human or an automatic checker is unlikely to catch it.
We propose to fine-tune small domain models with a loss that depends on the role of each number. Computed values get a number-line distance penalty. Copied values get a penalty defined over the numbers that actually appear in the document, so the model is pushed to pick the correct cell rather than a nearby magnitude. Labels get ordinary cross-entropy. Role labels come for free from the gold calculation programs in FinQA. The central empirical bet, and the thing that makes this more than a transfer of existing losses to a new domain, is a falsifiable prediction: applying a number-line loss to every numeral will reduce gross scale errors but increase plausible, near-miss fabrications of copied figures, and those fabrications will be harder for hallucination detectors to flag.
Core Claim
Metric-aware number losses are built around one assumption: closeness on the real line is the right notion of "almost correct." That assumption holds for regression targets. It fails for numbers whose correctness means agreement with a source document. In grounded domain generation, a single loss over all numeric tokens therefore trades one kind of hallucination, large scale errors, for another, subtle unattested figures. Training each number according to its role (computed, copied, or label) should keep the scale-error gains and remove the near-miss cost. The test is the rate of copied numerals that are absent from the context or point to the wrong cell, measured alongside answer accuracy.
What the prior work establishes
Four papers supply the number-line machinery we reuse. All four were evaluated on isolated arithmetic, clocks, or visual grounding, never on figures that must be faithful to a source document. Regress, Don't Guess adds a Wasserstein-style loss over digit-token probabilities (NTL-WAS). Teaching Metric Distance to Discrete Autoregressive Language Models replaces one-hot targets with a discrete Gaussian around the true value and weights digits by place value (DIST²Loss). Advancing Sequential Numerical Prediction combines token-level earth-mover penalties with a sequence-level relaxation (NTIL). Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment matches predicted digit distributions to targets with a kernel loss (SMMD).
SMMD's own analysis also motivates this proposal. Its main finding is that the loss turns large scale errors into near misses. In a grounded setting, near misses are a form of hallucination. The same authors also note that their loss cannot tell metric numbers from categorical ones like zip codes, which is exactly the label role we separate out. SMMD therefore both motivates the work and constrains the design: any number-line loss applied without role information should, by its own reported behavior, shift errors toward near misses.
Two lines of work define the application regime and the remaining failure. PHANTOM shows that fine-tuning Qwen2.5-7B-Instruct with ordinary cross-entropy on SEC-filing data raises hallucination-detection F1 on 10-K filings from 0.708 to 0.933. It also shows that detection degrades as context grows and when evidence sits mid-document. PHANTOM uses no number-aware objective. FinQA and APOLLO hand arithmetic off to a small calculation language, so calculation errors disappear. The remaining errors come from binding the wrong literal into the program: FinQA attributes roughly 15% of errors to retrieval failures, and DocFinQA shows that the problem gets worse on full-length filings. APOLLO's number-aware negative sampling acts on the retriever. Our copied-value term moves the same idea into the generator's training loss.
Benford's Curse traces numerical hallucination to a small-digit bias that pretraining data instills in late-layer feed-forward neurons. It removes that bias by pruning neurons after training, which breaks some correct predictions, and the authors call for training-time interventions. We use its digit-skew diagnostic as a secondary readout. SPR-RAFT adds a regression head to a frozen model and outputs single scalars. It cannot write text, so it is related work rather than a competitor here.
Research Program
Role labels. We derive labels automatically. Each FinQA gold program lists its arguments, which are copied from the table or text, and its intermediate and final results, which are computed. We convert each program into a short natural-language rationale ("2019 revenue was X; 2018 was Y; the change is Z%") and mark every numeral span by role. Arguments are tagged copied when they match a context span exactly. Execution outputs are tagged computed. Years, note numbers, and other numerals that match header or label patterns are tagged label. We will hand-audit a few hundred examples to estimate label noise. For ConvFinQA the same procedure applies turn by turn.
Training loss. We fine-tune with LoRA (low-rank adapters on a frozen backbone) using
$$\mathcal{L} = \mathcal{L}_{CE} + \alpha \sum_{t \in C} \mathcal{L}_{dist}(t) + \beta \sum_{s \in A} \mathcal{L}_{ctx}(s),$$
where $\mathcal{L}_{CE}$ is token cross-entropy, $C$ is the set of digit tokens inside computed spans, and $\mathcal{L}_{dist}$ is DIST²Loss with place-value weighting. $A$ is the set of copied spans, and $\mathcal{L}_{ctx}(s)$ is a contrastive loss. It scores the gold numeral string for span $s$ against every other numeral string present in the document, using the model's sequence log-probability, and pushes up the gold string relative to the others. Same-row and same-column table cells serve as hard negatives, in the spirit of APOLLO's number-aware sampling. Label tokens receive only $\mathcal{L}_{CE}$.
Two output modes. In direct mode, the model writes the rationale and final answer itself. In program mode, it writes a FinQA-style program that an interpreter executes. In program mode there are no computed numerals to generate, so only the copied term matters. This lets us separate where each term helps.
Model and data. The backbone is Qwen2.5-7B-Instruct, matching PHANTOM. We also run a 1.5B model to check that the effect holds at smaller scale. We train on FinQA and evaluate on FinQA, on ConvFinQA, and on DocFinQA with retrieved chunks. For long-context placement we use DocFinQA with evidence relocated to the beginning, middle, or end, following PHANTOM's protocol.
Baselines
The naive reference is LoRA fine-tuning with plain cross-entropy on the same rationales, data, and adapter rank. This is the PHANTOM-style recipe, and every gain is measured against it.
The strongest representation-side competitor is global DIST²Loss: the same distance term applied to every numeric token regardless of role. A global NTL-WAS variant will also be run, because NTL reported DeepMind-Math accuracy rising from 0.64 to 0.75 over cross-entropy. Neither method has a reported result on financial QA; no reported result was found on Lacuna for either on FinQA. Global DIST²Loss is the direct test of our thesis. Beating it means matching its reduction in scale errors on computed answers while showing a significantly lower rate of unattested or wrong-cell copied numerals.
The strongest system-level competitor is APOLLO, which reports 72.47% execution accuracy on FinQA and 78.76% on ConvFinQA. Zero-shot GPT-4 reaches 68.79% on FinQA as reported in APOLLO, FinQANet reaches 61.24%, and human experts reach 91.16%. We will re-run APOLLO-style program training with the same backbone. Our program-mode model shares that backbone, training data, and retrieval. Beating APOLLO means higher execution accuracy, with the gain traceable to fewer wrong-argument errors.
A constrained-decoding baseline restricts every generated numeral in a copied slot to strings present in the document. This removes unattested numerals by construction. It therefore isolates the remaining question: whether our loss picks the right cell more often than constrained decoding does.
Two ablations isolate the key ingredient. One routes the distance term to computed spans with random role labels at matched proportions. The other uses the copied term alone. All methods get the same tuning budget: a fixed sweep over the loss weights and learning rate, selected on the FinQA development set.
Why not just…?
Why not just let a calculator do all the math? Program-aided generation, as in PAL and APOLLO, does eliminate calculation errors, and in program mode it makes our computed-value term irrelevant. But a calculator executes whatever literal it is handed. FinQA and DocFinQA both locate a share of failures in retrieving and binding the wrong figure, and FinanceReasoning reports hallucinated intermediate values in multi-step chains. Our program-mode experiment tests exactly this: if the copied-value term does not reduce wrong-argument errors relative to APOLLO-style training, the copied half of the proposal fails.
Why not just apply an existing number-line loss to every number? This is the most natural reading of the gap, and it is our main comparison. SMMD's finding that such losses convert gross errors into near misses predicts a specific failure on copied figures. If global DIST²Loss shows no rise in near-miss copied numerals, role-awareness is unnecessary, and we will report that.
Why not just force the model to copy numbers from the document? Constrained decoding is a real threat on the narrow metric of "numeral absent from context," and it will likely tie or win there. It cannot produce computed values, and it does nothing about choosing a neighboring cell. That second gap is why we report wrong-cell rate separately and run constrained decoding as a baseline. If constrained decoding plus plain cross-entropy matches our wrong-cell rate, the contrastive term adds nothing, and the contribution narrows to role-routing the distance loss.
Why not just rely on digit-level tokenization? Qwen already splits digits into single tokens. Benford's Curse shows that digit-tokenized models still learn small-digit bias and hallucinate because of it. Cross-entropy on single digits remains blind to distance.
Proposed Experiments and Decisive Checks
The primary metrics are execution or answer accuracy on FinQA, ConvFinQA, and DocFinQA; the unattested copied-numeral rate; the wrong-cell rate, meaning attested but incorrect; and the scale-error rate on computed answers, meaning off by more than 50% or by a factor of ten, following SMMD's error categories.
The first decisive check is the predicted crossover. Relative to cross-entropy, global DIST²Loss should lower the scale-error rate and raise the near-miss rate on copied numerals. The role-aware loss should lower both. If global DIST²Loss shows no near-miss increase, the motivating claim is false.
The second check is detectability. We fine-tune a Qwen2.5-7B detector on PHANTOM, following that paper's recipe, and measure how often it flags each model's numeric errors. The prediction is that near-miss copied errors are flagged less often than scale errors. If so, global metric losses make hallucinations quieter as well as different. This is the outcome that would matter most in practice.
The third check is position robustness. We break down wrong-cell rate by evidence placement in DocFinQA, to see whether the contrastive term reduces the mid-document degradation that PHANTOM reports.
The fourth, secondary, check uses the Benford's Curse diagnostic. We compare the distribution of first erroneous digits in copied numerals across all models, to test whether training-time role-aware losses flatten the small-digit skew without the collateral damage of neuron pruning.
Finally, we check retention of general capability by re-scoring GSM8K and a held-out reading-comprehension set before and after fine-tuning.
References
- Regress, Don't Guess: A Regression-like Loss on Number Tokens for Language Models
- Teaching Metric Distance to Discrete Autoregressive Language Models
- Advancing Sequential Numerical Prediction in Autoregressive Models
- Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment
- PHANTOM: A Benchmark for Hallucination Detection in Financial Long-Context QA
- FinQA: A Dataset of Numerical Reasoning over Financial Data
- APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning
- DocFinQA: A Long-Context Financial Reasoning Dataset
- Benford’s Curse: Tracing Digit Bias to Numerical Hallucination in LLMs
- SPR-RAFT: Parameter-Efficient Regression-Aware Fine-Tuning for Biomedical LLM Regression
- PAL: Program-aided Language Models
- FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging