Slot-Checked Judging of Numbers in Financial Reports

When a language model summarizes a 10-K annual report or answers a question about one, its most common factual error is usually not a made-up number or a wrong calculation. It is a real number put in the wrong place: last year's revenue reported as this year's, one segment's margin credited to the whole company, a figure in thousands read as millions. A manual study of model summaries of SEC filings found overall hallucination rates around 5–6%, with this "context mismatch" the most frequent type, and found that GPT-4 produced zero fabricated numbers and zero arithmetic errors (Characterizing Multimodal Long-form Summarization). That finding implies a problem for the automated checkers now in use. A check that asks "does this number appear in the source?" passes a context mismatch, because the number does appear. A check that re-executes the arithmetic also passes it, because the arithmetic is valid on the wrong inputs. An LLM judge (a language model prompted to grade another model's output) reads the claim fluently and tends to accept it.

We propose a judge that ties every number to an address and checks that address. For each numeric claim in a generated answer, the judge extracts a typed record: entity, line item, reporting period, unit and scale, and value. It then retrieves the source cell using the key fields only, never the value, and compares. When the key cannot be resolved directly, it checks the claim against redundancies already present in financial statements, such as quarters summing to the fiscal year, segments summing to the consolidated total, and assets equaling liabilities plus equity. The aim is a hallucination judge whose recall on context mismatches is close to its recall on fabrications, whose verdicts trace to a specific cell, and which is therefore much less sensitive than a free-text judge to irrelevant prompt cues such as a claimed author.

Core Claim

Current approaches to judging financial hallucinations check whether a number exists in the source or whether a computation is valid. Neither of these checks the attribution of a number, meaning which entity, line item, period, and unit it belongs to. Attribution is where the dominant error mass lies. Our claim has three parts. First, lookup by key fields with value comparison will catch context mismatches that value-only matching, program re-execution, and strong rubric-prompted LLM judges miss. Second, the remaining errors will come mostly from the step that translates text into typed records rather than from the verification itself, and those cases can be detected and routed to abstention. Third, because the verdict rests on a cell comparison and not on a persuasive critique, the judge will show much smaller verdict shifts under the provenance and recency cues that move ordinary judges.

The third part matters because The Silent Judge showed that GPT-4o shifts pairwise verdicts by up to +30% on cue-only changes such as dates, and up to +18% when one answer is labeled "EXPERT". It also showed that the judge never mentions the cue: the Cue Acknowledgment Rate was 0%. Financial evaluation pipelines already use unverified GPT-4 judges, for example to score stock-prediction explanations in SEP and in the dual-rule judge of FinMTM. Neither has been tested for this weakness.

Relation to Prior Work

Two neuro-symbolic judges supply the general method. Here "neuro-symbolic" means an LLM translates claims into formal statements and a deterministic checker evaluates them. FormalJudge compiles agent trajectories into Dafny specifications, which are checked by the Z3 solver, and reports a 16.6% average gain over LLM judges on tool-use safety benchmarks. SymDiag compiles reasoning steps into Prolog and separates reasoning errors from translation errors, reaching faithfulness-detection F1 of 70.7 against 62.8 for an LLM judge. We reuse their pattern of an LLM front end with a deterministic back end, and SymDiag's translation-error accounting. Both, however, verify whether constraints or inferences hold. Neither checks whether a value is bound to the right real-world slot. That binding is exactly what an attribution error breaks, so it is what we add.

Several papers motivate the work by documenting the gap. The 10-K summarization study above supplies the four-way error taxonomy (fabricated, rounding, arithmetic, context mismatch) but relied on human annotation. PHANTOM shows that hallucination detection degrades with context length and with evidence placed mid-document. Halu-J, a 7B critique judge trained with Direct Preference Optimization, explicitly leaves numerical reasoning and long contexts open.

Several datasets constrain the design. FinQA, ConvFinQA, and DocFinQA score only final execution accuracy. DocFinQA treats a synthesized Python program as correct whenever its output matches. FinQA attributes roughly 15% of errors to retrieval failure. Together these suggest that retrieval by key will be a bottleneck that we must measure rather than assume away.

The visual channel adds a further constraint. MLLM-as-a-Judge found that chain-of-thought critique can make multimodal judges worse, because they invent visual details and then reason from them. This is why our design limits the LLM to producing typed records and leaves the verdict to a comparison.

Research Program

The system has three stages. In the first, an extractor model turns each sentence of a candidate answer into zero or more typed claims. It also turns the source document's tables into a keyed store with the same schema, keeping the text near each table so that footnotes and segment labels survive. The extractor may emit "unresolved" for any field it cannot determine.

The second stage is key lookup. A claim with a fully resolved key is compared to its cell. Equality is checked under explicit rounding tolerance and unit conversion, which separates rounding errors from real mismatches. If the comparison fails and the claimed value appears elsewhere in the store, we report a context mismatch and identify which field differs: period, entity or segment, line item, or scale. If the value appears nowhere, we report a fabrication.

The third stage handles derived claims, such as growth rates or margins, and claims whose keys are partly unresolved. These are encoded as linear arithmetic over cells together with the statement identities that apply, and passed to Z3. If no assignment of the claim's unresolved fields to existing cells satisfies both the claim and the identities, the claim is rejected. If several assignments do, the judge abstains. Abstention is reported as its own outcome and never counted as "supported".

Charts form a separate track. We use FinMTM chart items, where the "source cell" must first be read from an image. In this track the judge's verdict is logged together with whether the reading of the chart was right. This lets us separate perception errors from attribution errors on the visual track.

Baselines

The naive reference point is value-only string matching. A numeric claim is marked supported if its value, after normalizing rounding and units, occurs anywhere in the source. We expect this check to reach near-perfect recall on fabrications and near-zero recall on context mismatches. Its per-type recall profile is the clearest single picture of the gap.

The second baseline is program re-execution in the FinQA and DocFinQA style. The candidate answer's computation is rewritten as a Python program over retrieved values and executed. This is the deterministic check the field already uses, and it is strong on arithmetic errors. It runs on the same retrieved context as our judge.

The strongest prompted competitor is a frontier rubric judge with chain-of-thought. The rubric explicitly tells it to verify period, entity, segment, and unit for every number. It receives the same source context and the same answer, and its prompt is tuned on the same development split with an equal number of iterations.

The strongest trained competitor is a PHANTOM-style detector. PHANTOM reports that fine-tuning Qwen2.5-7B-Instruct lifts 10-K hallucination-detection F1 from 0.708 to 0.933. We retrain the same model on our training split with the same compute budget we give our extractor, and we also run a Halu-J-style critique judge. Halu-J reports 91% accuracy on ME-FEVER against GPT-4o's 83%, but no reported result was found on Lacuna for Halu-J on financial numeric claims.

The final baseline is a FormalJudge-style formal judge. It encodes claims as arithmetic constraints but has no typed keys, which isolates the contribution of slot typing from that of having a solver at all.

Beating these baselines means four things. Context-mismatch recall must exceed every baseline by a clear margin at matched false-positive rate. Macro-F1 across the four error types must exceed the fine-tuned detector on held-out documents. The verdict-shift rate under cue injection must be well below the rubric judge's. And these gains must hold on naturally occurring errors as well as on injected ones.

Ablations remove one ingredient at a time. We remove the key fields, which reduces the judge to value lookup inside the same pipeline. We remove the statement identities. We remove abstention, forcing a verdict on every claim. And we replace the extractor with the rubric judge's own model, to check that the gains do not come simply from using a stronger model.

Why not just…?

Why not just fine-tune a detector, as PHANTOM did? This is the real threat. A 7B model reaching 0.933 F1 on 10-K detection is strong, and on the error types it was trained on it may match us. Two experiments settle the question. First, we hold out entire mismatch subtypes, for example training on period swaps and testing on segment swaps, and we hold out entire filers. A learned detector should degrade under these shifts, while key lookup does not depend on which subtype it saw. Second, we measure the cue-injection verdict shift for the detector. If the fine-tuned detector matches us on both, the slot-typing claim fails, and we would report that.

Why not just re-execute the computation in code? Re-execution verifies that the operations are correct given the inputs. A context mismatch supplies a wrong input that is nonetheless genuine, so the program runs cleanly and returns the "right" answer to the wrong question. DocFinQA's practice of accepting a program whenever its output matches is exactly this blind spot. Reporting recall separately for each error type will show the effect directly.

Why not just instruct a strong LLM judge to check periods and entities? This might close part of the gap, and the rubric baseline tests it head on. Two cautions apply. Long-context detection degrades when evidence sits mid-document (PHANTOM). And free-text judges rationalize cue-driven verdicts without acknowledging the cues (The Silent Judge). We therefore report the prompted judge's results stratified by evidence position, and its verdict-shift rate under cue injection, alongside its accuracy.

Why not just apply FormalJudge unchanged? A solver with no slot types can only check whether the arithmetic is consistent. A misattributed value is arithmetically consistent, so the solver accepts it. The no-keys ablation is precisely this system, and its context-mismatch recall is the measured answer.

Proposed Experiments

The evaluation set has three sources. The first is controlled injection. We take the gold supporting facts and answers in FinQA, ConvFinQA, and DocFinQA, generate fluent answers and short summaries, and inject single errors of each type. Context mismatches are split by which field is swapped. Every injected value is drawn from a real cell of the same filing, so a value-only check cannot see the error.

The second source is natural errors. We generate full MD&A summaries with current open and closed models, following the setup of the 10-K summarization study, and expert annotators label a subset with the same four-way taxonomy. The third source is the PHANTOM 10-K split, used for comparability with its reported F1 of 0.933.

We report recall for each error type at a fixed 5% false-positive rate, macro-F1, abstention rate together with accuracy on the claims the judge does not abstain on, and translation-error rate measured against gold typed records on a hand-labeled sample, following SymDiag's accounting. We also report results stratified by evidence position and context length.

For bias, we use the Silent Judge protocol on pairwise financial-answer comparisons. We inject "EXPERT/CFA analyst" versus "UNKNOWN" provenance labels and date labels, and report the verdict-shift rate for every judge.

The chart track on FinMTM reports judge accuracy conditioned on whether the extractor read the chart correctly. This tests whether attribution errors remain distinguishable from perception errors once values must be read from images.

The decisive check is the comparison of context-mismatch recall on natural, non-injected summaries between our judge and the retrained PHANTOM-style detector. If typed lookup does not clearly beat the detector there while also showing a lower verdict-shift rate, the proposal's central claim is wrong.