Learned Libraries of Layout-Grounded Subroutines for Numerical Document QA

Systems that answer numerical questions about financial documents rarely fail at arithmetic anymore. Handing the calculation to a Python executor fixes that. They fail earlier, when they pick the wrong number off the page. The share of errors traced to picking the wrong cell or span is 55% for TAGOP on TAT-QA, 48% for the extractor in TAT-LLM, and 62% for the pointer ("index") errors of UniRPG. Yet the steps a human analyst takes to find those numbers are highly repetitive: find the row labelled "operating income", find the column headed 2019, check the table caption for "in thousands", then subtract. We propose to let a program-synthesis system discover these recurring lookup-and-compute routines from its own successful solutions. It would store them as named, reusable functions in a growing library, where each function operates on the page's layout (text boxes, rows, columns, captions) rather than on a flat OCR string.

The core idea fits in two sentences. Instead of a neural tagger guessing which numbers matter, a language model writes short programs that locate evidence through geometric page operations and then compute with it. Recurring program fragments are then compressed into higher-level subroutines, whose names and descriptions are checked by running them on other pages rather than trusted as the model wrote them. The bet is that learned, verifiable extraction subroutines cut the evidence-extraction error share and transfer better to unseen report templates than free-form code or fixed operator sets.

Core Claim

Library learning is the core mechanism, meaning the automatic growth of a vocabulary of reusable subroutines by compressing solved programs. Numerical reasoning over visually rich documents is the application. The gap between visual pages and symbolic programs shapes the design in one specific way: the base vocabulary is made of layout operations whose outputs are page regions that can be drawn back onto the image. Our claim is that the library should abstract over finding numbers as well as computing with them. Prior library learners have only abstracted over computation, or over already-symbolic inputs.

The claim is also that names for the learned abstractions must be grounded by execution. LILO showed two things: anonymous abstractions like fn_42 confuse the language model doing synthesis, and its LLM-written docstrings fix this. It also reported that wrong docstrings (for example, misdescribing an angle argument in LOGO graphics) propagate into later synthesis rounds. In documents this failure would be severe, because a routine named "get prior-year value" that actually reads the adjacent column silently corrupts every later answer.

Relation to Prior Work

Three systems supply the method. DreamCoder and its predecessor EC² established the loop of searching for programs, compressing the solutions into new primitives under a minimum-description-length objective, and searching again with the richer vocabulary. In text editing, this raised solve rates from 3.7% to 79.6%. LILO replaced most enumeration with language-model proposals and used the STITCH compressor plus automatic documentation, reaching 93.2% on REGEX against DreamCoder's 45.6%. We reuse LILO's loop almost wholesale. The difference is the domain. LILO's closest task, CLEVR, is solved over ground-truth symbolic scene descriptions, so it never confronts noisy perception or layout. Our base primitives consume OCR tokens with bounding boxes from real PDF pages.

Four systems define the numerical reasoning side and motivate the extraction focus. TAT-QA/TAGOP and OPERA classify each question into a fixed set of 10 or 11 operators. That fixed set cannot express, say, compound growth. UniRPG generates programs in a hand-built operator language and executes them deterministically. TAT-LLM fine-tunes LLaMA 2 into an extractor–reasoner–executor pipeline that writes free Python, and credits the external executor with a 16-point gain. All four keep extraction as a neural tagging or pointing step outside the program. None grows its operator vocabulary. We move extraction inside the program so that it becomes a target for compression.

The multimodal document baseline is MHST on TAT-DQA. It encodes page images and boxes with LayoutLMv2 and builds expression trees over the four basic arithmetic operators. It shows that layout signals help (41.5 EM versus 33.7 for text-only TagOp), but its grammar is static.

On the cross-modal side, Cross-Modal Discrete Representation Learning and UniCode close the vision–language gap by quantizing both modalities into shared discrete codes. These codes are flat cluster indices with no executable meaning. We take the opposite route: the discrete units are typed, executable layout operations, and alignment to the image is checked by rendering their outputs. ChartSketcher is the nearest precedent for grounding reasoning through drawn program outputs fed back to the model. Its drawing primitives are fixed, and its arithmetic stays in free text. We borrow the render-and-check idea only for verifying documentation, not as an open-ended reflection loop, which ChartSketcher reports can fail to converge.

Two results constrain the design. Zhao et al. show that the order in which tasks arrive can lock a library into premature, specialized abstractions, so our curriculum must be controlled and varied across seeds. ABL-Sym searches over a static SymPy library with only answer supervision, and still fails to recover about 57.7% of programs for complex problems. This warns that answer-only search is lossy, so we must measure how much of the training set the search actually solves.

Research Program

The base language has two layers. The first is a small set of layout primitives over a page's OCR tokens and boxes: find a text block matching a string, return the tokens sharing a row or column band with a box, return the header above a box, return the nearest caption, parse a number with its sign and parentheses convention, and read a scale word. The second is ordinary arithmetic and aggregation. Every value carries the box it came from, so any program's evidence can be drawn on the page.

In the wake phase, an instruction-tuned 7B language model sees the question, the serialized page layout and the current library with its documentation, and samples candidate programs. A program is kept if its execution matches the gold answer. We use answer-only supervision, as in UniRPG's weak-supervision setting, which lagged full supervision by only 1.4 F1 on TAT-QA. In the sleep phase, STITCH compresses the kept programs into new abstractions. We expect routines resembling "value at row label × year column, scaled by caption" and "percentage change between two such lookups" to emerge.

The ingredient we add is execution-grounded naming. An LLM proposes a name and docstring for each new abstraction. We then call the abstraction on held-out pages from other companies, render the returned boxes on the page images, and ask a separate vision-language model a narrow question: does the highlighted region match the docstring? We also ask a text-only model to predict the output from the docstring and the inputs alone. If an abstraction fails either check above a threshold, it is renamed from its execution traces or dropped. This is our response to LILO's cascading-docstring failure, and it is where the cross-modal check does real work.

Baselines

The naive reference point is the same 7B model writing free Python over the serialized OCR text, with no layout primitives and no library, trained on the same data. This is TAT-LLM's recipe re-run under our budget, and it tells us whether any of our structure matters.

The strongest existing method is TAT-LLM. Its reported 7B result on TAT-DQA is 69.45 EM, and its authors describe this as nearly 5 points above GPT-4. We also report MHST at 41.5 EM and 50.7 F1 as the strongest layout-encoder method. For fairness we use a 7B LLaMA-family backbone of the same size, the same training questions, the same OCR output, and a matched budget of sampled programs per question at training and test time. We tune the TAT-LLM reproduction with the same hyperparameter search budget as our system. Beating it means exceeding 69.45 EM on TAT-DQA with the 7B model, and lowering the share of errors attributable to wrong evidence well below TAT-LLM's 48%. Where annotated evidence is available, we compare the boxes a program actually read against the gold evidence.

The ablations isolate the key ingredients. A base-language-only variant keeps the layout primitives but never compresses, isolating the value of the library. A second variant learns a library over computation only, with extraction kept as a neural tagger. This isolates whether abstracting over extraction is what matters. A third variant uses LILO's unchecked AutoDoc naming, and a fourth uses anonymous names, together isolating execution-grounded documentation. A hand-written library of finance and layout helpers, written by one of us in a fixed time budget before seeing test data, and a retrieval-of-past-programs variant are covered below.

Why not just…?

Why not just prompt a frontier vision-language model with chain-of-thought over the page image? TAT-LLM-7B already outperforms GPT-4 on TAT-DQA by about 5 EM points, and ChartSketcher reports GPT-4o at 52.80 on EvoChart-QA against 63.28 for a grounded method. Both results point to perception errors and unaided arithmetic as the limit. We will still run a frontier model with image input and a code interpreter as an upper-reference point. If it matches our system on held-out templates, the contribution shrinks to cost and interpretability, and we will say so.

Why not just hand-write a library of financial and layout helpers? This is the most serious threat, because common financial formulas are well known and easy to code. It is plausible that a hand-written library matches ours on in-distribution questions. Our hypothesis is narrower: learned extraction routines capture template-specific conventions, such as "in thousands" appearing in a column header versus a caption, or negative numbers written in parentheses, that a fixed library misses on report layouts it was not designed for. The company-held-out split settles this. If the hand-written library matches the learned one there too, we will report library learning as unnecessary for this domain.

Why not just retrieve similar solved programs as in-context examples instead of compressing them? Retrieval gives reuse without abstraction and is cheap. We run it with the same pool of solved programs. The distinguishing prediction is on long, multi-lookup questions, where compressed routines shorten programs and should reduce pointer errors. If retrieval matches compression on those questions, compression adds nothing.

Why not just train a better evidence extractor? TAT-LLM already fine-tunes a dedicated extractor and still attributes 48% of its errors to it. The computation-only-library ablation keeps a tuned neural extractor and tests exactly this alternative against programmatic extraction.

Proposed Experiments

The main benchmark is TAT-DQA, with TAT-QA and FinQA for the text-and-table setting. We report EM and F1, the evidence-error share, library size, reuse rate and program length. Beyond the standard splits, we build a split that holds out companies, and hence reporting templates. This split is the test the proposal lives or dies on. The prediction is that the learned library's advantage over free Python and over the hand-written library is larger on held-out companies than in-distribution.

The second decisive check concerns naming. We count how many abstractions fail the render-and-predict checks, and measure downstream accuracy with and without renaming across three task orderings, to expose the path dependence Zhao et al. describe.

The third check is search coverage. We report the fraction of training questions for which any verified program was found, since ABL-Sym's 57.7% miss rate warns that answer-only search can starve the compressor.

A negative outcome would still be informative. That outcome would be a held-out-company gain within noise, together with an evidence-error share no better than TAT-LLM's. It would indicate that extraction errors on these benchmarks come from OCR and perception rather than from missing reusable structure.