Many practical tasks, such as computing a patient's kidney function score from a clinical note or a leverage ratio from a financial filing, come down to two steps. First the model finds the right numbers in a messy document and attaches each one to the right formula input. Then it does the arithmetic. Language models are unreliable at the first step. They attach numbers to formula inputs by matching surface labels ("creatinine", "income"). Rename the label, add a second value with the same label (last week's creatinine, a reference range), or change the unit, and the attachment breaks. The failures are already visible. Even GPT-4 solves only about half of MedCalc-Bench clinical calculations with one worked example, and a large share of its errors come from extracting the wrong value. Swapping column headers for synonyms cuts tabular models' F1 by 43% (Adversarial Attacks on Tables with Entity Swap).
We propose to make this attachment step, which we call variable binding, an explicit and separately rewarded output. The model first writes a small binding table: each formula input, the value it assigned, the unit, and the exact text span it took the value from. A deterministic calculator then does the arithmetic. The central question is whether binding is a domain-general skill. Could it be learned by reinforcement learning on cheap, procedurally generated, non-medical problems full of look-alike distractors, and then carry over to clinical and financial documents without clinical labels? Or does it have to be trained on in-domain data? Clinical labels are expensive, and non-expert raters are known to reward fluent but wrong medical output (Yang et al.; Balloccu et al.). If the skill transfers, reliable clinical calculators could be trained without clinical annotation.
Core Claim
We claim three things. First, a model trained with a reward on binding correctness, and not only on final-answer correctness, will lose far less accuracy when labels are rewritten, same-label distractors are injected, or units are shifted. Second, this robustness comes mostly from training on domain-free synthetic data. Third, in-domain training on synthetic clinical notes adds only a modest further gain. All three are falsifiable. If a model trained only on final answers is just as robust, the binding reward is unnecessary. If domain-free training gives nothing on clinical notes, binding is not a transferable skill, and the second claim fails.
The four research directions play distinct roles. Contextual noise sensitivity defines the regime. Semantic metadata dependency is the specific failure being targeted: binding by label rather than by context. Cross-domain reasoning transfer is the training strategy under test. The domain-specific metric, MedCalc-Bench's ±5% relative tolerance, is turned from a passive score into part of the reward.
Relation to Prior Work
Three lines of work motivate the gap. MedCalc-Bench defines the three-step task (retrieve the formula, extract the variables, compute) and splits failures between extraction and arithmetic. It proposes no training method. TabLLM shows that permuting column names makes tabular language models' accuracy plummet. Entity Swap shows a drop from 88.86% to 26.5% F1 under cell substitution. Neither offers a defense or looks at numerical reasoning. Thinking in a Crowd shows that step-by-step "thinking" can make things worse under misleading context: Qwen3-8B falls from 96.22% to 77.03%, below its 74.87% without thinking, because it builds justifications for the false premise. That paper diagnoses the problem without proposing a fix. UDA shows that numerical questions over long raw financial documents collapse (GPT-4 scores 0.4 on FinHybrid without retrieval, versus 45.9 with it).
The closest method is AbstRaL. It trains, with Group Relative Policy Optimization (GRPO, a reinforcement learning method that compares several sampled answers per prompt), a model to map word problems into symbolic conditions before solving, and reports resistance to distractor sentences on GSM-Plus. We reuse its idea of rewarding the intermediate abstraction and not only the answer. We differ in three ways. Our intermediate object is a grounded binding with evidence spans and units, not a symbolic rewrite of the problem. Our distractors are adversarial for binding specifically: same-label values, synonyms, and unit changes, rather than irrelevant sentences. And we test transfer to clinical and financial documents, which AbstRaL never touches.
Two lines of work constrain the design. CPL and Reasoning Vectors show that math-trained reasoning transfers to code and science. CPL reports +12.2% on HumanEval and +8.6% on GPQA. Reasoning Vectors reports +12.3% on BBH. This supports the plausibility of transfer. Their targets, however, supply clean, well-defined variables, and the file notes that transfer tends to fail when variables must be dug out of distorted context. That failure is exactly our regime.
TabReason and TableGPT-R1 apply reinforcement learning to tables with format, validity, and answer rewards. TabReason reports explanations that contradict the predictions. This is the "right for the wrong reason" failure that an answer-only reward permits and that a binding reward is meant to close.
Ext2Gen trains a model to extract relevant sentences before answering and beats separate compressors. Its unit of selection is a relevant sentence. Our hardest distractors are relevant sentences that contain the wrong value, so sentence-level filtering is a natural but likely insufficient competitor.
We deliberately do not pursue the more obvious intersection: using transferred reasoning to rescue tabular models whose headers are fully anonymized. When the headers carry no meaning, the language model's advantage comes from world knowledge that no amount of reasoning can recover from the numbers alone. Gradient-boosted trees already handle that case well, as TabLLM and TDTransformer both concede when metadata is stripped. Partial label corruption (synonyms, abbreviations, renamed fields) keeps the meaning recoverable from context, and there reasoning can plausibly help.
Research Program
Binding output format. The model sees a document and a target calculation. It must emit a binding table followed by a call to a calculator. For each of the $K$ formula inputs, the table gives a value, a unit, and a verbatim evidence span. The calculator applies the formula and returns the result. For equation-based tasks, the policy never does arithmetic itself.
Reward. The reward for a rollout is
$$ R = \lambda \cdot \frac{1}{K}\sum_{k=1}^{K} b_k \;+\; (1-\lambda)\cdot \mathbf{1}\big[\,|\hat{y}-y| \le 0.05\,|y|\,\big], $$
where $b_k \in \{0,1\}$ is 1 when input $k$ has the correct value (within tolerance after unit conversion) and its evidence span actually contains that value; $\hat{y}$ and $y$ are the predicted and gold results; and $\lambda \in [0,1]$ weights binding against the final answer. The second term is MedCalc-Bench's own ±5% rule used directly as a reward. The span check prevents the model from getting binding credit for values it did not read from the document. We train with GRPO, following AbstRaL's use of it.
Source curriculum. The domain-free training data is procedurally generated. It consists of word problems and small tables from non-medical settings such as retail inventory, physics labs, and sports statistics. Each item comes with a known formula and gold bindings. Every item is corrupted by one or more of four operators. Label rewriting replaces a variable's name with a synonym, an abbreviation, or an arbitrary renamed column header. Same-label distraction inserts other values under the same label, for example at a different time, for a different entity, or as a reference range. Unit shifting states values in alternative units. Misleading auxiliary text adds a snippet that asserts a wrong formula variant.
A second curriculum applies the same operators to synthetic clinical notes, generated from calculator formulas but never from MedCalc-Bench notes. A third mixes both. We will verify there is no overlap in text or patient content with the evaluation notes.
Perturbed evaluation suite. We apply the same four operators to the equation-based portion of MedCalc-Bench. Because we control the injected distractors, the gold binding under perturbation is known by construction. For clean notes, we will have clinicians annotate gold bindings for a subset of equation-based instances; we do not assume the benchmark ships them. We restrict binding scoring to equation-based tasks because the file notes that the rule-based tasks involve clinician judgment and therefore some label noise. Rule-based tasks are reported on final-answer exact match only.
Baselines
The naive reference point is the same base model, Qwen2.5-7B-Instruct, prompted with one-shot chain of thought (writing out reasoning steps before answering), exactly as in MedCalc-Bench's protocol. For scale context we quote that benchmark's reported numbers, taken from its paper rather than rerun: GPT-4 reaches 50.9% one-shot chain-of-thought accuracy and Llama 3-70B reaches 39.5% (MedCalc-Bench). The file reports no MedCalc-Bench number for a 7B Qwen model beyond open models hovering between 0% and 20% zero-shot, so we measure our own clean baseline.
The first ablation is tool-only. The base model gets the same calculator and binding format with no training. This separates what the calculator alone fixes (arithmetic errors) from what training fixes (binding errors).
The strongest competitor is answer-only reinforcement learning in-domain. This is GRPO on the synthetic clinical curriculum with only the ±5% answer reward. It is the standard verifiable-reward recipe used by TabReason and TableGPT-R1, applied to our task. Its advantage is that it trains on the target domain. It is our key ablation, because it differs from our method only in the binding term. Setting $\lambda = 0$ recovers it exactly.
We also run an AbstRaL-style baseline: GRPO with AbstRaL's answer-plus-symbolic-distance reward on the domain-free curriculum. This tests whether a generic symbolic abstraction transfers as well as grounded binding does. No reported MedCalc-Bench result was found on Lacuna for AbstRaL.
Finally we run an Ext2Gen-style baseline: sentence extraction trained with preference optimization before answering, retrained on our data. Ext2Gen reports that this lifts Llama-3.1-8B-Instruct from 0.473 to 0.557 average QA score under heavy retrieval noise. No reported MedCalc-Bench result was found on Lacuna for it.
Every trained system uses the same base model, the same number of training prompts and rollouts, the same GPU-hour budget, and the same hyperparameter search size (twelve configurations, chosen on a held-out synthetic split, never on MedCalc-Bench). Beating the strongest competitor requires two things. On the perturbed suite, our method's accuracy drop from clean to perturbed must be at least halved relative to answer-only reinforcement learning, with overlapping or better clean accuracy. And the domain-free-only variant must recover at least most of the in-domain variant's perturbed-suite accuracy. As a further bar, a 7B model surpassing Llama 3-70B's reported 39.5% on the clean equation subset would be notable, though clean accuracy is not our main claim.
Why not just…?
Why not just give the model a calculator? A calculator removes arithmetic errors, but MedCalc-Bench attributes a large share of failures to extraction, which a calculator cannot fix. The tool-only baseline measures this directly. If tool-only recovers nearly all of the gap on the perturbed suite, our training contribution is small, and we will report that.
Why not just run answer-only reinforcement learning on synthetic clinical notes? This is the real threat. It is cheap, standard, and in-domain. The case against it is that an answer reward credits lucky or compensating bindings, the same kind of disconnect TabReason observed between explanations and predictions. It also gives no pressure to resist same-label distractors that happen to yield answers within 5%. The $\lambda = 0$ ablation settles this. If answer-only training matches our robustness, the binding reward is unnecessary, and the proposal reduces to recommending in-domain answer-only RL.
Why not just filter context first, with retrieval or Ext2Gen-style extraction? Sentence filtering removes irrelevant text. Our hardest distractors are relevant sentences, such as an earlier creatinine value or a reference range, that carry the wrong number. UDA shows retrieval helps a lot on long documents, which is why we include retrieval in the financial experiments. The Ext2Gen baseline tests whether sentence-level selection is enough. We expect it to help on irrelevant-text perturbations and not on same-label ones, and the per-operator breakdown will show whether that holds.
Why not just let a reasoning model think longer? Thinking in a Crowd shows that longer thinking under misleading context produces rationalization, not rejection. We include Qwen3-8B with thinking on and off as a reference on the misleading-snippet operator. We will also test whether forcing an evidence-span commitment before reasoning reduces the length of the justifications it builds for false premises.
Proposed Experiments
The primary evaluation is MedCalc-Bench's equation-based tasks, clean and under each of the four perturbation operators, scored with the benchmark's ±5% tolerance. We also report binding accuracy per input and the rate at which evidence spans actually contain the claimed value. The error decomposition separates wrong formula, wrong binding, and wrong unit.
The decisive transfer check compares domain-free-only, clinical-only, and mixed training on the perturbed clinical suite. The key quantity is how much of the clinical-only model's robustness the domain-free-only model recovers.
A second target domain tests whether the skill generalizes beyond medicine. We use UDA's FinHybrid and TatHybrid with the retrieval pipeline held fixed. The reference point is GPT-4 with retrieval at 45.9 on FinHybrid (UDA), which we cite rather than claim to match.
To probe the metadata-dependency axis on structured input, we render each MedCalc-Bench note's lab values as a table and apply header synonym swaps modeled on Entity Swap. We then measure the accuracy drop against that paper's reported 43% F1 degradation pattern.
Finally, a sanity check guards against forgetting: in-source accuracy on GSM-Plus distractor problems and general reasoning on GSM8K must not fall by more than one point.
The result that would most change our view is this: if domain-free training improves the synthetic suites but not the real clinical notes, then binding learned on synthetic distractors does not match the way real clinical notes mislead a model. In that case we would redirect the effort toward cheap in-domain generation of synthetic clinical notes.
References
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
- Adversarial Attacks on Tables with Entity Swap
- Evaluating large language models in pediatric fever management: a two-layer study
- Ask the experts: sourcing a high-quality nutrition counseling dataset through Human-AI collaboration
- TabLLM: Few-shot Classification of Tabular Data with Large Language Models
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking
- CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks
- Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic
- TabReason: A Reinforcement Learning-Enhanced Reasoning LLM for Explainable Tabular Data Prediction
- TableGPT-R1: Advancing Tabular Reasoning Through Reinforcement Learning
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Language Models Are Good Tabular Learners