Large language models get worse at extracting a relation as the relation becomes more complex, for example when one clinical finding is an argument of another. The clearest evidence comes from CARE, where GPT-4 leads relation extraction yet degrades as arity and nesting increase. That curve can be read three ways. Models may truly fail at recursive structure. The expert labels on deep relations may be noisier, so the benchmark itself bends the curve. Or shallow relations may be easier partly because their phrasings and co-occurrences were seen in pretraining. A static, 700-abstract benchmark cannot tell these apart, yet the choice matters: each reading points to a different fix, namely better architectures, re-annotation, or fresh evaluation data.
We propose a generator of nested extraction problems whose labels are correct by construction. It produces synthetic "findings" documents whose statement trees have exactly controlled depth and arity, drawn from a typed hierarchy of statement templates. Each tree is rendered in two matched forms: realistic clinical vocabulary, and invented (nonce) terms that cannot have been memorized. The generator lets us measure the depth curve with no label noise and no possible contamination. We then add controlled, realistic noise to the generated gold labels to find how much noise at each depth is enough to erase the ranking between systems. Finally, we set the synthetic curve beside the CARE curve and a small re-annotation audit of CARE's deep relations, so the observed degradation can be split into its three sources.
Core Claim
The degradation with nesting depth reported on CARE is a mixture of structural failure, depth-dependent annotation noise, and familiarity effects. A generator whose labels are exact by construction, with each surface form paired against a nonce twin, can estimate the size of each part. The testable prediction has two halves. First, on noise-free synthetic data the depth curve will remain but be flatter than on CARE. Second, the gap between the curves will be accounted for by measured disagreement among annotators on CARE's deep relations, plus the gap between realistic and nonce renderings. If the synthetic curve is as steep as CARE's and nonce renderings cost nothing, the simple reading, that models cannot handle recursion, stands. That would also be a useful result.
Positioning Against Prior Work
The work closest to ours is DyVal. It generates directed acyclic reasoning graphs on the fly to avoid contamination, and it shows that depth hurts models more than width. Its tasks are arithmetic, boolean logic, and graph search, and its authors note that the approach does not reach linguistic extraction. We keep its central move, generating the structure first so the answer is known, but apply it to text-to-structure extraction over typed, nested statements. We also add two things it does not do: a matched real-versus-nonce contrast, and a controlled label-noise study.
CARE supplies the schema we reuse: n-ary findings built from entity and attribute types, in which observations serve as attributes of other observations (11% of its relations are nested). It is also the motivating evidence for this proposal. It is static, however, and it never asks whether its own labels are noisier at depth.
Two papers constrain how we model noise. Re-TACRED found 23.9% of TACRED labels wrong, mostly true relations marked as no relation. That noise made PA-LSTM and C-GCN look identical until the labels were cleaned. CIFAR-N showed that real human noise depends on the individual example and drifts toward common super-classes. Its 1–5% gap between synthetic and human noise warns that uniform random corruption is the wrong model. Both studies concern flat labels. We carry their two observed error patterns, missed relations and drift toward the parent class, into trees, where a dropped inner statement also orphans its children.
The two-stage ICD decoder predicts a parent code and then a child code conditioned on it. We use this parent-then-child view as a scoring lens rather than as a model: a prediction that gets the parent template right but the leaf wrong earns partial credit.
Snoopy links pretraining term frequency to accuracy. That link motivates the nonce arm, although frequency is correlational evidence and not proof of memorization. LiveBench and MMLU-Pro address contamination and noise at the level of whole benchmarks. Neither addresses structured extraction.
Research Program
The generator starts from a typed template hierarchy derived from CARE's schema. Top-level template families such as intervention outcome, condition-conditioned observation, and comparison split into leaf templates, and each leaf fixes a set of argument slots with their allowed types. Some slots accept a whole statement, which is what allows recursion. We sample trees with specified depth (1 to 4), branching, and arity, then fill the slots.
Slot filling has two modes. The realistic mode draws drug, condition, and measure names from clinical vocabularies. The nonce mode uses pronounceable invented words that do not occur in common corpora. Because each nonce argument is a unique string, span recovery can be checked exactly.
Trees are rendered to prose by a grammar-based realizer with many surface patterns for each template. The patterns include forward and reversed ordering, echoing DyVal's order effect, and discontinuous argument spans, matching the 9% rate in CARE. A second rendering arm has an LLM paraphrase the grammar output for fluency. Paraphrases are accepted only if every argument string survives. We audit 200 accepted paraphrases by hand and report their residual error rate, and we keep this arm separate because it is not noise-free.
The analysis has three parts. First, we measure accuracy against depth on grammar-rendered data, in both realistic and nonce forms. Second, we take the exact gold labels and add noise at a controlled rate at each depth, using two patterns. The first drops an inner statement together with everything beneath it, following Re-TACRED's missed-relation bias. The second replaces a leaf template with its most frequent sibling, following CIFAR-N's drift toward common classes. For each noise pattern and depth, we find the smallest rate at which the ranking between two systems stops being statistically distinguishable. Third, two annotators with clinical training independently re-annotate about 100 CARE abstracts, oversampling nested relations. This gives an empirical disagreement rate at each depth, which we place on the noise-budget curve to ask whether CARE's deep levels already lie in the zone where rankings are unreliable.
Baselines
The naive reference point is simply to accept CARE's reported numbers and read the arity curve as a pure capability curve. The strongest reported result there is GPT-4 (5-shot) with relation F1 of 55.04 (CARE). We rerun a GPT-4-class model with the same 5-shot protocol so our CARE numbers can be compared to that value. Beating this reading means showing that a substantial, measured share of the depth drop is explained by annotator disagreement or by the realistic-versus-nonce gap.
The strongest extraction system for entities and attributes on CARE is PURE, which that paper reports outperforms GPT-4 at entity and attribute recognition. No reported numeric result was found on Lacuna for PURE on CARE. We fine-tune PURE on CARE's training split and, separately, on generated data of matched size. Every system gets the same data, the same development-set tuning budget, and, for prompted models, the same number of in-context examples drawn from the same pool. We also run open models (Llama-3.1 and Qwen2.5 at their largest available sizes) under identical prompts.
The strongest existing method for contamination-free structural evaluation is DyVal-style generation. As a comparison point, we run DyVal's own depth sweep on the same models to see whether their depth sensitivity on formal tasks predicts their sensitivity on nested extraction. If it does, our generator adds little beyond DyVal. That would count against us.
Three ablations isolate the key ingredients. The first flattens each tree into its reified first-normal-form equivalent. Every nested statement becomes an auxiliary node with flat binary edges, while content and surface text stay identical. This separates nesting from the sheer number of relations. The second replaces the hierarchical partial-credit metric with flat exact-tuple F1. The third replaces structured noise with uniform random noise at the same rate. The generator is judged successful if the flattening ablation shows depth has an effect beyond relation count, and if structured noise erases rankings at a lower rate than uniform noise does.
Why not just…?
Why not just have an LLM clean CARE, or judge nested outputs directly? The labels that most need checking are the deep ones, and those are exactly where the models are weakest. LiveBench reports that LLMs are poor judges of problems they cannot solve, and GPT-4 reaches only 55.04 relation F1 on CARE. A judge with that error profile would distort the depth curve in the same direction as the models under test. We still run an LLM cleaning pass on the audited CARE subset and report how well it agrees with the human re-annotation at each depth. That measures the alternative directly rather than dismissing it.
Why not just perturb CARE itself, swapping entity names for nonce words, instead of generating from scratch? This is a real threat to the memorization part of the proposal, so we include it as an arm. Nonce-swapped CARE abstracts test familiarity while keeping natural prose, and if the swapped-CARE gap matches the synthetic realistic-versus-nonce gap, that half of the question is settled without the generator. What swapping cannot do is remove CARE's label noise or vary depth independently of topic. The noise-budget and flattening experiments therefore still require generated data.
Why not just use DyVal, since it already controls depth? DyVal's tasks never ask a model to locate typed spans in prose and assemble them into a tree, and that step is where CARE's errors concentrate. The cross-benchmark correlation described under Baselines tests this alternative empirically. A high correlation would mean the extraction setting adds little new information.
Why not just correct the noise statistically at evaluation time? CIFAR-N's 1–5% gap between synthetic and human noise shows that class-level noise models misrepresent real errors. In trees the problem is worse, because one missed inner statement removes a whole subtree. Our comparison of uniform and structured noise quantifies how wrong a flat correction would be.
Decisive Checks
The first check compares depth curves. On grammar-rendered nonce data, is the slope of relation F1 against depth significantly shallower than on CARE, using bootstrap confidence intervals across items? The second check compares audited CARE disagreement at depths 3–4 with the noise rate at which synthetic rankings collapse; if the audited rate exceeds that threshold, CARE's deep-level rankings should not be trusted as they stand. The third check asks whether the realistic-minus-nonce gap is non-zero and grows or shrinks with depth, confirmed by the swapped-CARE arm. The fourth check asks whether depth still hurts after flattening to equal relation counts.
Together these determine what the fix should be. A steep curve on noise-free nonce data with little audited disagreement means the problem is modeling. A curve that flattens once noise is removed means CARE's deep levels need re-annotation. A large gap between realistic and nonce renderings means extraction scores on public clinical text overstate what models can do on unfamiliar material.
References
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- CARE: Extracting Experimental Findings From Clinical Literature
- Re-TACRED: Addressing Shortcomings of the TACRED Dataset
- Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
- A Two-Stage Decoder for Efficient ICD Coding
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark