Large language models often find the right facts in a 100,000-token document and still fail to return them in the requested format. They break the JSON, drop required fields, or slide back into chatty prose. We propose to test a simple explanation that nobody has checked. Models may hold a format together over long inputs mainly when the format is one they memorized during pre-training, such as a familiar API response or a common clinical record layout. When the schema (the required output structure) is genuinely new, the instruction may have to be tracked actively, and that is what fails as the input grows. If this is right, current long-context benchmarks and the attention-boosting fixes evaluated on them overstate how well models follow instructions, much as static coding benchmarks overstated reasoning before time-split evaluation exposed the gap.
The study crosses three factors in one design. The first is input length. The second is document recency: documents released before versus after a model's training cutoff, which is the contamination control known as temporal isolation. The third is schema familiarity: schemas mined from public corpora versus schemas freshly generated with controlled nesting depth. On the mechanistic side, we ask whether the model's internal representation of a nested schema loses its tree shape as the input grows, and whether that loss predicts a broken output before any output is generated. We measure tree shape with a hyperbolic probe, a small map into negatively curved space, which is well suited to trees. We include it only as a hypothesis that must beat a flat Euclidean probe.
Core Claim
Long-context structural degradation is not one curve. It is an interaction between length and schema novelty. Compliance on familiar schemas should stay nearly flat with length. Compliance on novel schemas of the same depth should fall steeply. Document recency should mainly affect content accuracy, not compliance. A sharper corollary is that training-free attention interventions should recover much less of the lost compliance on novel schemas than on familiar ones, because the failure for novel schemas is not only attention dilution. Each of these is a direct, falsifiable prediction. The benchmark, whatever the outcome, separates "follows the instruction" from "recalls the format."
Why This Gap Is Real
Several papers motivate the question without answering it. MedOdyssey evaluates medical documents from 4K to 200K tokens and reports that open-source models fail to emit the requested JSON even when retrieval succeeds. It reports this only qualitatively: no reported result found on Lacuna for a numerical long-context JSON-compliance rate. CURIE tests scientific long-context extraction, and its best model, Gemini 2.0 Flash, reaches only 32% average normalized score. Both use static document collections and fixed output formats, so neither can tell whether success reflects familiarity with the format.
Temporal isolation, as practiced by LiveCodeBench and LiveBench, has so far been applied only to short tasks. LiveCodeBench time-stamps programming problems and evaluates only those released after each model's cutoff. LiveBench refreshes math, coding, and instruction-following questions monthly and finds that instruction following correlates weakly with reasoning.
Test of Time sets a constraint on our design. It shows that rephrasing LiveCodeBench problems erases the post-cutoff performance drop. It also shows that influence-function tracing on OLMo2-7B-Instruct finds a training source for 77.5% of verbatim fill-in-the-blank items but only 17.5% of transformed ones. The lesson is that recency alone is a weak contamination signal. We therefore do not rely on it for the factor we care most about. Schema familiarity is controlled by construction (freshly generated schemas) and by direct counting in OLMo2's open pre-training data, in the spirit of Snoopy, which ties downstream accuracy to pre-training term frequency.
The method for generating novel schemas comes from DyVal. DyVal builds reasoning problems from randomly generated directed acyclic graphs with tunable depth and width. It finds that depth hurts much more than width, and that GPT-3.5-Turbo's arithmetic accuracy drops 23% from its shallowest to its deepest level. We reuse the idea for output schemas rather than for reasoning problems, and we embed the instructions in long natural documents rather than short synthetic prompts.
The interventions we stress-test come from two training-free attention-steering papers:
- INSTABOOST (Instruction Following by Principled Attention Boosting) adds a constant bias to the attention logits of instruction tokens.
- GUIDE (Pay Attention to What Matters) biases attention toward user-tagged instruction spans and introduces an "Influence" metric. Influence weights attention flow by token embedding norms and correlates 0.72 with task success, versus 0.13 for raw attention.
Both papers report a narrow "Goldilocks" range for the bias. Too large a bias suppresses the context the model needs to answer.
Finally, the hyperbolic probe draws on Poincaré Embeddings, which embed large taxonomies in two or three hyperbolic dimensions with little distortion. It also draws on Coneheads, where 16-dimensional hyperbolic cone attention matches 128-dimensional dot-product attention. We do not modify any model's architecture. Hyperbolic transformers have been validated only up to roughly 500M parameters, and that risk is unnecessary for answering a measurement question.
Research Program
Documents. We build a rolling pool of long documents from sources with reliable release dates: arXiv papers, news, and public reports. We assemble them into inputs of 8K, 16K, 32K, 64K, and 128K tokens, truncated to each model's supported window. Each input pairs one extraction instruction with one schema.
Schemas. Familiar schemas are mined from public schema collections and frequent JSON layouts. We confirm familiarity by counting their key paths in the Dolma corpus used to train OLMo2. Novel schemas are generated as random trees whose depth (2 to 6) and branching match a familiar counterpart. Key names are newly composed, and the trees carry cross-field constraints such as "field B must list every entity named in field A."
Scoring. Every item is scored on two separate axes. Compliance is automatic schema validation, including the cross-field constraints. Content accuracy is field-level match against references that we extract and verify by hand on a subset. Matched pairs (same document, same depth, familiar versus novel schema) let us estimate the interaction effect without confounding from document difficulty. Matching also addresses the difficulty-drift confounder that Test of Time raised for time splits.
Probe. For the mechanistic analysis we take hidden states at the instruction's schema tokens. We fit a low-rank probe into the Poincaré ball, trained so that hyperbolic distances reproduce tree distances in the schema. We define tree recoverability as the probe's distortion on held-out schemas, and we track it as a function of the length of the context placed after the instruction. The prediction is that recoverability decays with length for novel schemas and that this decay predicts compliance failure before generation.
Baselines
Instruction only. The naive reference is the unmodified model with the instruction at the start of the input. This is the "Instruction-only" condition in INSTABOOST, which scores 0.860 on its AI Persona task against 0.925 for INSTABOOST. The comparison here is not about beating this baseline. It is about whether the gap between familiar and novel schemas appears under it.
Attention steering. The strongest existing interventions are INSTABOOST and GUIDE. Each is run on the same inputs. For both methods we use the same small validation slice per model for tuning, with the same number of trials. INSTABOOST's bias is swept over a fixed grid. For GUIDE, schema spans are tagged automatically, giving it its best case without manual markup. The decisive outcome is the ratio of compliance recovered on novel versus familiar schemas. The core claim predicts recovery on novel schemas at most half that on familiar ones at 64K and above. Equal recovery would falsify the claim that novel-schema failure is more than dilution.
Trained hierarchy. As a training-based reference we run HIPO (HIPO) under a matched compute budget on Qwen3-8B. HIPO is constrained reinforcement learning that treats system-prompt compliance as a hard constraint. On short prompts it reaches 0.7 system compliance where SFT and DPO hover near 0.6 on Qwen-1.7B. This baseline tells us whether training removes the familiarity gap, which matters for whether the gap is fundamental.
Pre-generation failure predictors. The strongest reported baseline is GUIDE's Influence metric, at 0.72 correlation with success. We also include raw instruction attention mass (0.13 in the same paper). All predictors are scored on identical held-out items by area under the ROC curve for predicting compliance failure.
Key ablation. The ablation that isolates the hyperbolic ingredient is a Euclidean structural probe of identical rank and training budget. The hyperbolic probe counts as beating it only if it improves failure prediction by a margin outside the bootstrap confidence interval, at matched rank. We test ranks 2, 4, and 8, where Coneheads' evidence suggests curvature should matter most.
Why not just…?
Why not just run INSTABOOST or GUIDE at long context and report the gains? That would show whether boosting helps. It would not show what the remaining failures are. Both papers evaluate on tasks whose output formats are common, so their gains could come from re-activating a memorized template. The matched familiar-versus-novel design is exactly the experiment that separates the two readings. If boosting recovers novel schemas as well as familiar ones, we report that the simpler story is correct.
Why not just use DyVal-style synthetic generation and skip temporal isolation? We do use synthetic generation for schemas, but DyVal's own limitation is its narrowness: synthetic content does not reproduce the density of real documents. The long context must be natural text for the results to transfer to settings like MedOdyssey. Temporal isolation is the cheapest way to make that natural content unseen. Because Test of Time shows recency is a fragile signal, we treat it as a secondary factor and expect it to affect content accuracy more than compliance. Observing the reverse would itself be a finding.
Why not just measure schema familiarity by counting or perplexity instead of time splits? This is not an alternative but part of the design. Counting in OLMo2's open data is our primary familiarity measure. For closed models, schema perplexity under the model stands in, and we report how well the two agree on OLMo2.
Why not just use a Euclidean probe, attention mass, or GUIDE's Influence metric instead of a hyperbolic probe? This is a real threat. Euclidean probes of modest rank often recover syntactic trees well, and the hyperbolic advantage shown in Coneheads was for trained attention, not for probes of frozen models. The rank-matched ablation settles it. If the Euclidean probe or Influence predicts failure as well, we drop the geometric claim. The behavioral result, the interaction between length and schema novelty, does not depend on the probe.
Why not just fine-tune on long structured-output data? HIPO reports SFT and DPO stalling near 0.6 compliance on short conflicting prompts. Fine-tuning on familiar formats would also deepen exactly the confound we are measuring. We include HIPO as the training-based reference instead.
Decisive Checks
Familiarity gap. On OLMo2-7B-Instruct, Qwen3-8B, and LLaMA-family models, the study succeeds if the compliance gap between matched familiar and novel schemas grows with length. Specifically, the interaction coefficient in a logistic model of compliance on length, novelty, recency, and depth must be significantly positive and survive the matched-pair controls.
Selective recovery. The study fails on its main claim if attention steering recovers novel and familiar schemas at equal rates.
Probe. The probe component counts as successful only if it beats both the rank-matched Euclidean probe and the Influence metric at predicting compliance failure before generation. Otherwise it is reported as a negative result.
Short-context control. As a sanity check, novel and familiar schemas should show near-equal compliance at short context, where LiveBench-style instruction following is already measured. Without that, the gap would reflect schema difficulty rather than the effect of length.
References
- MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokens
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Instruction Following by Principled Attention Boosting of Large Language Models
- Pay Attention to What Matters
- Poincaré Embeddings for Learning Hierarchical Representations
- Coneheads: Hierarchy Aware Attention
- HIPO: Instruction Hierarchy via Constrained Reinforcement Learning