LongForge: Measuring and Improving Long-Context Instruction Data via Evidence-Assertion Reasoning Graphs

Ziyang Chen, Junlong Jia, Xing Wu, Tinghao Yu, Songlin HuarXiv 2026 · arXiv

TL;DR

LongForge introduces a novel framework for long-context instruction tuning that moves beyond simple quality scores to a structured Evidence-Assertion Reasoning Graph. By analyzing and synthesizing data based on four key dimensions—breadth, depth, correctness, and completeness—the authors demonstrate that the structural distribution of training data is a powerful lever for improving downstream model performance.

What Problem Does This Paper Address?

Training models to handle long contexts (like entire books or codebases) requires high-quality instruction-tuning data. However, current quality control is a "black box." Most pipelines produce a single scalar score (e.g., "this sample is a 7.5/10") without explaining how the model used the document or how many reasoning steps it took.

This lack of transparency makes data unauditable (you can't verify the reasoning) and uncontrollable (you can't easily adjust the training distribution to favor more complex reasoning). The authors argue that quality should be defined by the alignment structure between the document, the question, and the answer.

Methods

The core of the paper is LongForge, which consists of two main components:

  1. The Analyzer: This tool reverse-reconstructs any (Document, Question, Answer) triple into a reasoning graph.
  2. Nodes: Verbatim document spans (Evidence) and answer sentences (Assertions).
  3. Edges: Reasoning steps connecting evidence to assertions.
  4. Metrics: It programmatically calculates four dimensions:
  5. Breadth: How many distinct parts of the document were used?
  6. Depth: How many reasoning steps were taken (after collapsing simple paraphrases)?
  7. Correctness: Are the reasoning steps actually supported by the source text?
  8. Completeness: Does the answer address all parts of the question?
Overview of the LongForge framework, showing the transition from scalar judgments to structural graphs.
  1. The Synthesizer: This component uses the Analyzer's feedback to generate better data. It uses a "gate" to reject or re-answer samples with low correctness or completeness. Crucially, it allows researchers to "mix" training sets based on breadth and depth targets without needing to regenerate the data.
The detailed Analyzer pipeline, from splitting the document to measuring the four dimensions.

Datasets

The authors evaluated seven mainstream long-context datasets: LongAlign, LongAlpaca, LongReward, SEALONG, LongMIT, LongMagpie, and LongFaith.

For evaluation, they tested two backbones (Llama3.1-8B and Qwen3.5-4B) across six benchmarks: LongBench, LongBench v2, LooGLE, MRCR, HELMET, and RULER.

Key Insights and Ideas

  • The Breadth Plateau: A major finding is that in existing datasets, "breadth" barely scales with document length. Even if a document is 10x longer, models typically only use about 40% more evidence. Most of the document remains "starved" or unused.
  • Shallow Reasoning is the Norm: Approximately 43.4% of samples in current datasets have a depth of 1, meaning they involve no explicit intermediate reasoning—they just point to a fact and restate it.
  • Orthogonality: Breadth and depth are nearly independent (orthogonal). Increasing the number of facts a model looks at doesn't naturally lead to deeper reasoning, and vice versa.
  • Auditability: Because every assertion is linked to a specific evidence span in the graph, human auditors can verify the model's work step-by-step. The authors found that their automated analyzer matches human judgment with over 90% agreement.
Analysis of evidence count and document utilization across baseline datasets.

Key Results

LongForge produced two data mixtures: LongForge-1 (single-turn) and LongForge-N (multi-turn dialogues designed to maximize document utilization).

  • Performance Gains: On average, LongForge mixtures outperformed the strongest baseline (LongMagpie) by 2.2–4.7 points across the six benchmarks.
  • The "Lever" Effect: In a controlled experiment, simply shifting the training distribution toward higher breadth and depth (while keeping the total number of samples and the "gate" quality fixed) accounted for a 1.16-point improvement. This proves that structure is a primary driver of quality.
  • Robustness: The gains were consistent across different model backbones (Llama and Qwen) and did not degrade the models' performance on short-context tasks.
Main results across six benchmarks for Llama3.1-8B and Qwen3.5-4B backbones.

Limitations and Future Work

  • Computational Cost: The Analyzer is computationally expensive. It requires scanning every chunk of a document for evidence, which dominates the synthesis cost. The authors suggest that smaller, specialized extraction models might reduce this overhead in the future.
  • Mixture Optimization: While the paper shows that raising breadth and depth helps, it doesn't define the optimal mixture. Future work could involve more systematic searches for the "sweet spot" of these structural dimensions.
  • Reasoning Diversity: The current graph focuses on evidence-assertion links. Future iterations could incorporate more diverse reasoning types, such as mathematical or symbolic logic, into the graph structure.

Summary updated

Abstract

Existing quality control for long-context instruction-tuning data is a one-shot judgment inside the generation pipeline that leaves only a scalar, revealing neither which document parts an answer uses nor how many reasoning steps it writes down; the data can thus be neither audited nor controlled. We formalize data quality as a property of the alignment structure among document, question, and answer, and build LongForge, which pairs an analyzer that measures long-context data with a synthesizer that generates it. The analyzer reverse-reconstructs each sample into an evidence-assertion reasoning graph and programmatically computes four dimensions on it: breadth, depth, correctness, and completeness. Every reading traces back to specific evidence spans and reasoning steps, making data auditable. Measuring seven mainstream datasets reveals two previously unquantified defects: breadth barely grows with document length, and depth is generally low, the two being nearly orthogonal. The synthesizer then embeds the analyzer into generation, gating on correctness and completeness and deferring breadth and depth to a training-time mixture over measured values, which makes the training distribution controllable. On two backbones, its two mixtures, LongForge-1 and LongForge-N, outperform the strongest baseline by 2.2-4.7 points on average over six long-context benchmarks, and in a controlled comparison that holds the gate and the candidate pool fixed, raising breadth and depth alone is worth a further 1.16 points, making the structural distribution of training data a lever on downstream quality rather than only a description of it.
BibTeX
@misc{chen2026longforge,
  author = {Ziyang Chen and Junlong Jia and Xing Wu and Tinghao Yu and Songlin Hu},
  title = {{LongForge: Measuring and Improving Long-Context Instruction Data via Evidence-Assertion Reasoning Graphs}},
  year = {2026},
  eprint = {2610.23167},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.23167},
}
Figure