TL;DR
LongForge introduces a novel framework for long-context instruction tuning that moves beyond simple quality scores to a structured Evidence-Assertion Reasoning Graph. By analyzing and synthesizing data based on four key dimensions—breadth, depth, correctness, and completeness—the authors demonstrate that the structural distribution of training data is a powerful lever for improving downstream model performance.
What Problem Does This Paper Address?
Training models to handle long contexts (like entire books or codebases) requires high-quality instruction-tuning data. However, current quality control is a "black box." Most pipelines produce a single scalar score (e.g., "this sample is a 7.5/10") without explaining how the model used the document or how many reasoning steps it took.
This lack of transparency makes data unauditable (you can't verify the reasoning) and uncontrollable (you can't easily adjust the training distribution to favor more complex reasoning). The authors argue that quality should be defined by the alignment structure between the document, the question, and the answer.
Methods
The core of the paper is LongForge, which consists of two main components:
- The Analyzer: This tool reverse-reconstructs any (Document, Question, Answer) triple into a reasoning graph.
- Nodes: Verbatim document spans (Evidence) and answer sentences (Assertions).
- Edges: Reasoning steps connecting evidence to assertions.
- Metrics: It programmatically calculates four dimensions:
- Breadth: How many distinct parts of the document were used?
- Depth: How many reasoning steps were taken (after collapsing simple paraphrases)?
- Correctness: Are the reasoning steps actually supported by the source text?
- Completeness: Does the answer address all parts of the question?
- The Synthesizer: This component uses the Analyzer's feedback to generate better data. It uses a "gate" to reject or re-answer samples with low correctness or completeness. Crucially, it allows researchers to "mix" training sets based on breadth and depth targets without needing to regenerate the data.
Datasets
The authors evaluated seven mainstream long-context datasets: LongAlign, LongAlpaca, LongReward, SEALONG, LongMIT, LongMagpie, and LongFaith.
For evaluation, they tested two backbones (Llama3.1-8B and Qwen3.5-4B) across six benchmarks: LongBench, LongBench v2, LooGLE, MRCR, HELMET, and RULER.
Key Insights and Ideas
- The Breadth Plateau: A major finding is that in existing datasets, "breadth" barely scales with document length. Even if a document is 10x longer, models typically only use about 40% more evidence. Most of the document remains "starved" or unused.
- Shallow Reasoning is the Norm: Approximately 43.4% of samples in current datasets have a depth of 1, meaning they involve no explicit intermediate reasoning—they just point to a fact and restate it.
- Orthogonality: Breadth and depth are nearly independent (orthogonal). Increasing the number of facts a model looks at doesn't naturally lead to deeper reasoning, and vice versa.
- Auditability: Because every assertion is linked to a specific evidence span in the graph, human auditors can verify the model's work step-by-step. The authors found that their automated analyzer matches human judgment with over 90% agreement.
Key Results
LongForge produced two data mixtures: LongForge-1 (single-turn) and LongForge-N (multi-turn dialogues designed to maximize document utilization).
- Performance Gains: On average, LongForge mixtures outperformed the strongest baseline (LongMagpie) by 2.2–4.7 points across the six benchmarks.
- The "Lever" Effect: In a controlled experiment, simply shifting the training distribution toward higher breadth and depth (while keeping the total number of samples and the "gate" quality fixed) accounted for a 1.16-point improvement. This proves that structure is a primary driver of quality.
- Robustness: The gains were consistent across different model backbones (Llama and Qwen) and did not degrade the models' performance on short-context tasks.
Limitations and Future Work
- Computational Cost: The Analyzer is computationally expensive. It requires scanning every chunk of a document for evidence, which dominates the synthesis cost. The authors suggest that smaller, specialized extraction models might reduce this overhead in the future.
- Mixture Optimization: While the paper shows that raising breadth and depth helps, it doesn't define the optimal mixture. Future work could involve more systematic searches for the "sweet spot" of these structural dimensions.
- Reasoning Diversity: The current graph focuses on evidence-assertion links. Future iterations could incorporate more diverse reasoning types, such as mathematical or symbolic logic, into the graph structure.
Summary updated