Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning

Jingyang Lin, Andy Kin On Wong, Xia Tian, Shenghua He, Hui Wei, Mei Han, Jiebo Luo2025 · DOI

Abstract

Recent advances in Large Language Models (LLMs) have enabled processing of sequences up to 2M tokens. However, extended context windows do not inherently guarantee effective understanding. This study introduces Property-based Agentic Inference (PAI) and LongFinanceQA, a synthetic dataset designed to integrate supervised Chain-of-Thought (CoT) reasoning into long-context models. By training models to perform explicit intermediate reasoning steps—extraction, retrieval, and synthesis—we bridge the gap between simple information retrieval and complex analysis. Our approach, evaluated on the Loong benchmark, demonstrates that GPT-4o-mini equipped with PAI outperforms its base version by 20.3%, surpassing even state-of-the-art models with much larger context windows.

The Reasoning Gap in Long Contexts

The current "long-context" paradigm often focuses on engineering larger windows rather than improving the model's ability to reason over the data within them. Standard training pairs massive documents with concise answers, forcing models to leap from input to conclusion without intermediate logic. This leads to a "performance ceiling" where models fail on tasks requiring multi-step synthesis across disparate document sections.

Figure 1: Comparison of model performance (Average Score) against context window size. The PAI-enhanced model achieves superior reasoning scores despite a standard context window, highlighting that reasoning capability is the primary bottleneck.

Methodology: Property-based Agentic Inference (PAI)

To address the lack of high-quality reasoning data, we developed the PAI framework. PAI mimics human financial analysis by decomposing complex queries into verifiable "properties" (Metric + Subject).

Figure 2: The Property-based Agentic Inference (PAI) framework. It consists of three stages: (1) Property Extraction, (2) Property-based Retrieval, and (3) Summarization, resulting in a structured CoT reasoning trace.

The framework operates in three distinct stages:

  1. Property Extraction: The model identifies specific evidence properties needed to answer a query (e.g., "Revenue" for "Company A in 2019").
  2. Property-based Retrieval: These properties are used to retrieve relevant 1024-token chunks from the long context.
  3. Summarization: The retrieved facts are synthesized into a final conclusion, preceded by an explicit reasoning trace.

LongFinanceQA and Distillation

Using PAI, we generated LongFinanceQA, a dataset of ~46k QA pairs from 6,911 financial reports. We then fine-tuned a LLaMA-3.1-8B model on these reasoning traces. This "distilled" model, LongPAI, learns to generate the reasoning and answer in a single pass, achieving 30x the token efficiency of the multi-step agentic baseline.

Key Results and Insights

The PAI-enhanced GPT-4o-mini achieved an overall score of 69.58% on the Loong benchmark, a significant jump from the base model's 49.25%.

  • Efficiency through Distillation: LongPAI (8B) achieved a 28.0% gain on financial subsets compared to its base model while using less than 3% of the tokens required by the full agentic pipeline.
  • Scalability of Reasoning: The performance advantage of CoT-trained models became more pronounced as context length increased (e.g., from 50k to 250k tokens), suggesting that explicit reasoning is essential for navigating larger "haystacks."
  • Generalization: Despite being trained on financial data, the model showed improved performance on general-purpose benchmarks like ∞Bench, indicating that the process of structured reasoning is transferable across domains.

Conclusion

True long-context understanding requires more than just a large window; it requires the ability to reason. By supervising the intermediate steps of analysis through PAI and LongFinanceQA, we demonstrate that even smaller models can outperform massive counterparts by adopting structured, human-like reasoning trajectories.

Summary updated

Abstract

Recent advances in Large Language Models (LLMs) have enabled them to process increasingly longer sequences, ranging from 2K to 2M tokens and even beyond.However, simply extending the input sequence length does not necessarily lead to effective long-context understanding.In this study, we integrate Chainof-Thought (CoT) reasoning into LLMs in a supervised manner to facilitate effective longcontext understanding.To achieve this, we introduce LongFinanceQA, a synthetic dataset in the financial domain designed to improve longcontext reasoning.Unlike existing long-context synthetic data, LongFinanceQA includes intermediate CoT reasoning before the final conclusion, which encourages LLMs to perform explicit reasoning, improving accuracy and interpretability in long-context understanding.To generate synthetic CoT reasoning, we propose Property-based Agentic Inference (PAI), an agentic framework that simulates humanlike reasoning steps, including property extraction, retrieval, and summarization.We evaluate PAI's reasoning capabilities by assessing GPT-4o-mini w/ PAI on the Loong benchmark, outperforming standard GPT-4o-mini by 20.0%.Furthermore, we fine-tune LLaMA-3.1-8B-Instruct on LongFinanceQA, achieving a 28.0% gain on Loong's financial subset.
BibTeX
@misc{lin2025facilitating,
  author = {Jingyang Lin and Andy Kin On Wong and Xia Tian and Shenghua He and Hui Wei and Mei Han and Jiebo Luo},
  title = {{Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.615},
}
Figure