Abstract
Recent advances in Large Language Models (LLMs) have enabled processing of sequences up to 2M tokens. However, extended context windows do not inherently guarantee effective understanding. This study introduces Property-based Agentic Inference (PAI) and LongFinanceQA, a synthetic dataset designed to integrate supervised Chain-of-Thought (CoT) reasoning into long-context models. By training models to perform explicit intermediate reasoning steps—extraction, retrieval, and synthesis—we bridge the gap between simple information retrieval and complex analysis. Our approach, evaluated on the Loong benchmark, demonstrates that GPT-4o-mini equipped with PAI outperforms its base version by 20.3%, surpassing even state-of-the-art models with much larger context windows.
The Reasoning Gap in Long Contexts
The current "long-context" paradigm often focuses on engineering larger windows rather than improving the model's ability to reason over the data within them. Standard training pairs massive documents with concise answers, forcing models to leap from input to conclusion without intermediate logic. This leads to a "performance ceiling" where models fail on tasks requiring multi-step synthesis across disparate document sections.
Methodology: Property-based Agentic Inference (PAI)
To address the lack of high-quality reasoning data, we developed the PAI framework. PAI mimics human financial analysis by decomposing complex queries into verifiable "properties" (Metric + Subject).
The framework operates in three distinct stages:
- Property Extraction: The model identifies specific evidence properties needed to answer a query (e.g., "Revenue" for "Company A in 2019").
- Property-based Retrieval: These properties are used to retrieve relevant 1024-token chunks from the long context.
- Summarization: The retrieved facts are synthesized into a final conclusion, preceded by an explicit reasoning trace.
LongFinanceQA and Distillation
Using PAI, we generated LongFinanceQA, a dataset of ~46k QA pairs from 6,911 financial reports. We then fine-tuned a LLaMA-3.1-8B model on these reasoning traces. This "distilled" model, LongPAI, learns to generate the reasoning and answer in a single pass, achieving 30x the token efficiency of the multi-step agentic baseline.
Key Results and Insights
The PAI-enhanced GPT-4o-mini achieved an overall score of 69.58% on the Loong benchmark, a significant jump from the base model's 49.25%.
- Efficiency through Distillation: LongPAI (8B) achieved a 28.0% gain on financial subsets compared to its base model while using less than 3% of the tokens required by the full agentic pipeline.
- Scalability of Reasoning: The performance advantage of CoT-trained models became more pronounced as context length increased (e.g., from 50k to 250k tokens), suggesting that explicit reasoning is essential for navigating larger "haystacks."
- Generalization: Despite being trained on financial data, the model showed improved performance on general-purpose benchmarks like ∞Bench, indicating that the process of structured reasoning is transferable across domains.
Conclusion
True long-context understanding requires more than just a large window; it requires the ability to reason. By supervising the intermediate steps of analysis through PAI and LongFinanceQA, we demonstrate that even smaller models can outperform massive counterparts by adopting structured, human-like reasoning trajectories.
Summary updated