OkraLong: A Flexible Retrieval-Augmented Framework for Long-Text Question Answering

Yulong Hui, Yihao Liu, Yao Lu, Huanchen ZhangEMNLP 2025 · OpenReview

TL;DR

This paper introduces OkraLong, a novel framework that dynamically orchestrates retrieval and generation processes to answer questions from long documents more accurately and cost-effectively. By intelligently analyzing the question and context, OkraLong adapts its strategy, outperforming existing methods in both performance and efficiency.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful tools for understanding and processing information, but they struggle with long documents. Applications like analyzing financial reports or enterprise documents require LLMs to handle vast amounts of text, which leads to significant challenges. Traditional approaches like "long-context" processing (feeding the entire document to the LLM) are prohibitively expensive due to the high computational cost of processing lengthy inputs. Retrieval-Augmented Generation (RAG) offers a more cost-effective solution by retrieving only relevant text snippets. However, basic RAG can miss crucial details, leading to inaccurate answers, especially for complex, multi-step questions. More advanced RAG methods, like context compression or dynamic retrieval, often sacrifice critical information or incur high iterative costs, failing to strike a balance between accuracy and efficiency. OkraLong aims to bridge this gap by providing a flexible and intelligent framework that optimizes the entire question-answering pipeline for long texts.

Methods

OkraLong is built around three synergistic components:

  • Analyzer: This component acts as the "brain" of the system. It's a fine-tuned lightweight model that analyzes the incoming question and initial retrieved context to characterize the task state. It identifies the question type (e.g., arithmetic, extractive, multi-source), the information pattern (exact keywords vs. semantic meaning), and whether the provided context is sufficient. This analysis is crucial for guiding the subsequent steps.
  • Organizer: Based on the analyzer's insights, the organizer dynamically schedules the workflow. It constructs task-specific processing pipelines, adjusts retrieval granularity, and decides how to aggregate evidence. For instance, a multi-source question might trigger a pipeline that splits the question into sub-questions for parallel retrieval, while a factoid question might focus on precise context slicing. This adaptive orchestration is key to OkraLong's flexibility.
  • Executor: This component is the workhorse that carries out the planned workflow. It comprises a suite of modular operators, including basic text processing, an "assembled retriever" that can combine multiple retrieval strategies (sparse and dense), a "context processor" for merging and extending retrieved text, a "question splitter" for multi-source questions, and a "step-wise reasoner" for complex, multi-hop queries.

This modular design allows OkraLong to adapt its processing strategy to the specific demands of each question, optimizing for both accuracy and cost.

Datasets

OkraLong was evaluated on six diverse long-text question-answering datasets:

  • FINQA: Financial numerical reasoning from earnings reports.
  • TAT-DQA: Financial reasoning from annual reports.
  • Qasper: Reading comprehension from NLP research papers.
  • Multi-fieldQA: Question-answering pairs from diverse fields like legal documents and government reports.
  • HotpotQA: Two-hop questions based on Wikipedia paragraphs.
  • 2Wiki-MultihopQA: Multi-hop questions from Wikipedia content.

These datasets were derived from the UDA and LongBench collections, which provide unsegmented full documents.

Key Insights and Ideas

  • Dynamic Orchestration is Key: The core innovation is moving beyond static or coarse-grained adaptive strategies to a fine-grained, dynamic orchestration of the entire QA pipeline. This allows for a much more tailored approach to different question types and complexities.
  • Analyzer as the Smart Guide: The analyzer's ability to accurately characterize task states (question type, information pattern, evidence sufficiency) is critical. This fine-grained analysis directly informs the organizer's decisions, leading to more effective workflows.
  • Modular Executor for Flexibility: The executor's suite of specialized operators (question splitter, step-wise reasoner, assembled retriever) provides the building blocks for constructing diverse and efficient processing pipelines.
  • Balancing Accuracy and Cost: OkraLong demonstrates that it's possible to achieve both high accuracy and significant cost savings, addressing a major limitation of previous approaches that often prioritized one over the other.

Key Results

OkraLong achieved impressive results across the evaluated datasets:

  • Accuracy Improvement: OkraLong enhanced answer accuracy by 5.7% to 41.2% compared to prior advancements.
  • Cost Savings: It achieved cost savings of 1.3x to 4.7x.
  • Precise Mode: A "Precise Mode" was introduced, which falls back to full-context processing for initially unanswerable questions, achieving comparable accuracy to long-context processing while maintaining a 4.4x cost advantage.
  • Latency: While OkraLong has a slightly higher overhead than basic RAG due to its analysis and orchestration steps, this is considered justifiable given the significant improvements in accuracy and cost-efficiency.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Analyzer Training Data: The current analyzer relies on supervised fine-tuning, which requires annotated training datasets. Future work could explore semi-supervised or weakly supervised methods to reduce annotation dependence.
  • Text-Centric Focus: OkraLong currently focuses on text-based documents. Future research could explore efficient strategies for querying and processing multi-modal content (e.g., documents with images or tables).

The modular and plug-in design of OkraLong is highlighted as a strength, enabling seamless integration of new operations and heuristics, suggesting its adaptability for future advancements in LLM-based question answering.

Summary updated

Abstract

Large Language Models (LLMs) encounter challenges in efficiently answering long-text questions, as seen in applications like enterprise document analysis and financial report comprehension. While conventional solutions employ long-context processing or Retrieval-Augmented Generation (RAG), they suffer from prohibitive input expenses or incomplete information. Recent advancements adopt context compression and dynamic retrieval loops, but still sacrifice critical details or incur iterative costs. To address these limitations, we propose OkraLong, a novel framework that flexibly optimizes the entire processing workflow. Unlike prior static or coarse-grained adaptive strategies, OkraLong adopts fine-grained orchestration through three synergistic components: analyzer, organizer and executor. The analyzer characterizes the task states, which guide the organizer in dynamically scheduling the workflow. The executor carries out the execution and generates the final answer. Experimental results demonstrate that OkraLong not only enhances answer accuracy by 5.7%-41.2%, but also achieves cost savings of 1.3x-4.7x.
BibTeX
@inproceedings{hui2025okralong,
  author = {Yulong Hui and Yihao Liu and Yao Lu and Huanchen Zhang},
  title = {{OkraLong: A Flexible Retrieval-Augmented Framework for Long-Text Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=0YcEITNv9Q},
}
Figure