Flora: Effortless Context Construction to Arbitrary Length and Scale

Tianxiang Chen, Zhentao Tan, Xiaofan Bo, Yue Wu, Tao Gong, Qi Chu, Jieping Ye, Nenghai YuarXiv 2025 · DOI · arXiv

Introduction

Expanding the context window of Large Language Models (LLMs) is critical for processing long documents and complex dialogues. However, current approaches face significant hurdles: data scarcity, high synthesis costs (often requiring expensive human labeling or GPT-4 level models), and the "short-context tax"—a performance degradation on standard tasks after long-context fine-tuning.

Flora addresses these challenges by introducing an "effortless" data-construction strategy. Instead of generating new long-form content from scratch, Flora repurposes high-quality short-context Supervised Fine-Tuning (SFT) data through structured concatenation, allowing for theoretically infinite context scaling without human or LLM intervention.

Methods: The Stacking Strategy

The core of Flora is a curriculum of four specific long-context strategies designed to transform short instructions into meaningful long-context training signals. These strategies move from simple pattern matching to complex retrieval and summarization:

  1. Fewshot QA (FQA): Uses multiple examples to teach pattern-based answering.
  2. Answer Before or After (ABA): Tests positional awareness by requiring the model to find questions relative to a "key" anchor.
  3. Answer No Answer (ANA): Trains the model to identify missing information within a large context.
  4. Answer to ID (AID): A complex retrieval task matching answers to specific question IDs within a massive list.
Flora's data construction strategies: FQA, ABA, ANA, and AID.

Flora also employs an exponential decay function for token length distribution, ensuring the model remains proficient at both short and extremely long scales. By categorizing data into domains like Math and Code before stacking, Flora preserves specialized knowledge during the training process.

Key Results

Flora was evaluated using Llama-3-8B and QwQ-32B models. The results demonstrate that Flora-enhanced models achieve state-of-the-art (SOTA) performance on benchmarks like LongBench v2, often outperforming models with much larger native context windows. For instance, Flora-80k achieved a 33.2% score on LongBench v2, surpassing Llama-3.1-8B-Instruct-128k.

Needle-in-a-Haystack performance and short-context task results.

As shown in the Needle-in-a-Haystack results, Flora models maintain nearly 100% retrieval accuracy across their entire context window (up to 128k). Crucially, the "short-context tax" is significantly mitigated; while many long-context models see an 8%+ drop in standard task performance, Flora-trained models typically see only a 3-4% decrease, maintaining high scores on the Open LLM Leaderboard.

Conclusion

Flora provides a cost-effective, human-free, and LLM-free pathway to extending LLM context windows. By leveraging existing high-quality SFT data through clever concatenation, it achieves superior long-context reasoning without sacrificing short-context sharpness. Future work will look toward extending this strategy to multimodal domains and even larger model scales.

Data-construction code is available at: https://github.com/txchen-USTC/Flora

Summary updated

Abstract

Effectively handling long contexts is challenging for Large Language Models (LLMs) due to the rarity of long texts, high computational demands, and substantial forgetting of short-context abilities. Recent approaches have attempted to construct long contexts for instruction tuning, but these methods often require LLMs or human interventions, which are both costly and limited in length and diversity. Also, the drop in short-context performances of present long-context LLMs remains significant. In this paper, we introduce Flora, an effortless (human/LLM-free) long-context construction strategy. Flora can markedly enhance the long-context performance of LLMs by arbitrarily assembling short instructions based on categories and instructing LLMs to generate responses based on long-context meta-instructions. This enables Flora to produce contexts of arbitrary length and scale with rich diversity, while only slightly compromising short-context performance. Experiments on Llama3-8B-Instruct and QwQ-32B show that LLMs enhanced by Flora excel in three long-context benchmarks while maintaining strong performances in short-context tasks. Our data-construction code is available at \href{https://github.com/txchen-USTC/Flora}{https://github.com/txchen-USTC/Flora}.
BibTeX
@misc{chen2025flora,
  author = {Tianxiang Chen and Zhentao Tan and Xiaofan Bo and Yue Wu and Tao Gong and Qi Chu and Jieping Ye and Nenghai Yu},
  title = {{Flora: Effortless Context Construction to Arbitrary Length and Scale}},
  year = {2025},
  eprint = {2507.19786},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.19786},
}
Figure