Introduction
Expanding the context window of Large Language Models (LLMs) is critical for processing long documents and complex dialogues. However, current approaches face significant hurdles: data scarcity, high synthesis costs (often requiring expensive human labeling or GPT-4 level models), and the "short-context tax"—a performance degradation on standard tasks after long-context fine-tuning.
Flora addresses these challenges by introducing an "effortless" data-construction strategy. Instead of generating new long-form content from scratch, Flora repurposes high-quality short-context Supervised Fine-Tuning (SFT) data through structured concatenation, allowing for theoretically infinite context scaling without human or LLM intervention.
Methods: The Stacking Strategy
The core of Flora is a curriculum of four specific long-context strategies designed to transform short instructions into meaningful long-context training signals. These strategies move from simple pattern matching to complex retrieval and summarization:
- Fewshot QA (FQA): Uses multiple examples to teach pattern-based answering.
- Answer Before or After (ABA): Tests positional awareness by requiring the model to find questions relative to a "key" anchor.
- Answer No Answer (ANA): Trains the model to identify missing information within a large context.
- Answer to ID (AID): A complex retrieval task matching answers to specific question IDs within a massive list.
Flora also employs an exponential decay function for token length distribution, ensuring the model remains proficient at both short and extremely long scales. By categorizing data into domains like Math and Code before stacking, Flora preserves specialized knowledge during the training process.
Key Results
Flora was evaluated using Llama-3-8B and QwQ-32B models. The results demonstrate that Flora-enhanced models achieve state-of-the-art (SOTA) performance on benchmarks like LongBench v2, often outperforming models with much larger native context windows. For instance, Flora-80k achieved a 33.2% score on LongBench v2, surpassing Llama-3.1-8B-Instruct-128k.
As shown in the Needle-in-a-Haystack results, Flora models maintain nearly 100% retrieval accuracy across their entire context window (up to 128k). Crucially, the "short-context tax" is significantly mitigated; while many long-context models see an 8%+ drop in standard task performance, Flora-trained models typically see only a 3-4% decrease, maintaining high scores on the Open LLM Leaderboard.
Conclusion
Flora provides a cost-effective, human-free, and LLM-free pathway to extending LLM context windows. By leveraging existing high-quality SFT data through clever concatenation, it achieves superior long-context reasoning without sacrificing short-context sharpness. Future work will look toward extending this strategy to multimodal domains and even larger model scales.
Data-construction code is available at: https://github.com/txchen-USTC/Flora
Summary updated