As Large Language Models (LLMs) expand their context windows to handle entire libraries or code repositories, the "garbage in, garbage out" principle becomes increasingly critical. While heuristics for cleaning general web data are well-established, a systematic method for evaluating the quality of massive, 100,000-token documents has been lacking. Researchers from Fudan University and the Shanghai AI Laboratory address this gap with LongWanjuan, a framework and dataset designed to move beyond simple length-based filtering toward a linguistic-first approach for long-text quality.
The Three Pillars of Long-Text Quality
The researchers grounded their approach in three fundamental linguistic dimensions to quantify "deep" quality traits that simple perplexity scores often miss:
- Coherence: This measures whether the text makes sense as a unified whole. It is quantified by "predictive gain"—the improvement in a model's ability to predict the next token when provided with a long context compared to a short one.
- Cohesion: This evaluates how well individual sentences are linked. Metrics include the density of connectives (e.g., "however," "therefore") and pronouns, alongside a specialized "Distributed Marker Representation" (DMR) model to verify semantic relationships between adjacent sentences.
- Complexity: This assesses the sophistication of the language through the Type-Token Ratio (lexical richness) and average paragraph length.
The LongWanjuan Framework
The LongWanjuan framework applies these metrics to categorize long texts into three distinct types: Holistic (complete, high-quality works), Aggregated (collections of related short texts), and Chaotic (garbled or nonsensical content).
By filtering and labeling data from SlimPajama and Wanjuan, the team created the LongWanjuan dataset, a bilingual (English/Chinese) corpus containing over 160 billion tokens. This systematic approach allows for strategic data-mixing recipes that prioritize structural integrity over mere volume.
Key Insights and Results
The study's findings highlight that not all long text is created equal. A key insight is the detrimental effect of "Chaotic" text; removing just the bottom 0.7% of such data led to noticeable performance gains. Furthermore, while "Holistic" texts like books are high-quality, the best results were achieved by a balanced "recipe" that upsampled "Aggregated" texts, suggesting that variety in informational linking is crucial for model generalization.
Fine-tuning models like LLaMA3-8B and InternLM2-7B using the LongWanjuan recipe yielded significant improvements:
- Performance Boost: InternLM2-7B achieved a 13.07% improvement over its baseline on long-context benchmarks.
- State-of-the-Art Scores: The model reached a total score of 50.26 on LongBench, outperforming specialized models like ChatGLM3-6B-32K.
- Task-Specific Success: The refined data mixture particularly enhanced performance in multi-document QA and summarization, where tracking information across long distances is vital.
Conclusion
LongWanjuan provides a much-needed roadmap for building the next generation of long-context AI. By moving from shallow heuristics to deep linguistic measurements, researchers can ensure that the massive context windows of modern LLMs are filled with coherent, high-quality information rather than chaotic noise.
Summary updated