Abstract
As Large Language Models (LLMs) transition toward handling massive context windows, the quality of long-form training data has become a critical bottleneck. We present LongWanjuan, a systematic framework and a 160B-token bilingual dataset designed to evaluate and optimize long-text data. By grounding data quality in the linguistic dimensions of coherence, cohesion, and complexity, we develop a data-mixing strategy that significantly enhances LLM performance on long-context tasks. Our results demonstrate that models trained with the LongWanjuan recipe achieve state-of-the-art performance among 7B-parameter models on the LongBench benchmark.
Introduction
The evolution of LLMs to handle "infinite" contexts—enabling the processing of entire books or complex codebases—has shifted the focus from architectural innovation to data engineering. Traditional data-cleaning methods often rely on simple heuristics like deduplication or perplexity filtering. However, these "blunt force" approaches fail to capture the structural and logical requirements of long-form content. A high-quality long document is characterized by internal consistency and structural flow, rather than just a sequence of high-probability sentences. To address this, we propose a linguistically grounded approach to distinguish between structured, holistic works and "chaotic" or disjointed text.
Methodology: A Linguistic Framework for Data Quality
We define long-text quality across three fundamental linguistic dimensions:
- Coherence: This measures whether a text maintains a consistent theme. We utilize a "contextual gain" approach, where a coherent text is defined by the model's increased ability to predict tokens when provided with a long preceding context compared to a short one.
- Cohesion: This refers to the grammatical and lexical linking within a text. We measure this through the density of connectives (e.g., "however," "therefore") and pronouns, alongside a Distributed Marker Representation (DMR) model that evaluates the logical relationship between adjacent sentences.
- Complexity: This captures the sophistication of the language through metrics such as the Type-Token Ratio (TTR)—indicating lexical variety—and average paragraph length.
By applying these metrics, we categorize data into three distinct types: Holistic (complete, structured works), Aggregated (semi-structured content like lists or tables), and Chaotic (nonsensical or garbled data).
The LongWanjuan Dataset and Pipeline
The LongWanjuan dataset is a 160-billion-token bilingual (English and Chinese) corpus derived from SlimPajama and Wanjuan. The construction pipeline involves a length filter followed by the application of our linguistic metrics to categorize and filter the data.
The final dataset composition is strategically balanced: 85.7% holistic, 13.6% aggregated, and a minimal 0.7% chaotic (primarily used for validation). Our "data mixture recipe" involves filtering out chaotic data and upsampling aggregated content to ensure the model learns both narrative flow and the structural navigation required for "needle-in-a-haystack" retrieval tasks.
Key Results and Insights
Fine-tuning InternLM2-7B and LLaMA2-7B using the LongWanjuan recipe yielded significant performance gains. Specifically, the InternLM2-7B model showed a 13.07% improvement over the baseline on the LongBench benchmark. Our 7B models outperformed several larger or specialized competitors, such as LongChat-v1.5-7B and Yi-6B-200K.
Key insights from our study include:
- Linguistic Theory as Engineering: Abstract concepts like Halliday & Hasan’s theory of cohesion can be effectively translated into scalable data engineering tools.
- The "Chaotic" Threshold: Removing chaotic text—often identified by anomalous complexity scores—is vital for training stability.
- Human Alignment: Our automated metrics demonstrate a high correlation (0.91 for English) with human judgment, validating their effectiveness in capturing perceived text quality.
Conclusion and Future Work
LongWanjuan provides a systematic path forward for long-context LLM training. While currently focused on English and Chinese, future work will aim to scale this framework to other languages, which will require mapping specific linguistic markers like connectives and pronouns. Additionally, while our current data-mixing ratios are highly effective, further research is needed to determine the mathematically optimal balance between holistic and aggregated content.
Summary updated