LongWanjuan: Towards Systematic Measurement for Long Text Quality

Kai Lv, Xiaoran Liu, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, Dahua Lin

Abstract

As Large Language Models (LLMs) transition toward handling massive context windows, the quality of long-form training data has become a critical bottleneck. We present LongWanjuan, a systematic framework and a 160B-token bilingual dataset designed to evaluate and optimize long-text data. By grounding data quality in the linguistic dimensions of coherence, cohesion, and complexity, we develop a data-mixing strategy that significantly enhances LLM performance on long-context tasks. Our results demonstrate that models trained with the LongWanjuan recipe achieve state-of-the-art performance among 7B-parameter models on the LongBench benchmark.

Introduction

The evolution of LLMs to handle "infinite" contexts—enabling the processing of entire books or complex codebases—has shifted the focus from architectural innovation to data engineering. Traditional data-cleaning methods often rely on simple heuristics like deduplication or perplexity filtering. However, these "blunt force" approaches fail to capture the structural and logical requirements of long-form content. A high-quality long document is characterized by internal consistency and structural flow, rather than just a sequence of high-probability sentences. To address this, we propose a linguistically grounded approach to distinguish between structured, holistic works and "chaotic" or disjointed text.

Methodology: A Linguistic Framework for Data Quality

We define long-text quality across three fundamental linguistic dimensions:

  1. Coherence: This measures whether a text maintains a consistent theme. We utilize a "contextual gain" approach, where a coherent text is defined by the model's increased ability to predict tokens when provided with a long preceding context compared to a short one.
  2. Cohesion: This refers to the grammatical and lexical linking within a text. We measure this through the density of connectives (e.g., "however," "therefore") and pronouns, alongside a Distributed Marker Representation (DMR) model that evaluates the logical relationship between adjacent sentences.
  3. Complexity: This captures the sophistication of the language through metrics such as the Type-Token Ratio (TTR)—indicating lexical variety—and average paragraph length.
The three linguistic dimensions used to evaluate long-text quality: Coherence, Cohesion, and Complexity.

By applying these metrics, we categorize data into three distinct types: Holistic (complete, structured works), Aggregated (semi-structured content like lists or tables), and Chaotic (nonsensical or garbled data).

The LongWanjuan Dataset and Pipeline

The LongWanjuan dataset is a 160-billion-token bilingual (English and Chinese) corpus derived from SlimPajama and Wanjuan. The construction pipeline involves a length filter followed by the application of our linguistic metrics to categorize and filter the data.

The LongWanjuan data processing pipeline, showing the transition from raw bilingual texts to categorized datasets based on linguistic metrics.

The final dataset composition is strategically balanced: 85.7% holistic, 13.6% aggregated, and a minimal 0.7% chaotic (primarily used for validation). Our "data mixture recipe" involves filtering out chaotic data and upsampling aggregated content to ensure the model learns both narrative flow and the structural navigation required for "needle-in-a-haystack" retrieval tasks.

Key Results and Insights

Fine-tuning InternLM2-7B and LLaMA2-7B using the LongWanjuan recipe yielded significant performance gains. Specifically, the InternLM2-7B model showed a 13.07% improvement over the baseline on the LongBench benchmark. Our 7B models outperformed several larger or specialized competitors, such as LongChat-v1.5-7B and Yi-6B-200K.

Key insights from our study include:

  • Linguistic Theory as Engineering: Abstract concepts like Halliday & Hasan’s theory of cohesion can be effectively translated into scalable data engineering tools.
  • The "Chaotic" Threshold: Removing chaotic text—often identified by anomalous complexity scores—is vital for training stability.
  • Human Alignment: Our automated metrics demonstrate a high correlation (0.91 for English) with human judgment, validating their effectiveness in capturing perceived text quality.

Conclusion and Future Work

LongWanjuan provides a systematic path forward for long-context LLM training. While currently focused on English and Chinese, future work will aim to scale this framework to other languages, which will require mapping specific linguistic markers like connectives and pronouns. Additionally, while our current data-mixing ratios are highly effective, further research is needed to determine the mathematically optimal balance between holistic and aggregated content.

Summary updated

BibTeX
@inproceedings{lv2024longwanjuan,
  author = {Kai Lv and Xiaoran Liu and Qipeng Guo and Hang Yan and Conghui He and Xipeng Qiu and Dahua Lin},
  title = {{LongWanjuan: Towards Systematic Measurement for Long Text Quality}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.327},
}
Figure