LongWanjuan: Towards Systematic Measurement for Long Text Quality

Xiaoran Liu, Kai Lv, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, Dahua Lin

As Large Language Models (LLMs) expand their context windows to handle entire libraries or code repositories, the "garbage in, garbage out" principle becomes increasingly critical. While heuristics for cleaning general web data are well-established, a systematic method for evaluating the quality of massive, 100,000-token documents has been lacking. Researchers from Fudan University and the Shanghai AI Laboratory address this gap with LongWanjuan, a framework and dataset designed to move beyond simple length-based filtering toward a linguistic-first approach for long-text quality.

The three linguistic pillars of long-text quality: Coherence, Cohesion, and Complexity.

The Three Pillars of Long-Text Quality

The researchers grounded their approach in three fundamental linguistic dimensions to quantify "deep" quality traits that simple perplexity scores often miss:

  1. Coherence: This measures whether the text makes sense as a unified whole. It is quantified by "predictive gain"—the improvement in a model's ability to predict the next token when provided with a long context compared to a short one.
  2. Cohesion: This evaluates how well individual sentences are linked. Metrics include the density of connectives (e.g., "however," "therefore") and pronouns, alongside a specialized "Distributed Marker Representation" (DMR) model to verify semantic relationships between adjacent sentences.
  3. Complexity: This assesses the sophistication of the language through the Type-Token Ratio (lexical richness) and average paragraph length.

The LongWanjuan Framework

The LongWanjuan framework applies these metrics to categorize long texts into three distinct types: Holistic (complete, high-quality works), Aggregated (collections of related short texts), and Chaotic (garbled or nonsensical content).

The LongWanjuan pipeline: from raw bilingual data to categorized long-text datasets using systematic metrics.

By filtering and labeling data from SlimPajama and Wanjuan, the team created the LongWanjuan dataset, a bilingual (English/Chinese) corpus containing over 160 billion tokens. This systematic approach allows for strategic data-mixing recipes that prioritize structural integrity over mere volume.

Key Insights and Results

The study's findings highlight that not all long text is created equal. A key insight is the detrimental effect of "Chaotic" text; removing just the bottom 0.7% of such data led to noticeable performance gains. Furthermore, while "Holistic" texts like books are high-quality, the best results were achieved by a balanced "recipe" that upsampled "Aggregated" texts, suggesting that variety in informational linking is crucial for model generalization.

Fine-tuning models like LLaMA3-8B and InternLM2-7B using the LongWanjuan recipe yielded significant improvements:

  • Performance Boost: InternLM2-7B achieved a 13.07% improvement over its baseline on long-context benchmarks.
  • State-of-the-Art Scores: The model reached a total score of 50.26 on LongBench, outperforming specialized models like ChatGLM3-6B-32K.
  • Task-Specific Success: The refined data mixture particularly enhanced performance in multi-document QA and summarization, where tracking information across long distances is vital.

Conclusion

LongWanjuan provides a much-needed roadmap for building the next generation of long-context AI. By moving from shallow heuristics to deep linguistic measurements, researchers can ensure that the massive context windows of modern LLMs are filled with coherent, high-quality information rather than chaotic noise.

Summary updated

Abstract

The quality of training data is crucial for enhancing the long-text capabilities of foundation models. Despite existing efforts to refine data quality through heuristic rules and evaluations based on data diversity and difficulty, there’s a lack of systematic approaches specifically tailored for assessing long texts. Addressing this gap, our work systematically measures the quality of long texts by evaluating three fundamental linguistic dimensions: coherence, cohesion, and complexity. Drawing inspiration from the aforementioned three dimensions, we introduce a suite of metrics designed to evaluate the quality of long texts, encompassing both statistical and pre-trained language model-based ones. Leveraging these metrics, we present LongWanjuan, a bilingual dataset specifically tailored to enhance the training of language models for long-text tasks with over 160B tokens. In LongWanjuan, we categorize long texts into holistic, aggregated, and chaotic types, enabling a detailed analysis of long-text quality. Furthermore, we devise a data mixture recipe that strategically balances different types of long texts within LongWanjuan, leading to significant improvements in model performance on long-text tasks.
BibTeX
@inproceedings{liu2024longwanjuan,
  author = {Xiaoran Liu and Kai Lv and Qipeng Guo and Hang Yan and Conghui He and Xipeng Qiu and Dahua Lin},
  title = {{LongWanjuan: Towards Systematic Measurement for Long Text Quality}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.327},
}
Figure