Is This Collection Worth My LLM's Time? Automatically Measuring Information Potential in Text Corpora

Tristan Karch, Luca Engel, Philippe Schwaller, Frédéric KaplanarXiv 2025 · DOI · arXiv

Abstract

As Large Language Models (LLMs) reach a plateau in architectural innovation, the focus of AI progress has shifted toward high-quality, novel data. However, evaluating the utility of a dataset—such as specialized academic archives or historical records—traditionally requires expensive fine-tuning or extensive benchmarking. This paper introduces an automated, training-free pipeline to calculate the "Information Potential" (IP) of a text collection. By generating high-quality multiple-choice questions (MCQs) and comparing model performance with and without text access, the system identifies datasets that offer genuinely new knowledge versus those that are redundant or already known to the model.

Introduction

The "Data-Centric AI" movement posits that future model improvements depend on feeding LLMs high-quality, novel information. Yet, the process of acquiring, digitizing, and processing specialized data is incredibly resource-intensive. Researchers currently lack "prospective" tools to determine if a specific corpus is worth the cost of integration before committing resources. This study addresses this gap by proposing a lightweight metric to measure the "value add" of a dataset relative to a model's existing internal knowledge.

The Information Potential (IP) Pipeline

The proposed pipeline uses an LLM to "interrogate" itself about a new text collection through a four-step process:

  1. MCQ Generation: The corpus is divided into 2,000-word chunks, from which an LLM generates 4-option MCQs based on specific facts.
  2. Two-Stage Quality Filtering: Questions are filtered using ROUGE-L and Jaccard similarity to ensure grounding, while sentence embeddings (NV-Embed-v2) ensure that distractors are semantically plausible, preventing the model from winning by simple elimination.
  3. Position-Debiased Evaluation: To counter the inherent biases of LLMs, each question is asked four times, rotating the correct answer through every position (A, B, C, D). A model only receives a point if it answers correctly in all four rotations.
  4. IP Scoring: The final Information Potential score is the delta between the model's performance "Directly" (internal weights only) and "With Context" (text provided).

Addressing Model Bias

A critical finding of the study was the significant position bias in modern LLMs. Models like GPT-4o exhibit a strong tendency to select options B and C, regardless of the content. Without the 4x rotation consistency check, IP scores would be heavily inflated by these "lucky guesses."

Figure 5: Distribution of answer choices showing a strong bias toward options B and C in baseline models.

Experimental Results

The pipeline was validated across five distinct collections, ranging from specialized EPFL PhD manuscripts to common Wikipedia subsets and synthetic baselines.

Rare Data vs. Redundant Knowledge

The results clearly distinguish between "rare" and "common" knowledge. The EPFL PhD theses and Venetian historical records yielded the highest IP scores (0.21–0.26), confirming they contain specialized knowledge not well-represented in standard pre-training sets. Conversely, Wikipedia subsets showed significantly lower IP scores (0.11–0.18), suggesting the models already possessed much of this information. Synthetic data, representing "zero novelty," provided the lowest baseline.

Figure 3: Fraction of correct responses and calculated IP scores across different models and datasets.

Key Insights

  • Performance Gap as Novelty Proxy: If a model succeeds with context but fails without it, the text contains information the model has not yet mastered.
  • Model Scaling Effects: Larger models (e.g., GPT-4o) generally show lower IP scores than smaller ones (e.g., Llama 3 70B). This is because larger models have a higher baseline of internal knowledge, making it more difficult for any new text to provide truly novel facts.
  • Dataset Agnosticism: Because the pipeline generates its own evaluation criteria, it can be applied to any niche domain—from legal archives to molecular biology—without requiring human experts to write tests.

Conclusion and Future Work

This automated pipeline provides a cost-effective way to prioritize data for LLM training. While highly effective, future work could improve efficiency by integrating quality checks directly into the generation prompts to reduce API costs. Additionally, while the tool identifies if a collection is valuable, further research is needed to determine the optimal integration strategy, such as whether to use the data for RAG or full fine-tuning.

Summary updated

BibTeX
@misc{karch2025is,
  author = {Tristan Karch and Luca Engel and Philippe Schwaller and Frédéric Kaplan},
  title = {{Is This Collection Worth My LLM's Time? Automatically Measuring Information Potential in Text Corpora}},
  year = {2025},
  eprint = {2502.13691},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.13691},
}
Figure