A Two-Sample Test of Text Generation Similarity

Jingbin Xu, Chen Qian, Meimei Liu, Feng GuoarXiv 2025 · DOI · arXiv

Introduction

Comparing sets of documents is a fundamental challenge in Natural Language Processing (NLP). Whether distinguishing human-written text from AI-generated content or comparing medical records across institutions, researchers need robust methods to determine if two collections of text originate from the same underlying probabilistic process. Traditional statistical methods often fail here because text is high-dimensional, sparse, and unstructured.

Existing approaches typically suffer from two main drawbacks. First, "bag-of-words" models often ignore semantics, focusing on word frequencies while neglecting the context and order that define meaning. Second, semantically rich metrics like Maximum Mean Discrepancy (MMD) are computationally expensive ($O(N^2V)$) and often rely on slow permutation tests to establish statistical significance.

Probabilistic mapping of documents in a sample space.

This paper introduces a rigorous statistical framework for a "two-sample test" of text similarity. By viewing documents as paths through a probabilistic sample space (as shown in Figure 1), the authors propose using neural network-based language models to estimate document-level entropy. This approach provides a way to determine generation similarity that is both semantically aware and statistically grounded.

Methodology

The core innovation lies in using Entropy as the primary metric for comparison. The hypothesis is that if two groups of documents exhibit significantly different entropy levels when processed by the same language model, they likely stem from different generation processes.

Neural Entropy Estimation

The authors employ an autoregressive Neural Network Language Model (NNLM) to estimate the conditional probability of each word given its preceding context. This architecture, illustrated in Figure 2, allows for the calculation of a comprehensive document-level entropy score.

Architecture of the Neural Network Language Model (NNLM) for entropy estimation.

Estimation-and-Inference Framework

To ensure statistical validity and prevent overfitting, the authors implement a data-splitting strategy:

  1. Estimation Set: A portion of the data is used to train the NNLM.
  2. Inference Set: The remaining data is used to calculate the actual test statistic.

A key theoretical contribution is the proof that the resulting test statistic follows an asymptotic standard normal distribution. This allows researchers to use a simple Z-test to derive p-values, bypassing the need for computationally intensive permutation tests. To maximize statistical power despite data splitting, the process is repeated $M$ times, and results are aggregated using p-value combination methods like the Cauchy combination.

Experimental Evaluation

The proposed method was validated across three distinct environments:

  • Synthetic Simulations: Controlled "toy" data with specific word dependencies (Autoregressive and Compound Symmetry) were used to verify the model's sensitivity to subtle rule changes.
  • AG News Benchmark: The test successfully distinguished between news categories (World, Sports, Business, Sci/Tech) with high accuracy.
  • U.S. National Crash Reports: In a real-world application using NHTSA data (2018–2022), the test compared narratives of crashes involving vehicles with and without Advanced Driver Assistance Systems (ADAS).

Key Results

The proposed test demonstrated superior statistical power. In the AG News benchmark ($N=100$), it achieved an average power of 0.8, significantly outperforming MMD and LDA-based tests, which hovered around 0.25. Crucially, the test maintained a controlled Type I error rate (0.05), ensuring reliability.

In the crash report study, the test revealed that "Intervention" systems (e.g., automatic emergency braking) significantly altered crash narratives in high-risk scenarios, whereas "Warning" systems (e.g., blind-spot alerts) did not produce a statistically significant change in the resulting narratives.

Conclusion

By bridging the gap between deep learning and frequentist statistics, this framework offers a scalable, semantically aware tool for text analysis. While the current approach relies on autoregressive assumptions and requires training overhead for multiple data splits, it provides a powerful alternative to traditional metrics. Future work will explore data reuse strategies to further enhance statistical power and investigate the applicability of the framework to non-autoregressive neural architectures.

Summary updated

BibTeX
@misc{xu2025two,
  author = {Jingbin Xu and Chen Qian and Meimei Liu and Feng Guo},
  title = {{A Two-Sample Test of Text Generation Similarity}},
  year = {2025},
  eprint = {2505.05269},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.05269},
}
Figure