TextDescriptives: A Python package for calculating a large variety of metrics from text

Lasse Hansen, Ludvig Renbo Olsen, Kenneth C. EnevoldsenJ. Open Source Softw 2023 · DOI · DBLP

Introduction

In the rapidly evolving landscape of natural language processing (NLP) and the digital humanities, the ability to quantitatively characterize text corpora is paramount. Researchers frequently require a diverse suite of metrics to describe their data, ranging from fundamental descriptive statistics like word counts to sophisticated linguistic measures such as syntactic complexity, readability, and semantic coherence. However, the process of extracting these features has historically been fragmented. A typical workflow might involve using one library for readability scores, another for part-of-speech (POS) tagging, and custom-written scripts for calculating dependency distances.

This fragmentation introduces two significant problems. First, it is inefficient, requiring researchers to manage multiple dependencies and data formats. Second, and more critically, it threatens scientific reproducibility. Subtle differences in how different libraries implement the same metric—such as how they handle punctuation in readability formulas or how they define sentence boundaries—can lead to inconsistent results across studies. TextDescriptives addresses these issues by providing a unified, modular, and rigorously tested Python package built on top of the industry-standard spaCy framework.

System Architecture and Methods

TextDescriptives is architected as a collection of custom components for the spaCy pipeline (v3.0.0+). This strategic integration allows the package to benefit from spaCy’s optimized tokenization, lemmatization, and parsing capabilities.

Modular Design and API

The library’s modularity is a core strength. Users are not forced to calculate every available metric; instead, they can selectively add components to a spaCy Language object. For instance, a researcher focused solely on data cleaning might only load the "quality" component, while a linguist might load "dependency_distance" and "readability."

The package provides a high-level API designed for ease of use. With simple calls to extract_df or extract_dict, users can transform a collection of processed spaCy documents into structured formats like pandas DataFrames. This design facilitates a smooth transition from raw text processing to statistical analysis and visualization.

Broad Metric Coverage

The package offers a comprehensive suite of metrics categorized into several key domains:

  1. Descriptive Statistics: Basic measures including token counts, sentence counts, and average lengths, providing a baseline for corpus comparison.
  2. Readability Metrics: Standardized scores such as Gunning-Fog, Flesch-Kincaid, and the Automated Readability Index (ARI). These are essential for assessing the accessibility of text in educational and communication research.
  3. Syntactic Complexity: TextDescriptives implements measures like dependency distance, which quantifies the structural complexity of sentences based on the distance between related words in a dependency tree.
  4. Linguistic Proportions: It calculates the relative frequency of various parts of speech (e.g., the ratio of adjectives to nouns), which can serve as a proxy for stylistic variation.
  5. Semantic Coherence: By analyzing the similarity between word embeddings across sentences, the package can estimate the thematic consistency and flow of a document.
  6. Information Theory: Metrics like Shannon entropy are included to measure the unpredictability or information density of the text.
  7. Quality Filtering: Crucially for modern NLP, the library implements quality filters used by teams at Google (Gopher) and other major labs. These filters help identify repetitive, "gibberish," or low-quality text, which is vital for curating high-quality training data for large language models.

Applications and Impact

The utility of TextDescriptives is evidenced by its application in diverse academic fields. In clinical settings, it has been used to analyze the linguistic stability of medical notes over a ten-year period, helping researchers understand how documentation practices evolve. In psychiatry, the package has provided features for models aimed at predicting neuropsychiatric conditions based on patient speech or writing. Furthermore, in the field of education, it has enabled the large-scale analysis of writing development in primary school students, allowing researchers to track how linguistic complexity grows over time.

Discussion and Limitations

Compared to existing libraries like textstat, which focuses primarily on readability, or textacy, which offers a broader but less specialized set of metrics, TextDescriptives provides a more comprehensive and integrated solution. It includes specific measures—such as dependency distance and semantic coherence—that were previously difficult to access within a single, unified package. Furthermore, its focus on "document-level" analysis makes it particularly valuable for researchers who need to characterize entire datasets rather than individual tokens.

To further lower the barrier to entry, the authors provide a web-based application (hosted on Hugging Face Spaces), enabling researchers without extensive programming backgrounds to generate metrics via a simple CSV upload.

Despite its strengths, the package has defined boundaries. It is strictly a calculation engine and does not handle text preprocessing or visualization. Additionally, its reliance on the spaCy ecosystem, while providing a robust foundation, may present a hurdle for researchers using other frameworks like NLTK or Stanza.

Conclusion

TextDescriptives fills a critical gap in the NLP toolkit by unifying disparate metrics into a single, validated framework. Its emphasis on modularity and ease of use supports the broader goal of making complex linguistic analysis more accessible and reproducible across scientific disciplines. As NLP continues to move toward larger datasets and more complex models, tools that prioritize reproducibility and ease of use will remain essential for robust scientific inquiry.

Summary updated

BibTeX
@article{hansen2023textdescriptives,
  author = {Lasse Hansen and Ludvig Renbo Olsen and Kenneth C. Enevoldsen and Kenneth Enevoldsen},
  title = {{TextDescriptives: A Python package for calculating a large variety of metrics from text}},
  journal = {J. Open Source Softw},
  year = {2023},
  doi = {10.21105/joss.05153},
}
Figure