TextDescriptives: A Python package for calculating a large variety of metrics from text

Lasse Hansen, Ludvig Renbo Olsen, Kenneth Enevoldsen

Abstract

TextDescriptives is a Python package for calculating a large variety of metrics from text. It is built on top of spaCy and can be easily integrated into existing workflows. The package has already been used for analysing the linguistic stability of clinical texts, creating features for predicting neuropsychiatric conditions, and analysing linguistic goals of primary school students. This paper describes the package and its features.

Introduction

In natural language processing (NLP), researchers often need to characterize a corpus beyond simple word counts. Metrics like text complexity, readability, and semantic coherence are vital for tasks such as identifying low-quality data, assessing clinical speech patterns, or analyzing educational materials.

However, the current ecosystem is fragmented. Researchers often have to juggle multiple libraries or write custom, untested code to calculate these metrics. This "DIY" approach is not only time-consuming but also prone to bugs and subtle implementation differences that undermine the reproducibility of scientific results. TextDescriptives aims to solve this by providing a unified, tested, and easy-to-use library for document-level metrics.

System Architecture

The package is built as a series of spaCy pipeline components. This architectural choice is significant because it allows the package to leverage spaCy’s highly optimized tokenization, part-of-speech (POS) tagging, and dependency parsing. By functioning as a native component, TextDescriptives does not require users to learn a new workflow; if a researcher already has a spaCy pipeline, adding these metrics is a "plug-and-play" operation.

Core Features and Modules

TextDescriptives is organized into several key modules designed to provide a holistic view of text structure and quality:

  • Descriptive Statistics: Calculates fundamental metrics including token and sentence counts, unique token ratios, and syllable counts.
  • Readability: Implements standard formulas such as Gunning-Fog, SMOG, and Flesch-Kincaid to assess the accessibility of text.
  • Dependency Distance: Measures the average distance between a word and its "head" word in a sentence. This serves as a common proxy for linguistic complexity and comprehension difficulty.
  • POS Proportions: Calculates the distribution of different parts of speech, allowing for stylistic and structural analysis.
  • Coherence: Uses word embedding similarity between sentences to measure the logical flow and semantic consistency of a document.
  • Quality: Includes heuristics to filter out "noise" (e.g., excessive stop words, duplicate lines, or "lorem ipsum" placeholders). This is particularly essential for cleaning web-scraped data before training large language models (LLMs).

Research Applications

While TextDescriptives is a software tool rather than a model, its utility has been validated through various real-world research applications:

  1. Clinical Texts: It has been used to analyze the linguistic stability of psychiatric notes over time.
  2. Neuropsychiatric Prediction: Researchers have utilized the package to create features for predicting conditions such as psychosis or schizophrenia based on speech patterns.
  3. Educational Analysis: The tool has assisted in analyzing the writing goals and progress of primary school students.

Comparison and Performance

The primary advantage of TextDescriptives is its API efficiency. Users can extract all metrics into a structured format, such as a Pandas DataFrame or a Python dictionary, using a single call: textdescriptives.extract_df(doc).

The package fills several gaps left by existing tools:

  • Unlike textstat, it allows for the batch extraction of multiple metrics simultaneously.
  • Unlike spacy-readability, it maintains compatibility with modern versions of spaCy (v3.0+).
  • Unlike textacy, it focuses specifically on document-level descriptive metrics rather than broader information extraction.

Limitations and Future Work

As the package is tied to the spaCy ecosystem, its utility depends on the availability and accuracy of spaCy models for specific languages. Furthermore, calculating complex metrics like embedding-based coherence or dependency distances introduces more computational overhead than simple surface-level statistics. Future updates are expected to include more specialized metrics tailored for researchers in the digital humanities and social sciences.

Conclusion

TextDescriptives streamlines the extraction of a wide array of text metrics, from basic statistics to complex measures of coherence. By centralizing these metrics into a single, tested library, the package promotes reproducibility and provides a robust framework for linguistic feature extraction in modern NLP pipelines.

Summary updated

Abstract

Natural language processing (NLP) tasks often require a thorough understanding and description of the corpus. Document-level metrics can be used to identify low-quality data, assess outliers, or understand differences between groups. Further, text metrics have long been used in fields such as the digital humanities where e.g. metrics of text complexity are commonly used to analyse, understand and compare text corpora. However, extracting complex metrics can be an error-prone process and is rarely rigorously tested in research implementations. This can lead to subtle differences between implementations and reduces the reproducibility of scientific results. TextDescriptives offers a simple and modular approach to extracting both simple and complex metrics from text. It achieves this by building on the spaCy framework (Honnibal et al., 2020). This means that TextDescriptives can easily be integrated into existing workflows while leveraging the efficiency and robustness of the spaCy library. The package has already been used for analysing the linguistic stability of clinical texts (Hansen et al., 2022), creating features for predicting neuropsychiatric conditions (Hansen et al., 2023), and analysing linguistic goals of primary school students (Tannert, 2023).
BibTeX
@misc{hansen2023textdescriptives,
  author = {Lasse Hansen and Ludvig Renbo Olsen and Kenneth Enevoldsen},
  title = {{TextDescriptives: A Python package for calculating a large variety of metrics from text}},
  year = {2023},
  eprint = {2301.02057},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.02057},
}
Figure