Abstract
TextDescriptives is a Python package for calculating a large variety of metrics from text. It is built on top of spaCy and can be easily integrated into existing workflows. The package has already been used for analysing the linguistic stability of clinical texts, creating features for predicting neuropsychiatric conditions, and analysing linguistic goals of primary school students. This paper describes the package and its features.
Introduction
In natural language processing (NLP), researchers often need to characterize a corpus beyond simple word counts. Metrics like text complexity, readability, and semantic coherence are vital for tasks such as identifying low-quality data, assessing clinical speech patterns, or analyzing educational materials.
However, the current ecosystem is fragmented. Researchers often have to juggle multiple libraries or write custom, untested code to calculate these metrics. This "DIY" approach is not only time-consuming but also prone to bugs and subtle implementation differences that undermine the reproducibility of scientific results. TextDescriptives aims to solve this by providing a unified, tested, and easy-to-use library for document-level metrics.
System Architecture
The package is built as a series of spaCy pipeline components. This architectural choice is significant because it allows the package to leverage spaCy’s highly optimized tokenization, part-of-speech (POS) tagging, and dependency parsing. By functioning as a native component, TextDescriptives does not require users to learn a new workflow; if a researcher already has a spaCy pipeline, adding these metrics is a "plug-and-play" operation.
Core Features and Modules
TextDescriptives is organized into several key modules designed to provide a holistic view of text structure and quality:
- Descriptive Statistics: Calculates fundamental metrics including token and sentence counts, unique token ratios, and syllable counts.
- Readability: Implements standard formulas such as Gunning-Fog, SMOG, and Flesch-Kincaid to assess the accessibility of text.
- Dependency Distance: Measures the average distance between a word and its "head" word in a sentence. This serves as a common proxy for linguistic complexity and comprehension difficulty.
- POS Proportions: Calculates the distribution of different parts of speech, allowing for stylistic and structural analysis.
- Coherence: Uses word embedding similarity between sentences to measure the logical flow and semantic consistency of a document.
- Quality: Includes heuristics to filter out "noise" (e.g., excessive stop words, duplicate lines, or "lorem ipsum" placeholders). This is particularly essential for cleaning web-scraped data before training large language models (LLMs).
Research Applications
While TextDescriptives is a software tool rather than a model, its utility has been validated through various real-world research applications:
- Clinical Texts: It has been used to analyze the linguistic stability of psychiatric notes over time.
- Neuropsychiatric Prediction: Researchers have utilized the package to create features for predicting conditions such as psychosis or schizophrenia based on speech patterns.
- Educational Analysis: The tool has assisted in analyzing the writing goals and progress of primary school students.
Comparison and Performance
The primary advantage of TextDescriptives is its API efficiency. Users can extract all metrics into a structured format, such as a Pandas DataFrame or a Python dictionary, using a single call: textdescriptives.extract_df(doc).
The package fills several gaps left by existing tools:
- Unlike textstat, it allows for the batch extraction of multiple metrics simultaneously.
- Unlike spacy-readability, it maintains compatibility with modern versions of
spaCy(v3.0+). - Unlike textacy, it focuses specifically on document-level descriptive metrics rather than broader information extraction.
Limitations and Future Work
As the package is tied to the spaCy ecosystem, its utility depends on the availability and accuracy of spaCy models for specific languages. Furthermore, calculating complex metrics like embedding-based coherence or dependency distances introduces more computational overhead than simple surface-level statistics. Future updates are expected to include more specialized metrics tailored for researchers in the digital humanities and social sciences.
Conclusion
TextDescriptives streamlines the extraction of a wide array of text metrics, from basic statistics to complex measures of coherence. By centralizing these metrics into a single, tested library, the package promotes reproducibility and provides a robust framework for linguistic feature extraction in modern NLP pipelines.
Summary updated