Abstract
SentSpace is an open-source modular framework designed to evaluate text through "cognitively motivated" features—metrics derived from human psychology and brain processing. By bridging the gap between natural language processing (NLP) and cognitive science, SentSpace enables researchers to quantify subtle differences between human-written and machine-generated text that standard metrics often overlook.
Introduction
As Large Language Models (LLMs) advance, they produce text that is surface-level fluent and often indistinguishable from human writing. However, standard NLP metrics like BLEU or perplexity fail to capture "human-likeness" in terms of how the human brain actually processes language. Researchers across cognitive science, linguistics, and NLP lack a unified tool to characterize text across diverse dimensions such as word-level psychological norms, syntactic complexity, and semantic embeddings. SentSpace addresses this by providing a streamlined pipeline to benchmark text against massive human-generated corpora, identifying where AI text deviates from human linguistic behavior.
System Architecture and Methods
SentSpace is built as a modular Python framework complemented by a web-based interface for visualization and ease of use. The system processes text through a structured pipeline, as illustrated in the data flow diagram below.
The framework characterizes text across three primary feature spaces:
- Lexical Features: Word-level metrics derived from behavioral experiments, including Age of Acquisition, Concreteness, and emotional impact (Arousal/Valence).
- Contextual Features: Sentence-level metrics including Syntactic Integration Cost (based on Dependency Locality Theory) and Surprisal (predictability).
- Embeddings: High-dimensional semantic fingerprints from models like GloVe or Transformers.
The modular design allows for "plug-and-play" integration of custom user-contributed modules. For non-programmatic access, a web interface allows users to submit text for feature computation.
Normative Baselines and Evaluation
To establish a "normative" baseline, the authors pre-computed features for approximately 4,000 sentences across eight major corpora, including the Brown Corpus, Wall Street Journal, and C4. In a demonstration case study, they compared 52 human-written paragraphs against 260 paragraphs generated by GPT-2 XL.
Key Insights: Human vs. Machine Text
The evaluation revealed that surface-level fluency does not equate to linguistic normativity. Key findings include:
- The Simplicity Bias: Machine-generated text often exhibits lower 3-gram surprisal and lower Syntactic Integration Cost, suggesting AI sentences are more predictable and syntactically simpler than human-produced structures.
- Vocabulary Safety: GPT-2 XL tends to use more frequent, less concrete words. Humans used significantly more concrete language (Mean: 2.69) compared to the model (Mean: 2.58).
- Structural Complexity: Human text showed a higher Syntactic Integration Cost (0.68 vs. 0.62), indicating more complex dependency structures.
Conclusion and Future Work
SentSpace provides a novel lens for evaluating AI safety and alignment by quantifying the brain-language link. Current limitations include a focus on English due to the availability of behavioral psycholinguistic data. Future work aims to expand the feature set to include metrics like Body-Object Interaction and Dominance, and to extend support to other languages as data becomes available.
Summary updated