TL;DR
This paper investigates whether Large Language Models (LLMs) can replicate the nuanced, often chaotic, style of human stream-of-consciousness (SoC) writing. Researchers found that while LLMs can produce coherent text, they struggle to capture the subjective, personality-driven qualities that characterize human SoC, even when explicitly prompted with personality traits.
What Problem Does This Paper Address?
Stream-of-consciousness (SoC) writing is a literary technique that mimics the free flow of human thought, characterized by digressions, fragmented structures, and a lack of linear progression. This style presents a unique challenge for LLMs, which are typically trained for coherence and fluency. While LLMs excel at many tasks, their ability to capture the spontaneous, emotionally rich, and personality-infused qualities of human writing, particularly in a genre as unstructured as SoC, remains an open question. This paper aims to bridge this gap by comparing LLM-generated SoC continuations with human-written ones, focusing on narrative coherence, textual complexity, and personality expression.
Methods
The researchers employed a text continuation paradigm. They took the first half of human-written SoC essays and used them as prompts for two LLMs: Llama-3.1-8B and DeepSeek-R1-Distill-Llama-8B. Two prompting conditions were tested: one without explicit personality information and another that included descriptions of the writer's Big Five personality traits. The generated continuations were then evaluated using several metrics:
- Narrative Coherence: Measured by perplexity (predictability of text), cosine similarity (semantic alignment with human text), BLEU score (n-gram overlap), and SARI score (text simplification quality).
- Textual Complexity: Assessed using traditional readability formulas like Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level, SMOG Index, Automated Readability Index (ARI), and Dale-Chall Score (DCS).
- Personality Expression: Quantified using a BERT-based model fine-tuned to detect Big Five personality traits.
A key aspect of the DeepSeek model's analysis involved distinguishing between text generated before and after its explicit "thinking" process, allowing for an examination of how internal reasoning might affect output.
Datasets
The study utilized the Pennebaker's SoC dataset, which contains over 2000 essays written by undergraduate students, each paired with Big Five personality assessments. This dataset was chosen for its spontaneous, unedited nature, making it ideal for capturing natural linguistic patterns and psychological expressiveness.
Key Insights and Ideas
- LLMs are Predictable, Humans are Not: LLM-generated continuations consistently showed low alignment with human texts, suggesting that while LLMs can be coherent, they lack the inherent unpredictability and subjective depth of human SoC.
- Personality Prompts Help, But Don't Solve Everything: Explicitly providing personality traits to LLMs improved Llama-3.1-8B's performance, particularly in consistency, but did not fully bridge the gap to human-level authenticity.
- The "Thinking" of DeepSeek is Revealing: DeepSeek's internal reasoning process led to dramatic shifts in personality expression and readability, indicating that its "thinking" phase significantly influences its output style.
- Openness is a Key Differentiator: Human texts consistently scored higher in Openness, a trait associated with creativity and non-linear thinking, which LLMs struggled to replicate. This suggests a fundamental limitation in LLMs' ability to emulate the core characteristics of SoC writing.
Key Results
- Low Coherence Alignment: Cosine similarity scores between LLM and human continuations were consistently low, falling below moderate and high similarity thresholds.
- Perplexity Differences: Llama-3.1-8B exhibited lower perplexity than DeepSeek, suggesting better structural prediction, but this might reflect fluency rather than genuine narrative alignment. DeepSeek's perplexity increased slightly when given personality prompts.
- Readability Similarities: Llama-3.1-8B's readability metrics were closer to human texts, while DeepSeek's outputs were generally more complex.
- Personality Trait Discrepancies: Human texts showed higher Agreeableness and Openness. Llama consistently showed high Extraversion and low Agreeableness. DeepSeek's personality expression shifted dramatically during its "thinking" process, especially with personality prompts. For instance, Extraversion dropped significantly while Agreeableness increased.
Limitations and Future Work
- Limited Model Scope: The study used only two relatively small LLMs, which may not represent the full spectrum of LLM capabilities. Future work could explore larger, more diverse models.
- Temperature Control: The researchers did not experiment with different temperature settings, which are known to influence creativity.
- Token Length for Classifier: The 512-token limit of the BERT classifier required chunking, potentially losing contextual information for personality detection.
- Prompt Design: The fixed 24-sentence constraint in prompts might have imposed unnatural writing patterns.
- No Human Analysis: The study lacked a qualitative human analysis of the texts, which could offer deeper insights into narrative structure and richness.
- Single Language/Domain: The analysis was limited to English SoC essays, and findings may not generalize to other languages, styles, or domains.
Future work could involve adding human judgment, exploring multilingual settings, and investigating the ethical implications of LLM text detection.
Summary updated