Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking

Harsh Kumar, Jonathan Vincentius, Ewan Jordan, Ashton Anderson

Introduction

As Large Language Models (LLMs) become integrated into professional workflows, a critical debate has emerged regarding their long-term impact on human cognition. Does AI act as "steroids"—providing short-term performance gains at the cost of long-term skill atrophy? Or does it function as a "coach" that fosters cognitive growth? This research addresses the "residual effects" of AI assistance, investigating how human creative abilities change after the tool is removed. The study specifically examines two pillars of creativity: Divergent Thinking (generating a wide range of unique ideas) and Convergent Thinking (finding a single correct solution to a complex problem).

Methodology

The researchers conducted two pre-registered experiments involving 1,100 participants. The study utilized a "sandwich" experimental design to isolate the impact of AI exposure on subsequent independent performance.

Experimental workflow showing the progression from randomization to exposure rounds, a distractor task, and the final unassisted test round.

Participants were randomized into three conditions with a 33.3% probability each:

  1. Control (No LLM): Participants performed tasks entirely unassisted.
  2. LLM Solution (Standard): GPT-4o provided direct answers to the prompts.
  3. LLM Guidance (Coach): The AI provided strategies (e.g., the SCAMPER technique) to nudge participants toward solutions without revealing them.

Following three "Exposure Rounds," participants engaged in a one-minute distractor task (Snake game) to reset cognitive states. Finally, a "Test Round" was conducted where all participants performed tasks unassisted to measure residual effects. Performance was evaluated using automated scoring via fine-tuned GPT-3 models for originality and SBERT embeddings for idea diversity.

The four experimental interfaces: (a) No LLM Response, (b) LLM Answer, (c) LLM Guidance, and (d) the unassisted Test Phase.

Key Findings

The "Crutch" Effect and Homogenization

While AI assistance provided an immediate performance boost during exposure rounds, it acted as a cognitive crutch. Participants previously exposed to AI performed significantly worse than the control group in the unassisted test round. Furthermore, AI-exposed groups exhibited a "homogenization of thought," producing ideas that were mathematically more similar to one another (lower group-level diversity) than those in the control group. This suggests that AI frameworks may inadvertently funnel human creativity into predictable, narrow patterns.

Counterproductive Guidance

Surprisingly, the "Coach-like" guidance was often detrimental. In convergent thinking tasks (Remote Associates Test), the guidance group performed the worst in the test round, achieving approximately 50% accuracy compared to the control group's 60% ($p=0.005$). Researchers hypothesize that the complex strategies provided by the AI increased cognitive load and disrupted the intuitive "aha!" moments necessary for insight-based problem solving.

Subjective Impact

Participants across all conditions reported a decline in perceived creativity post-experiment. This drop was most pronounced in the LLM Guidance group, suggesting that struggling alongside an AI may undermine a user's confidence in their own creative agency.

Conclusion and Limitations

The study concludes that while LLMs are powerful productivity tools, they may weaken the "natural creative muscles" of users. However, the authors note several limitations: the study measured effects over minutes rather than months, focused purely on verbal tasks, and utilized a static interface. Future research is required to determine if more adaptive, interactive "human-in-the-loop" systems could avoid the cognitive load trap and truly augment human skill without creating dependency.

Summary updated

BibTeX
@inproceedings{kumar2025human,
  author = {Harsh Kumar and Jonathan Vincentius and Ewan Jordan and Ashton Anderson},
  title = {{Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking}},
  booktitle = {CHI},
  year = {2025},
  doi = {10.1145/3706598.3714198},
}
Figure