TL;DR
While "AI slop" and ChatGPT-favored vocabulary like "delve" are now visible in the literature, Large Language Models (LLMs) have not yet caused a fundamental shift in the macro-level trajectory of science. Analyzing 2.5 million papers, researchers found that most trends in lexical diversity, paper structure, and topic variety are simply continuations of patterns that began long before the 2022 AI "shock."
What Problem Does This Paper Address?
Since the public release of ChatGPT in late 2022, the scientific community has worried about an "AI paradox": a world where researchers become more productive but at the cost of research quality, rigor, and novelty. Critics fear that LLMs might foster a "scientific monoculture," where papers become homogenized, less creative, and filled with superficial "slop."
The authors of this paper sought to move beyond anecdotal evidence. They developed a comprehensive framework to quantify whether the emergence of LLMs caused a "structural break"—a sudden change in direction—across the lexical, epistemological, and social dimensions of global science.
Methods
The team employed a multi-dimensional analytical approach:
- Segmented Linear Regression: They fitted models to various metrics with a "knot" (a potential breaking point) fixed at 2022 to see if trends significantly accelerated, decelerated, or reversed after LLMs arrived.
- Lexical Analysis: They tracked the frequency of 14,484 common English lemmas and measured lexical diversity using Moving Average Type-Token Ratio (MATTR) to account for varying paper lengths.
- Topic Diversity: They used Jensen-Shannon (JS) distances to measure how much papers within a discipline overlap in their vocabulary and terminology.
- Structural Mapping: They used a dictionary of over 80 naming variants to map section headings (e.g., "Methodology" vs. "Proposed Method") to a standard 9-section core structure.
- Geographic Cascading: To identify paper origins (not always in metadata), they used a priority-based search: first checking email domains, then paper headers, then the full text.
Datasets
The study analyzed approximately 2.5 million full-text papers published between 2015 and 2025 across five disciplines:
- Computer Science, Mathematics, and Physics: Sourced from the arXiv database.
- Health Sciences and Life Sciences: Sourced from PubMed Central (PMC).
Key Insights and Ideas
- The "Delve" Peak: Certain words have become markers of LLM usage. "Surge" words (like underscore and establish) are sophisticated terms LLMs prefer, while "decline" words (like good and happen) are plain terms LLMs avoid. Interestingly, the word delve showed a "rise-and-drop" pattern—peaking in 2024 and declining in 2025—suggesting researchers are becoming aware of "AI slop" markers and are actively editing them out.
- Science is Getting "More Scientific": The density of scientific terminology has been on a steady upward trend since 2015. LLMs haven't "diluted" papers; if anything, the trend toward more technical language is continuing.
- Homogenization is an Illusion (Mostly): While topic diversity in Health and Life Sciences appeared to drop sharply after 2022, the authors discovered this was largely due to standardization in article formats (boilerplate text for disclosures, licenses, and contributor roles). Once they filtered out 105 common boilerplate phrases, the "decline" mostly vanished.
- The Asian Acceleration: The most significant post-2022 "break" wasn't in language, but in social dynamics. Computer Science co-author teams in Asia saw a significant acceleration in size, suggesting LLMs may be acting as a "linguistic equalizer," helping non-English speaking regions participate more heavily in global research.
Key Results
- Vocabulary Impact is Narrow: Only about 1,000 words (out of millions of unique terms) showed significant frequency shifts post-2022. The overall lexical composition of science remains largely unchanged.
- Structural Continuity: Computer Science papers are adopting more standardized section names and growing longer, but these trends were well-established by 2018-2019.
- Productivity Gains: Research productivity (number of papers) in Computer Science and Physics appears amplified after 2022, confirming that LLMs are likely boosting output volume.
Limitations and Future Work
- Short Observation Window: With only about two years of post-LLM data, it may be too early to see deep epistemological shifts that take a decade to manifest.
- Disciplinary Scope: The study focused on five STEM-heavy fields; the impact on Social Sciences or Humanities might differ.
- Boilerplate Noise: Despite filtering, the authors acknowledge that ongoing metadata standardization in biomedical journals still creates "noise" that makes papers look more similar than they actually are.
- Future Direction: The authors suggest their framework should be used for long-term monitoring to see if the "AI paradox" eventually takes hold as LLM adoption deepens and models evolve.
Summary updated