Large Language Models Have Not Yet Transformed Science

Ron Bekkerman, Nichole Li, Bahareh Kargar, Mark SomersarXiv 2026 · arXiv

TL;DR

While "AI slop" and ChatGPT-favored vocabulary like "delve" are now visible in the literature, Large Language Models (LLMs) have not yet caused a fundamental shift in the macro-level trajectory of science. Analyzing 2.5 million papers, researchers found that most trends in lexical diversity, paper structure, and topic variety are simply continuations of patterns that began long before the 2022 AI "shock."

What Problem Does This Paper Address?

Since the public release of ChatGPT in late 2022, the scientific community has worried about an "AI paradox": a world where researchers become more productive but at the cost of research quality, rigor, and novelty. Critics fear that LLMs might foster a "scientific monoculture," where papers become homogenized, less creative, and filled with superficial "slop."

The authors of this paper sought to move beyond anecdotal evidence. They developed a comprehensive framework to quantify whether the emergence of LLMs caused a "structural break"—a sudden change in direction—across the lexical, epistemological, and social dimensions of global science.

Figure 1: A comprehensive framework for analyzing changes in scientific research following the emergence of LLMs, as reflected in scientific writing. The framework spans the broad spectrum of relevant attributes, organized into nested lexical, epistemological, and social dimensions. We measure a limited number of representative attributes within each dimension, from the narrowest (word usage) to the widest (global landscape of science). The comprehensiveness of our framework lies in the breadth of dimensions covered, while the selected attributes provide an initial operationalization of this broad spectrum and a foundation for further work.

Methods

The team employed a multi-dimensional analytical approach:

  • Segmented Linear Regression: They fitted models to various metrics with a "knot" (a potential breaking point) fixed at 2022 to see if trends significantly accelerated, decelerated, or reversed after LLMs arrived.
  • Lexical Analysis: They tracked the frequency of 14,484 common English lemmas and measured lexical diversity using Moving Average Type-Token Ratio (MATTR) to account for varying paper lengths.
  • Topic Diversity: They used Jensen-Shannon (JS) distances to measure how much papers within a discipline overlap in their vocabulary and terminology.
  • Structural Mapping: They used a dictionary of over 80 naming variants to map section headings (e.g., "Methodology" vs. "Proposed Method") to a standard 9-section core structure.
  • Geographic Cascading: To identify paper origins (not always in metadata), they used a priority-based search: first checking email domains, then paper headers, then the full text.

Datasets

The study analyzed approximately 2.5 million full-text papers published between 2015 and 2025 across five disciplines:

  • Computer Science, Mathematics, and Physics: Sourced from the arXiv database.
  • Health Sciences and Life Sciences: Sourced from PubMed Central (PMC).

Key Insights and Ideas

  • The "Delve" Peak: Certain words have become markers of LLM usage. "Surge" words (like underscore and establish) are sophisticated terms LLMs prefer, while "decline" words (like good and happen) are plain terms LLMs avoid. Interestingly, the word delve showed a "rise-and-drop" pattern—peaking in 2024 and declining in 2025—suggesting researchers are becoming aware of "AI slop" markers and are actively editing them out.
  • Science is Getting "More Scientific": The density of scientific terminology has been on a steady upward trend since 2015. LLMs haven't "diluted" papers; if anything, the trend toward more technical language is continuing.
  • Homogenization is an Illusion (Mostly): While topic diversity in Health and Life Sciences appeared to drop sharply after 2022, the authors discovered this was largely due to standardization in article formats (boilerplate text for disclosures, licenses, and contributor roles). Once they filtered out 105 common boilerplate phrases, the "decline" mostly vanished.
  • The Asian Acceleration: The most significant post-2022 "break" wasn't in language, but in social dynamics. Computer Science co-author teams in Asia saw a significant acceleration in size, suggesting LLMs may be acting as a "linguistic equalizer," helping non-English speaking regions participate more heavily in global research.

Key Results

  • Vocabulary Impact is Narrow: Only about 1,000 words (out of millions of unique terms) showed significant frequency shifts post-2022. The overall lexical composition of science remains largely unchanged.
  • Structural Continuity: Computer Science papers are adopting more standardized section names and growing longer, but these trends were well-established by 2018-2019.
  • Productivity Gains: Research productivity (number of papers) in Computer Science and Physics appears amplified after 2022, confirming that LLMs are likely boosting output volume.
Trends for top Surge, Rise-and-Drop, and Decline words across disciplines.

Limitations and Future Work

  • Short Observation Window: With only about two years of post-LLM data, it may be too early to see deep epistemological shifts that take a decade to manifest.
  • Disciplinary Scope: The study focused on five STEM-heavy fields; the impact on Social Sciences or Humanities might differ.
  • Boilerplate Noise: Despite filtering, the authors acknowledge that ongoing metadata standardization in biomedical journals still creates "noise" that makes papers look more similar than they actually are.
  • Future Direction: The authors suggest their framework should be used for long-term monitoring to see if the "AI paradox" eventually takes hold as LLM adoption deepens and models evolve.

Summary updated

Abstract

The emergence of large language models (LLMs) in late 2022 has raised questions about the future of science, but assessing their impact requires understanding how scientific practices were already evolving. We analyze 2.5 million full-text papers across five disciplines from 2015 to 2025, tracing trends over a broad spectrum of lexical, structural, epistemological, and social dimensions. Aside from frequency shifts in some words, most pre-2023 trends continued with limited detectable breaks. Among the notable exceptions, growth in computer science coauthor teams accelerated significantly in Asia, while apparent declines in biomedical topic diversity largely reflected standardization in article formats. Our study establishes an empirical foundation for monitoring the health of global science as LLM adoption deepens.
BibTeX
@misc{bekkerman2026large,
  author = {Ron Bekkerman and Nichole Li and Bahareh Kargar and Mark Somers},
  title = {{Large Language Models Have Not Yet Transformed Science}},
  year = {2026},
  eprint = {2611.22975},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22975},
}
Figure