What Changed in Engineering Writing After Generative AI Chat Assistants: Evidence from 1.8 Million Articles

Taner CokyasararXiv 2026 · arXiv

TL;DR

Generative AI has fundamentally shifted the linguistic landscape of engineering research. By analyzing 1.8 million articles, this study finds a massive surge in AI-favored vocabulary and a shift toward more confident, less nuanced writing, even as structural publishing metrics like author counts and publication volume remain stable.

What Problem Does This Paper Address?

Since the public release of ChatGPT in late 2022, the academic community has been flooded with anecdotes and small-scale studies about AI-assisted writing. However, measuring this shift at scale is difficult, especially in fields like engineering that lack a centralized, uniform index for abstracts (unlike biomedicine’s PubMed or computer science’s arXiv).

Taner Cokyasar tackles this by asking: What measurably changed in engineering research literature after November 30, 2022? The paper aims to move beyond "vibes" and detection tools (which are notoriously biased against non-native English speakers) to provide a descriptive, large-scale account of how the actual prose of engineering has evolved.

Methods

The study employs an interrupted time series (ITS) design. The author established a baseline by fitting trends to data from 2015 through November 2022 and then projected those trends forward to 2025. The "departure from trend"—the gap between what was observed and what was projected—serves as the measure of change.

Key methodological innovations include:

  • Marker vs. Control Words: The study tracks 31 "marker words" known to be favored by LLMs (e.g., delve, intricate, showcase, underscore) and compares them against 4 "control words" of the same formal register (e.g., holistic, moreover) that are not typically overused by AI.
  • Correcting for "False Januarys": The author discovered that many bibliographic records lacking a specific month default to January 1st, creating artificial spikes. These were removed to ensure a clean monthly series.
  • Full-Text Validation: While the bulk of the 1.8M articles were analyzed via abstracts, a random sample of ~12,800 full texts was used to verify if the linguistic shifts seen in short abstracts persisted throughout the entire paper.

Datasets

The study assembled a massive corpus of 1,839,962 articles published between 2015 and 2025 across 256 top-tier engineering journals (selected by Google Scholar h5-index).

Data sources included:

  • OpenAlex: For primary metadata and abstracts.
  • Semantic Scholar: To fill in missing abstracts.
  • Unpaywall: To locate legal full-text copies for the validation sample.

The availability of abstracts varies significantly by publisher, a factor the author accounted for in the analysis:

Table 1: Share of Articles with an Abstract of at Least Forty Words, by Publisher.

Key Insights and Ideas

  • The Death of Hedging: One of the most striking findings is the shift in tone. "Qualifying" words that signal scientific uncertainty (may, might, suggests) fell by 13%, while "emphatic" words (clearly, substantially) rose by 20%. AI-assisted prose tends to be more confident and less nuanced than traditional human-written engineering text.
  • Linguistic Fingerprints: The rise in marker words (+82% in abstracts) was not just a general shift toward formal writing. If it were, the control words would have risen too; instead, they fell by 16%. This suggests authors are specifically adopting (or failing to edit out) the idiosyncratic vocabulary of LLMs.
  • Structural Stability: Despite the radical change in how papers are written, the structure of research remains unchanged. Author counts, reference list lengths, and page counts all stayed within 4% of their pre-AI trends. AI is currently a writing assistant, not a tool that has (yet) fundamentally altered the scale of research collaboration or output.
  • The "Delve" Dilemma: The word delve—the poster child for AI-writing detection—rose from 0.7 to 8.1 occurrences per million words. Interestingly, the study noted that some marker words began to decline in late 2025, suggesting authors may be adapting their prompts to avoid words that have become "red flags" for AI use.

Key Results

  • Marker Word Surge: Marker words appeared 82% more often than projected in abstracts and 99% more often in article bodies.
  • Vocabulary Complexity: The share of long words (9+ letters) rose by 11%, and word variety increased by 4.3%.
  • Titular Colons: The use of two-part titles (separated by a colon), a known preference of LLMs, rose 23.9% above trend.
  • Low Disclosure: Despite the clear linguistic evidence of AI use, only 3.7% of articles in 2025 included a formal disclosure statement. Interestingly, 80% of these disclosures came from a single publisher (Elsevier), likely due to their specific manuscript submission policies.

Limitations and Future Work

  • Lagging Indicators: Because the study uses publication dates, there is an inherent lag between when a paper is written (and AI is used) and when it appears in the dataset.
  • Descriptive, Not Causal: While the correlation with the release of ChatGPT is overwhelming, the study cannot strictly prove causality, as there is no "control group" of researchers who were denied access to AI.
  • Publisher Bias: The reliance on machine-readable abstracts means the data under-represents publishers who do not openly deposit abstract text.
  • Future Directions: The author suggests that future research should use submission dates (where available from publishers) to get a more real-time view of AI adoption and expand full-text analysis to a larger percentage of the corpus.

Summary updated

Abstract

This study asks what changed in engineering articles after conversational language models became publicly available on November 30, 2022, and develops methods for measuring such change in a field without a uniform abstract index. It assembled 1,839,962 articles published from 2015 to 2025 in 256 engineering journals, tracked twelve measures of wording, style, and article structure, and compared each with the projection of its own pre-2023 trend. Building the monthly series showed that records dated only by year fall on January 1 and create false January peaks unless removed. In abstracts, 31 words associated with language model output appeared 82% more often than projected (55.4 against 27.0 per 10,000 words in 2025), while four control words of the same formal register fell 16%. Words that soften a claim, such as may, fell 13%, words that strengthen one, such as clearly, rose 20%, and long words rose 11%. Author counts, reference lists, page counts, and publication volume stayed within 4% of trend. The change appears in all fourteen subject categories, and false release dates, dropping any one journal or word, and word lists chosen from the data support it. A random sample of 12,866 full texts from the same corpus, including subscription articles and weighted back to it, shows the same words rising 99% in article bodies. Statements disclosing AI writing assistance, measured apart from mentions of language models, went from none to 3.7% in 2025, four in five of them at one publisher. The design is descriptive, not causal.
BibTeX
@misc{cokyasar2026what,
  author = {Taner Cokyasar},
  title = {{What Changed in Engineering Writing After Generative AI Chat Assistants: Evidence from 1.8 Million Articles}},
  year = {2026},
  eprint = {2611.11174},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11174},
}
Figure