Where Language-Model Vocabulary Rose on Stack Exchange

Fengming LiuarXiv 2026 · arXiv

This paper investigates the "fingerprint" of Large Language Models (LLMs) across the Stack Exchange network following the release of ChatGPT. By tracking a specific set of "marker words" that LLMs tend to overuse, the author reveals that AI-generated content didn't flood the network evenly; instead, its presence is deeply tied to the editing culture of individual sub-communities.

TL;DR

Despite widespread bans, LLM-associated vocabulary rose by approximately 14% across Stack Exchange after ChatGPT's release. The strongest predictor of this rise was a site's "edit share"—communities where authors rarely revise their own posts saw a nearly 30% jump in AI-like prose, while highly active editing communities remained virtually unchanged.

What Problem Does This Paper Address?

When ChatGPT launched in late 2022, platforms like Stack Overflow immediately banned AI-generated answers due to concerns about "hallucinated" or incorrect information. However, enforcing these bans is notoriously difficult.

Previous research has shown that LLMs have a distinct "style," often overusing certain adjectives and connectives (like delve, crucial, or tapestry). This paper seeks to quantify how much of this "AI style" has leaked into Stack Exchange and, more importantly, to identify which site characteristics—such as the technical nature of the topic or the community's moderation habits—predict where AI content is most likely to persist.

Methods

The author employs a "marker vocabulary" approach, tracking a fixed list of 56 words identified by prior studies as being overused by LLMs relative to humans.

Key methodological features include:

  • Control Groups: To distinguish AI influence from general linguistic drift, the author compared the marker list against 1,000 frequency-matched lists of ordinary words.
  • Trait Screening: The study screened 15 different site traits (e.g., code exposure, math notation, answer length) to see which best predicted the vocabulary shift.
  • The "Edit Share" Metric: The author introduced a novel metric: the share of a site’s answers that received a body edit within 30 days of posting. Crucially, this was measured using data from before the release of ChatGPT to ensure the trait was a stable characteristic of the community, not a reaction to AI.

Datasets

The study uses the April 2024 Stack Exchange data dump, encompassing:

  • 3.83 million answers posted between January 2019 and March 2024.
  • 170 main sites (Stack Overflow was excluded from weighted estimates to prevent it from drowning out smaller communities).
  • Revision histories were analyzed to distinguish between "self-edits" (by the author) and "other-edits" (by moderators or peers).

Key Insights and Ideas

  • The Scrutiny Effect: The most surprising finding was that "code exposure" (how much a site deals with programming) was not the primary driver of AI vocabulary. Instead, the community's editing habits were the dominant factor.
  • Self-Editing as a Shield: The "edit share" gradient is almost entirely driven by authors who habitually revise their own work. Communities where authors "post and forget" were far more susceptible to the rise of marker words.
  • Vocabulary as a Fingerprint: The rise in words like ensure, crucial, and furthermore serves as a clear signal of LLM adoption. On the least-edited sites, the document share of crucial nearly doubled post-ChatGPT.
Figure 1: Left, marker-list document share relative to a linear trend fitted to 2019Q1–2022Q3, equal weight per site (sites with at least 20 answers in a quarter). The gray band holds 95 percent of 1,000 frequency-matched control lists. Colored lines split the sites by tercile of the edit share, the share of 2021–22 answers that received a body edit within 30 days of posting. Right, the 2023Q2–2024Q1 ratio to trend for the marker list (diamond) and for each control list (dots), by edit tercile and for all sites.

Key Results

  • Network-Wide Rise: Across the 170 sites, marker vocabulary rose 14% above its pre-2023 trend.
  • The Editing Divide: On the least-edited third of sites (e.g., small product-support forums like Magento or Tezos), the marker list rose by 29%. On the most-edited third (e.g., Skeptics or Role-playing Games), there was no significant change.
  • Predictive Power: The edit share explained 17–18% of the variance in marker change across sites, surviving rigorous statistical corrections.
  • Specific Word Spikes: Ensure alone accounted for a massive portion of the shift. Other heavy hitters included essential, furthermore, and moreover.
Figure 4: Ratio of document share, 2023Q2–2024Q1 over 2021Q1–2022Q3, for list words with a pre-period share of at least 0.2 percent in both the least and the most edited third of sites, by the edit share. Filled circles are the least edited third, open circles the most edited third, crosses the placebo ratio 2021–22 over 2019–20 on the least edited third.

Limitations and Future Work

The author acknowledges a major hurdle: use of the model is unobserved. We cannot definitively say if users on high-edit sites used LLMs less, or if they simply edited the "AI-isms" out of their posts before they became part of the permanent record.

Furthermore, the marker list was built from scientific text; a list tailored specifically to the technical jargon of Stack Exchange might reveal even more subtle shifts. Future research could look at "id-gaps" (missing post IDs) to better understand how much content is being deleted by moderators before it can be analyzed. This work highlights a growing challenge for AI researchers: as LLM-generated text is scraped back into training sets, it may be concentrated in specific, less-scrutinized corners of the internet, potentially leading to model collapse or bias.

Summary updated

Abstract

Stack Overflow banned answers generated by ChatGPT five days after the model's release, and many other Stack Exchange sites announced similar rules. I measure the vocabulary of the answers posted afterwards, 3.8 million answers on 170 sites from 2019 to 2024. A fixed list of 56 words that language models overuse rose about 14 percent above its pre-2023 trend, while the highest of 1,000 frequency-matched lists of ordinary words rose about 7 percent. In a screen of 15 site traits measured before the release, the strongest predictor of where the rise happened was the share of answers that received a body edit within 30 days of posting. On the least edited third of sites the list rose about 29 percent relative to matched controls, and on the most edited third it did not change. The gradient is also present among answers that were never edited. It follows the share of answers edited by their own authors, and the share edited by others adds little once that share is held fixed. Use of the model is unobserved, so these results cannot separate lower use on sites with a high self-edit share from any effect of editing on what the vocabulary records. Any vocabulary-based estimate for the network as a whole therefore pools sites where the marker vocabulary rose sharply with sites where it did not change.
BibTeX
@misc{liu2026where,
  author = {Fengming Liu},
  title = {{Where Language-Model Vocabulary Rose on Stack Exchange}},
  year = {2026},
  eprint = {2611.15172},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15172},
}
Figure