How do Language Models Generate Slang: A Systematic Comparison between Human and Machine-Generated Slang Usages

Siyang Wu, Zhewei SunEMNLP 2025

TL;DR

This paper systematically compares slang generated by large language models (LLMs) like GPT-4o and Llama-3 with human-attested slang. The authors find that while LLMs can capture some creative aspects of slang, their generated usages exhibit systematic biases and don't fully align with human patterns, suggesting limitations in their understanding for complex linguistic tasks.

What Problem Does This Paper Address?

Slang is a dynamic and informal aspect of language that poses a significant challenge for Natural Language Processing (NLP) systems. While recent advancements in LLMs have shown promise in tasks like slang detection and interpretation, their reliability hinges on whether they truly grasp the underlying structural knowledge of slang, mirroring human usage. This paper investigates this gap by systematically comparing machine-generated slang with human-attested slang to understand the extent to which LLMs have captured and can generalize slang's structural nuances. The authors aim to gauge the trustworthiness of LLMs for downstream applications that rely on accurate slang characterization, such as model distillation or linguistic analysis.

Methods

The researchers developed an evaluative framework focusing on three core aspects:

  1. Characteristics: Analyzing systematic biases in how LLMs perceive and generate slang.
  2. Creativity: Measuring both lexical coinages (creating new words) and word reuses (assigning new meanings to existing words). This involved assessing morphological complexity, coherence, novelty, and surprisal.
  3. Informativeness: Evaluating how well machine-generated slang examples can inform other models, particularly in a knowledge distillation setting.

They collected a large dataset of machine-generated slang from GPT-4o and Llama-3 under various controlled and uncontrolled conditions, including different prompting strategies for word coinage and reuse. This dataset was then systematically compared against human-attested slang from the Online Slang Dictionary (OSD).

Datasets

  • Human-Attested Slang: The Online Slang Dictionary (OSD) was used as the primary source for human-generated slang. Entries were partitioned into categories based on their degree of conventionalization (Highly Conventionalized, Conventionalized, Non-Conventionalized) using English Wiktionary.
  • Machine-Generated Slang: A new dataset was created by prompting GPT-4o and Llama-3-8B to generate slang usages. This dataset spans various conditions:
  • Uncontrolled Generation: Models generated slang freely.
  • Controlled Generation: Models were prompted with existing definitions from OSD.
  • Word Choice Control: Generations were further categorized into "Coinage" (creating new terms), "Reuse" (using existing terms), and "Free-form" (allowing either).
  • Evaluation Datasets: For downstream tasks, the OSD test set and the OpenSubtitles-Slang corpus were used.

Key Insights and Ideas

  • LLMs Show a Strong Bias Towards Coinage: Both GPT-4o and Llama-3 heavily favor creating new slang terms (coinage) over reusing existing words with new meanings, unlike human slang which shows a more balanced distribution.
  • Human Slang is More Diverse and Less Predictable: Human slang exhibits a wider creative spectrum and is less predictable in its usage compared to LLM-generated slang, which tends to cluster around specific creativity levels.
  • LLMs Capture Creative Aspects, But Not Nuance: While LLMs can generate plausible and creative slang, their understanding doesn't fully align with human patterns, particularly in terms of semantic coherence and contextual surprisal.
  • Alignment Techniques May Influence Topics: Machine-generated slang topics tend to be more positive and less concrete than human slang, potentially due to alignment techniques (like RLHF) steering models away from controversial content.
  • Fine-tuning Can Propagate Biases: Fine-tuning smaller models on LLM-generated slang can transfer creative preferences but also propagates the biases of the teacher model, rather than necessarily reflecting human knowledge.

Key Results

  • Word Formation: LLMs, especially GPT-4o, show a strong inclination towards coinage, with GPT-4o producing significantly more coinages than human slang. Llama models showed less preference for specific word formation strategies like compounding or blending.
  • Creativity in Coinage: GPT-4o generated coinages were morphologically more complex and semantically coherent than human coinages. However, when conditioned on human data, GPT-4o's coherence decreased.
  • Creativity in Reuse: LLMs generated slang with high novelty scores, suggesting they can create semantically divergent usages. However, human slang showed higher dispersion, indicating a broader creative range. Surprisal scores were high across the board for both, suggesting LLMs can control contextual surprisal, though GPT-4o in controlled settings was slightly more coherent than humans.
  • Informativeness: In downstream tasks like slang generation and interpretation, GPT-4o consistently outperformed Llama models. Fine-tuning Llama on both human and machine-generated slang showed minimal gains on basic tasks but improved definition generation quality, with human data being more informative for this specific task.

Limitations and Future Work

  • Limited Model Scope: The study was limited to GPT-4o and Llama-8B-INT. Future work should include a broader range of commercial and open-source LLMs, including larger variants like Llama-70B.
  • English-Centric: The research focused solely on English slang. Future work should explore slang in other languages and dialects to assess cross-cultural generalizability.
  • Quantitative Evaluation: The reliance on quantitative metrics might not fully capture the richness of slang. Future work could complement these with qualitative human studies.
  • Static Nature of Dictionaries: Using slang dictionaries for human data might not fully represent the dynamic nature of evolving slang. However, this is consistent with the static nature of LLM outputs and allows for objective comparisons.

Summary updated

BibTeX
@inproceedings{wu2025how,
  author = {Siyang Wu and Zhewei Sun},
  title = {{How do Language Models Generate Slang: A Systematic Comparison between Human and Machine-Generated Slang Usages}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure