Corpus-Driven Analysis of Conceptual Metaphor in Artificial Intelligence Language: A Sample of ChatGPT-Written Speeches

Yang YangJournal of Contemporary Educational Research 2023 · DOI

TL;DR

This study investigates how ChatGPT employs conceptual metaphors across various topics, revealing that while the AI is highly capable of generating diverse metaphorical mappings, it struggles to "turn off" this behavior even when explicitly instructed. The research highlights a significant gap between AI metaphor generation and its ability to identify metaphors according to human linguistic standards, suggesting a need for more nuanced automated identification models.

What Problem Does This Paper Address?

Metaphor is not just a literary device; according to Conceptual Metaphor Theory (CMT), it is a fundamental cognitive tool humans use to understand abstract concepts (target domains) through concrete experiences (source domains)—for example, "argument is war" or "time is money."

While the technical capabilities of Large Language Models (LLMs) like ChatGPT are well-documented, there is a "noticeable dearth" of research from a linguistic standpoint. The author seeks to understand if AI-generated language mirrors human cognitive patterns in metaphor usage. Specifically, the paper addresses whether ChatGPT can intentionally use (or avoid) metaphors and how its selection of "source domains" (the concrete concepts used for comparison) varies across different subjects.

Methods

The researcher employed a corpus-driven approach combined with manual linguistic analysis. The methodology followed these steps:

  1. Corpus Generation: ChatGPT was prompted to write 20 speeches across 10 topics (Education, Ecology, Poverty, etc.). For each topic, it was asked to write one version with metaphors and one without.
  2. Identification: The author used the MIPVU (Metaphor Identification Procedure of VU University Amsterdam), a standardized protocol for identifying metaphorical words in text.
  3. Quantification: To measure the "productivity" of specific metaphors, the author used a formula for Source Domain Resonance:
  4. Resonance = Σ(type) × Σ(token) (where 'type' is the unique metaphor category and 'token' is the frequency of occurrence).
  5. Statistical Validation: Pearson’s correlation was used for inter-rater reliability, and an independent samples t-test was used to compare AI performance against manual identification.

Datasets

The study utilized a custom-built "tiny corpus" created specifically for this research:

  • Size: 10,553 words and 10,599 tokens.
  • Structure: 20 speeches covering 10 diverse social and scientific topics.
  • Comparison: The dataset is split into "Group 1" (intended to be metaphor-free) and "Group 2" (intended to include metaphors).

Key Insights and Ideas

  • The "Planting" Preference: Across topics like Education, Economy, and Health, ChatGPT heavily favored the "Planting" source domain (e.g., "sowing seeds of growth," "cultivating knowledge"). This suggests the AI relies on deeply entrenched, conventional human metaphors.
  • Non-Arbitrary Selection: The AI used a higher variety of source domains for abstract topics (like Politics, which used 21 different source domains) than for concrete ones. The author suggests that the number of source domains chosen reflects the AI's "level of understanding" or the perceived complexity of the target domain.
  • The "Leaky" Metaphor Problem: One of the most interesting findings is that ChatGPT cannot easily produce literal language. Even when told to avoid metaphors, 6.27% of the text remained metaphorical. This suggests that metaphor is so baked into the training data and the structure of language that the AI cannot fully decouple the two.
  • Linguistic Crutches: ChatGPT relies heavily on explicit markers like "like" (used 100% of the time in certain contexts), "is," and "as" to establish metaphorical connections, mirroring common human analogical expressions.

Key Results

  • Metaphorical Density: In speeches where metaphors were requested, they accounted for 12% of the total text.
  • Failed Avoidance: In speeches where metaphors were forbidden, they still accounted for 6.27% of the text. The topic of "Corruption" was the most difficult for the AI to describe literally, with a 10.02% metaphorical frequency despite instructions to avoid them.
  • Identification Gap: The t-test yielded a P-value of 0.00000045619, indicating a statistically significant difference between how the AI identifies/utilizes metaphors and how a human linguist identifies them. This points to a "comprehension bias" where the AI uses metaphors rhetorically without deeply "understanding" their conceptual boundaries.

Limitations and Future Work

The primary limitation is the scale of the corpus; 20 speeches provide a snapshot but not a comprehensive map of LLM linguistic behavior. The author also notes that "subjective bias" is always a risk in manual metaphor identification.

Future directions include: * Constructing larger, more diverse AI language corpora. * Developing high-performance automatic metaphor recognition models to reduce the burden of manual labeling in NLP. * Exploring the intersection of AI language with ethics, ideology, and aesthetics to better understand the societal impact of machine-generated discourse.

Summary updated

Abstract

Based on Conceptual Metaphor Theory (CMT), this paper creates a tiny corpus of ChatGPT-written speeches. Through employing a corpus-driven approach, this study analyzes the identification and utilization of conceptual metaphors in artificial intelligence (AI) languages. The AI demonstrated its capacity to utilize metaphors in the metaphoric corpora through the display of diversity, non-arbitrariness, repetition, and intersectionality in the selection of source domains. It often uses vocabulary combinations with clear similarities to establish metaphorical meaning. In the literal sense, the outcomes of metaphor identification by artificial intelligence differ significantly from those of humans. Therefore, there is a need to develop advanced automatic models for identifying metaphors in order to enhance the precision of metaphor identification consistently.
BibTeX
@article{yang2023corpus,
  author = {Yang Yang},
  title = {{Corpus-Driven Analysis of Conceptual Metaphor in Artificial Intelligence Language: A Sample of ChatGPT-Written Speeches}},
  journal = {Journal of Contemporary Educational Research},
  year = {2023},
  doi = {10.26689/jcer.v7i12.5713},
}
Figure