Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry

Bolei Ma, Yina Yao, Anna-Carolina Haensch

TL;DR

This paper evaluates six state-of-the-art LLMs on classical Chinese poetry generation and uncovers a critical "echo chamber" effect where LLMs systematically overrate flawed machine-generated poems. By introducing a hybrid three-step evaluation framework, the authors demonstrate that while LLMs excel at surface-level fluency, they share blind spots regarding strict structural rules and deep cultural nuances that only human experts can reliably detect.

What Problem Does This Paper Address?

Classical Chinese poetry, particularly Tang poetry, represents a pinnacle of literary tradition governed by strict tonal, metrical, and rhyming constraints alongside rich metaphorical imagery. Evaluating machine-generated poetry is notoriously difficult; traditional automated metrics like BLEU or ROUGE only capture surface-level text overlap and fail to judge aesthetic or structural quality.

While using LLMs as evaluators ("LLM-as-a-judge") has become a popular alternative, it introduces potential systematic biases. This paper addresses the lack of a rigorous, multi-dimensional understanding of how LLMs perform both as creators and judges in highly structured, culturally nuanced domains.

Methods

The authors propose a comprehensive Three-Step Evaluation Framework:

  1. Computational Feature Extraction: Objective metrics including TF-IDF for lexical fingerprinting, Shannon entropy to measure vocabulary diversity, and sentence embeddings to compute semantic similarity and line-to-line coherence.
  2. LLM-as-a-Judge Cross-Assessment: A $6 \times 6$ evaluation matrix where six open-source LLMs act as both generators and evaluators, scoring poems from 1 to 5 across five dimensions: Prosodic Adherence, Thematic Relevance, Emotional Consistency, Imagery & Structure, and Classical Language Authenticity.
  3. Human Expert Validation: Blind evaluation by professional scholars of classical Chinese literature to serve as the ultimate ground truth and calibrate the automated metrics.

The basic framework is illustrated below:

The basic framework of poetry generation and evaluation.

Datasets

The authors generated a controlled dataset of 15,000 poems (around 2,500 per model) across six open-source instruction-tuned LLMs: DeepSeek-V2-Lite-Chat, Qwen2.5-7B-Instruct, GLM-4-9B-Chat-HF, Mistral-7B-Instruct-v0.3, Baichuan2-7B-Chat, and Gemma-2-9B-it.

For the human validation phase, a stratified random sample of 150 anonymized poems (50 each from the top three peer-ranked models: Qwen, DeepSeek, and Gemma) was evaluated by experts.

Key Insights and Ideas

  • The Echo Chamber Effect: LLMs trained on overlapping datasets tend to validate each other's mistakes. They consistently overrate machine-generated poems that mimic statistical patterns but completely violate strict prosodic rules.
  • The Entropy Paradox: High lexical diversity (measured by Shannon entropy) does not equate to literary richness. Models like GLM and Mistral showed high entropy but failed to translate it into semantic nuance, while low-entropy models like Baichuan relied heavily on rigid, template-driven generation.
  • Cultural Blind Spots: While LLMs easily grasp simple canonical associations (e.g., Moon $\rightarrow$ Tranquility), they fail at deeper, structurally embedded cultural metaphors, completely missing the quintessential "willow-sadness" pairing (rooted in Chinese homophony where "willow" sounds like "staying").
  • Semantic Convergence: Despite using different vocabularies, LLMs exhibit strong semantic convergence ($>0.94$ similarity). They struggle to truly differentiate distinct historical poetic styles (e.g., Li Bai vs. Du Fu) in practice, trapping themselves in a shared semantic space.

The distribution of vocabulary preferences and entropy variations across models highlight these behavioral differences:

Figure 2: Top 10 keywords for all models, with highest TF-IDF scores marked red for each model.
Figure 3: Entropy scores across models.

Key Results

  • Human vs. AI Divergence: Human experts judged Qwen2.5-7B-Instruct as the clear winner with a mean score of 4.53, praising its near-perfect formal control and imagery.
  • Prosodic Failure Overlooked: Gemma-2-9B-it failed catastrophically on prosody according to humans (scoring a dismal 1.72 due to rule violations and random English word intrusions), yet the LLM judges completely missed these flaws, giving it high scores.
  • Statistical Misalignment: Spearman rank correlations between human and LLM judges were weak or even negative for structural dimensions like Prosodic Adherence ($\rho = -0.176$ for Gemma), proving that LLM ensembles cannot yet replace human experts for complex cultural validation.

Limitations and Future Work

  • Domain and Language Constraints: The study focuses exclusively on classical Chinese poetry, meaning the findings might not directly generalize to other languages or less constrained creative domains.
  • Prose-Biased Embeddings: The semantic similarity metrics relied on embedding models trained primarily on standard prose, which inherently struggles with highly metaphorical and compact poetic language.
  • Prompt Engineering Scope: Advanced prompting techniques like Chain-of-Thought or role-playing protocols were not exhaustively explored, which might otherwise improve poetic reasoning.
  • Safety Alignment Interference: The models' internal safety guardrails may inadvertently suppress certain historical themes or archaic expressions necessary for authentic classical poetry.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly applied to creative domains, yet their performance in classical Chinese poetry generation and evaluation remains poorly understood. We propose a three-step evaluation framework that combines computational metrics, LLM-as-a-judge assessment, and human expert validation. Using this framework, we evaluate six state-of-the-art LLMs across multiple dimensions of poetic quality, including themes, emotions, imagery, form, and style, in the context of Tang poetry generation. Our analysis reveals a critical "echo chamber" effect: LLMs systematically overrate machine-generated poems that mimic statistical patterns yet fail strict prosodic rules, diverging significantly from human expert judgments. These findings underscore the limitations of using LLMs as standalone evaluators for culturally complex tasks, highlighting the necessity of hybrid human-model validation frameworks.
BibTeX
@inproceedings{ma2026capabilities,
  author = {Bolei Ma and Yina Yao and Anna-Carolina Haensch},
  title = {{Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.836},
}
Figure