Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry

Xiaocong Du, Haoyu Pei, Haipeng ZhangIJCAI 2025 · DOI

Abstract

Classical Chinese poetry is a unique art form where meaning is conveyed not just through text, but through rhythmic recitation and evocative visual imagery. Standard NLP models often struggle with its extreme conciseness and archaic grammar. We propose a tri-modal framework that integrates textual, audio, and visual features to capture the "DNA" of ancient verse. By utilizing Large Language Models (LLMs) for translation, dialect-specific audio (e.g., Cantonese, Minnan) for rhythmic patterns, and Stable Diffusion for latent visual imagery, our approach significantly improves sentiment analysis. Experimental results on the FSPC and CCPD datasets show improvements of up to 2.51% in accuracy over previous state-of-the-art methods.

Introduction

Classical Chinese poetry is notoriously difficult for standard NLP models. It is characterized by extreme conciseness, archaic grammar, and a heavy reliance on "imagery" (visual metaphors) and "recitation" (rhythmic and tonal patterns). Most existing sentiment analysis tools treat these poems as flat text, ignoring two-thirds of their essential components: the visual and the rhythmic.

Ancient poems were often written to evoke specific paintings or scenes, and they were meant to be chanted or recited. Modern Mandarin often loses the tonal rhyme schemes that existed when these poems were written. To truly understand the emotion of a poem, a model needs to "see" the imagery and "hear" the rhythm, particularly through dialects that preserve ancient phonetic features better than modern Mandarin.

Examples of visual imagery and phonetic variations in classical poetry.

Methodology: The Tri-modal Framework

We propose a tri-modal framework that processes information through three distinct pipelines before fusion:

Textual Modality

Since classical Chinese is challenging even for modern speakers, we use a Large Language Model (GLM-4) to translate the poem into modern Chinese and provide an explanation. Both the original and translated versions are then embedded using GuwenBERT, a model pre-trained specifically on classical Chinese.

Audio Modality

To capture rhythm, we incorporate audio clips of characters across four dialects: Cantonese, Wu, Minnan, and Chaozhou. These dialects often retain phonetic features of ancient Chinese (like "entering tones") that Mandarin has lost. A cross-attention module fuses these dialects, allowing the model to focus on shared phonetic features while filtering out noise.

Visual Modality

Instead of searching for existing images, we use Stable Diffusion (the 'Taiyi' model) to generate latent visual features directly from the poem’s text. We utilize the intermediate "latent" representation rather than the final pixels to preserve high-level semantic imagery while avoiding the noise of image decoding.

The proposed tri-modal architecture for sentiment analysis of classical Chinese poetry.

The framework employs a two-stage training strategy. First, multimodal contrastive representation learning aligns the three modalities in a shared space. Second, these features are concatenated for the final sentiment classification task.

Key Insights and Results

The tri-modal approach consistently outperformed text-only and bi-modal baselines on two public benchmarks: the THU Fine-grained Sentimental Poetry Corpus (FSPC) and the Chinese Classical Poetry Dataset (CCPD).

  • Performance Gains: On the FSPC dataset, the model achieved 64.85% accuracy, a 2.51% improvement over the previous best-performing model (CGCLM). On CCPD, it reached a 95.84% Micro F1, a 1.63% improvement in Macro F1 over the previous SOTA.
  • Dialects as a Time Machine: Models trained on Cantonese audio performed better on poems written by authors from southern regions, while Mandarin-trained models fared better with northern poems, suggesting a "geographic synergy" in poetic sentiment.
  • Generative Imagery: Generating "latent" images via Stable Diffusion outperformed using real images found via search engines, indicating that text-to-image models capture the "mood" of a poem better than literal photographs.

Conclusion

By looking beyond the text and integrating the rhythmic and pictorial essence of ancient verse, our framework sets a new standard for sentiment analysis in classical Chinese literature. Future work will explore reconstructing ancient pronunciations (Middle Chinese) and moving toward deeper, poem-level audio integration to account for structural prosody.

Summary updated

BibTeX
@inproceedings{du2025picturized,
  author = {Xiaocong Du and Haoyu Pei and Haipeng Zhang},
  title = {{Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/310},
}
Figure