A Multi-Modal Knowledge Graph for Classical Chinese Poetry

Yuqing Li, Yuxin Zhang, Bin Wu, Ji-Rong Wen, Ruihua Song, Ting Bai

Introduction

Classical Chinese poetry represents a pinnacle of cultural expression, yet it remains largely inaccessible to modern readers and automated systems due to a significant "semantic gap." Over three millennia, the meanings of Chinese characters have evolved; for example, wulai (无赖) shifted from meaning "playful" in ancient verses to "rascally" in modern usage. Existing multi-modal models like CLIP, trained on contemporary web data, often fail to capture these archaic nuances and metaphors. To bridge this gap, researchers have introduced the Multi-modal Knowledge Graph (PKG) and a specialized pre-training model, PKG-Bert.

The Poetry Knowledge Graph (PKG) and PKG-Bert

The proposed solution involves two core components: a massive multi-modal knowledge graph and a transformer-based model designed to leverage it.

1. The PKG Architecture

The PKG was constructed by extracting words from nearly a million poems. To address semantic drift, these words were linked to sememes—the smallest units of semantic meaning—using HowNet. Each text node was then associated with high-definition images retrieved via the BriVL model. The resulting graph contains over 111,000 text nodes and 96,000 image nodes, connected by 1.3 million edges.

2. PKG-Bert Model

PKG-Bert is a Transformer-based model that integrates visual knowledge directly into its encoding process. Unlike standard text-only models, PKG-Bert's input layer combines token embeddings with associated image entity embeddings from the PKG.

Architecture of the PKG and the PKG-Bert pre-training framework.

The model is trained with dual objectives:

  • Masked Language Modeling (MLM): To capture linguistic patterns within the poetry corpus.
  • Space Distance Loss: To minimize the distance between poetic text representations and their corresponding visual representations, ensuring the model aligns ancient metaphors with visual concepts.

Key Insights: Knowledge as a Bridge

A pivotal innovation of this work is the use of sememes as an intermediate layer. By linking ancient vocabulary to universal semantic units, and subsequently to images, the model overcomes the scarcity of direct "ancient text-to-image" training pairs. This "visual injection" allows PKG-Bert to resolve semantic ambiguities by using the knowledge graph to "see" the intended context of a word.

An example of PKG-Bert processing a query, showing related words and corresponding image entities from the PKG.

Experimental Results

PKG-Bert demonstrates superior performance over general-purpose models like CLIP and ViLT in poetry-image retrieval tasks:

  • Poetry-to-Image Retrieval: PKG-Bert achieved a Hit Ratio (HR@5) of 0.46, representing a 77% improvement over the BriVL baseline.
  • Image-to-Poetry Retrieval: The model reached an HR@5 of 0.72, significantly outperforming ViLT (0.62) and CLIP (0.42).

Ablation studies confirmed that the inclusion of the PKG and visual information is critical; removing these components led to a substantial decline in retrieval accuracy.

Conclusion and Future Directions

While PKG-Bert marks a significant advancement, challenges remain. The model is currently domain-specific, and modern photography sometimes lacks the precise architectural or cultural details found in ancient verses. Future work may involve expanding the image dataset to include traditional Chinese paintings to better align with the aesthetic and historical context of the poetry.

Summary updated

Abstract

Classical Chinese poetry has a long history and is a precious cultural heritage of humankind. Displaying the classical Chinese poetry in a visual way, helps to cross cultural barriers in different countries, making it enjoyable for all the people. In this paper, we construct a multi-modal knowledge graph for classical Chinese poetry (PKG), in which the visual information of words in the poetry are incorporated. Then a multi-modal pre-training language model, PKG-Bert, is proposed to obtain the poetry representation with visual information, which bridges the semantic gap between different modalities. PKG-Bert achieves the state-of-the-art performance on the poetry-image retrieval task, showing the effectiveness of incorporating the multi-modal knowledge. The large-scale multi-modal knowledge graph of classical Chinese poetry will be released to promote the researches in classical Chinese culture area.
BibTeX
@inproceedings{li2022multi,
  author = {Yuqing Li and Yuxin Zhang and Bin Wu and Ji-Rong Wen and Ruihua Song and Ting Bai},
  title = {{A Multi-Modal Knowledge Graph for Classical Chinese Poetry}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.171},
}
Figure