Abstract
Researchers from Soochow University have developed a novel multimodal approach to Chinese Aspect-Based Sentiment Classification (ABSC) by treating Chinese text as a logographic image. By combining BERT-based text representations with Vision Transformer (ViT) features extracted from character images—including "structural images" that encode dependency relations—their model significantly outperforms traditional text-only and previous glyph-aware methods.
Introduction
Aspect-Based Sentiment Classification (ABSC) aims to identify the sentiment polarity (positive/negative) toward specific "aspects" in a sentence (e.g., in "The dessert tastes delicious," the aspect is "dessert" and the sentiment is positive).
In Chinese, this task is particularly challenging for two reasons:
- Internal Semantics: Chinese is a logographic language. Characters are built from radicals that carry inherent meaning. For instance, characters with the "wood" (木) radical often relate to furniture, while those with the "heart" (心) radical often relate to emotions. Standard NLP models that treat characters as discrete IDs lose this visual semantic richness.
- External Relations: Identifying word boundaries and the syntactic links between an opinion word (like "delicious") and its target aspect ("dessert") is difficult in Chinese. Traditional graph-based methods to solve this can be computationally heavy and complex to implement.
Methodology
The authors propose a multimodal language model that processes both raw text and "logographic images."
Logographic Image Construction
- Sequence Images: Character sequences are converted into images using four different typefaces (Song, Kai, Cursive, and Traditional). This helps the model generalize across different writing styles and capture historical semantic nuances.
- Structural Images: This key innovation involves taking an aspect-oriented dependency tree and drawing it directly onto the logographic image. This provides a visual, spatial representation of syntactic relations.
Model Architecture
- Text Encoder: Uses BERT to obtain contextual embeddings of the review sentence.
- Image Encoder: Uses a Vision Transformer (ViT) to extract visual features from the logographic images.
- Multimodal Interaction: A cross-attention mechanism allows the model to align text tokens with specific regions of the logographic image, effectively "looking" at the character's structure while reading the text.
Experimental Setup
The model was evaluated on a new Chinese dataset crawled from Taobao.com, consisting of 10,800 reviews across three domains: Chandlery (Chan), Furniture (Furn), and Kitchen (Kith). The dataset is balanced between positive and negative polarities.
Analysis and Insights
- Radicals as Semantic Anchors: Morphological information (radicals) is a powerful signal for sentiment. For example, the character for "hurt" (痛) contains a radical (疒) associated with illness, which inherently leans negative.
- Typeface Diversity: Using multiple typefaces acts as data augmentation, helping the model capture "pictographic evidence" that might be obscured in simplified, modern fonts.
- Images as a Natural Graph: Instead of using complex Graph Neural Networks (GNNs), encoding relations into a "structural image" allows a standard Vision Transformer to learn syntactic connections more easily.
Results
The proposed model consistently outperformed strong baselines, including text-only models like BERT and previous glyph-aware models like ChineseBERT.
| Method | Chan (Acc) | Kith (Acc) | Furn (Acc) |
|---|---|---|---|
| TD-BERT | 81.56 | 88.93 | 87.72 |
| R-GAT (Graph-based) | 81.98 | 89.02 | 88.16 |
| ChineseBERT | 83.23 | 89.94 | 89.62 |
| Ours (Multimodal) | 85.72 | 91.57 | 91.35 |
Ablation studies showed that while text is the strongest single modality, adding the logographic vision component provides a significant boost (approx. 3-4% improvement over BERT alone).
Conclusion and Future Work
The authors identify several areas for future research, including the development of a more unified multimodal model to replace separate encoders, and addressing the computational overhead associated with high-resolution image generation and dependency tree calculation.
Summary updated