ViCGCN: Graph Convolutional Network with Contextualized Language Models for Social Media Mining in Vietnamese

Chau-Thang Phan, Quoc-Nam Nguyen, Chi-Thanh Dang, Trong-Hop Do, Kiet Van NguyenarXiv 2023 · DOI · arXiv

Abstract

Social media mining in Vietnamese presents unique challenges due to linguistic nuances like teencode, slang, and sarcasm. This paper introduces ViCGCN, a hybrid model that leverages the contextual power of PhoBERT and the structural intelligence of Graph Convolutional Networks (GCN). By modeling global word-document relationships, ViCGCN effectively mitigates noise and data imbalance, achieving state-of-the-art results across emotion recognition, toxicity detection, and sentiment analysis tasks.

Introduction

Social media text is notoriously difficult for Natural Language Processing (NLP) systems. In Vietnamese, this difficulty is compounded by the frequent use of abbreviations, intentional misspellings (teencode), and sarcasm. While Transformer-based models like BERT excel at capturing local context within a sentence, they often struggle with the "noisy" and "imbalanced" nature of social media datasets. For instance, rare "toxic" classes are hard to learn when drowned out by neutral comments.

ViCGCN addresses these limitations by incorporating global structure. By viewing the entire dataset as a graph, the model can infer the meaning of noisy or rare tokens based on their relationships with other words and documents across the corpus.

Methodology: The ViCGCN Framework

The ViCGCN architecture integrates two complementary components to capture both local semantics and global structural information.

The overall architecture of ViCGCN, integrating PhoBERT for contextual embeddings and GCN for global structural analysis.

1. PhoBERT (Contextual Layer)

The model first utilizes PhoBERT, a state-of-the-art RoBERTa-based model pre-trained on Vietnamese text. PhoBERT generates high-quality contextualized word embeddings, capturing the specific meaning of words within their immediate sentence context.

2. Graph Convolutional Network (Structural Layer)

To capture global context, ViCGCN constructs a large, heterogeneous graph. In this graph, both individual words and entire documents are represented as nodes.

  • Word-Word Edges: These are weighted using Positive Point-wise Mutual Information (PPMI), reflecting how frequently words co-occur across the entire dataset.
  • Word-Document Edges: These are weighted using Term Frequency-Inverse Document Frequency (TF-IDF) scores.
A representation of the heterogeneous graph where words and documents are nodes, facilitating classification through global relationships.

3. Joint Training and Fusion

The final classification is a weighted combination of the outputs from the PhoBERT branch and the GCN branch. A hyperparameter $\lambda$ (typically optimized between 0.6 and 0.8) balances the contribution of the two branches, allowing the model to prioritize global structural patterns over local sequence data when necessary.

Experimental Evaluation

The model was evaluated on three benchmark Vietnamese datasets: UIT-VSMEC (Emotion), UIT-ViCTSD (Toxicity), and UIT-VSFC (Sentiment).

Key Results

ViCGCN consistently outperformed 13 baseline models, including standard PhoBERT and multilingual BERT.

  • Emotion Recognition: 6.21% improvement in F1-score over the best contextualized models.
  • Toxicity Analysis: 4.61% improvement, reaching a Macro F1-score of 85.88%.
  • Sentiment Analysis: 2.63% improvement.

Discussion and Insights

The primary advantage of ViCGCN lies in the GCN's role as a regularizer. If a word is misspelled in one instance, the graph connects it to other instances of the correct spelling or related terms, allowing the model to "recover" the intended meaning. Ablation studies confirmed that the synergy between PhoBERT (semantic "meat") and GCN (structural "skeleton") is crucial for performance.

However, challenges remain. Error analysis shows that sarcasm and irony are still difficult to detect, as the model may take literal meanings at face value. Furthermore, while GCNs improve accuracy, the "black box" nature of graph nodes limits interpretability.

Conclusion

ViCGCN represents a significant step forward for Vietnamese social media mining. By combining local contextual embeddings with global graph-based relationships, it effectively tames the noise inherent in social platforms. Future work will explore Graph Attention Networks (GATs) for better explainability and automated text normalization to further enhance robustness.

Summary updated

BibTeX
@misc{phan2023vicgcn,
  author = {Chau-Thang Phan and Quoc-Nam Nguyen and Chi-Thanh Dang and Trong-Hop Do and Kiet Van Nguyen},
  title = {{ViCGCN: Graph Convolutional Network with Contextualized Language Models for Social Media Mining in Vietnamese}},
  year = {2023},
  eprint = {2309.02902},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.02902},
}
Figure