Cantonese Natural Language Processing in the Transformers Era

Rong Xiang, Ming Liao, Jing Li2024 · DOI

Abstract

Despite being spoken by a large population of speakers worldwide, Cantonese is underresourced in terms of the data scale and diversity compared to other major languages. This limitation has excluded it from the current "pretraining and fine-tuning" paradigm that is dominated by Transformer architectures. In this paper, we provide a comprehensive review on the existing resources and methodologies for Cantonese Natural Language Processing, covering the recent progress in language understanding, text generation and development of language models. We finally discuss two aspects of the Cantonese language that could make it potentially challenging even for state-of-the-art architectures: colloquialism and multilinguality.

Introduction

The "pre-training and fine-tuning" paradigm has revolutionized natural language processing (NLP) for major languages like English and Standard Chinese (SCN). However, Cantonese exists in a state of digital neglect. Despite having over 85 million speakers globally, it remains a "low-resource" language in the research community, largely left behind by the Transformer-driven revolution.

A survey of the ACL Anthology reveals a staggering disparity in research attention. While English and Mandarin boast nearly 10,000 and 5,000 publications respectively, Cantonese is represented by a mere 61 papers. This lack of data and attention means that state-of-the-art (SOTA) models often fail to capture Cantonese’s distinct grammar, vocabulary, and its inherent tendency for code-switching.

Figure 1: Comparison of publication counts across different languages in the ACL Anthology, highlighting the significant resource gap for Cantonese.

Methodology

This study employs a two-pronged approach: a comprehensive literature review and an empirical pilot study.

  1. Literature Review: Existing Cantonese corpora and benchmarks are categorized, tracking the evolution from early linguistic dictionaries to modern social media datasets.
  2. Surprisal Analysis: To quantify the linguistic distance between Cantonese and Standard Chinese, the authors utilized surprisal scores—the negative log probability of a word given its context. Two SCN-trained models (BERT-CKIP and RoBERTa-HFL) were tested on Cantonese sentences and their translations.
  3. Language Identification: Using the fastlangid tool, the authors analyzed the ratio of Cantonese, SCN, and English in popular Hong Kong forums to measure the extent of multilinguality.

Data Resources

The paper identifies several key Cantonese resources, ranging from historical linguistic corpora like CANCORP and HKCAC to modern benchmarks such as KddRES (food-ordering dialogue) and OpenRice (sentiment analysis). For the empirical pilot study, the authors curated a significant dataset by scraping approximately 1.1 GB of text (0.924 billion tokens) from three major Hong Kong-based platforms: DISCUSS, LIHKG, and OpenRice.

Figure 6: Statistics of the datasets scraped from DISCUSS, LIHKG, and OpenRice for the pilot study, showing token counts and text sizes.

Key Insights and Results

The study highlights several critical challenges for Cantonese NLP:

  • The Surprisal Gap: Using the RoBERTa-HFL model, original Cantonese sentences yielded a surprisal score of 4.39, significantly higher than the 2.89 score for their Traditional Chinese translations. This confirms that SCN models are poorly calibrated for Cantonese-specific syntax and vocabulary.
  • Vocabulary Overlap: Only 65.5% to 69.1% of characters overlap between Cantonese sentences and their Mandarin translations, meaning over 30% of characters in typical Cantonese text are used differently or are "invisible" to standard models.
  • The Code-Switching Reality: Cantonese NLP is inherently multilingual. On platforms like LIHKG, Cantonese accounts for only about 40% of the text, with the remainder being a fluid mix of SCN and English. Models must be robust to "D" being used for the possessive particle di1 (啲) or English nouns being dropped into Cantonese syntax.

Conclusion and Future Directions

The "vitality" of Cantonese as a spoken language is not reflected in its digital support. Standard Chinese models are insufficient for capturing the nuances of colloquial Cantonese and its complex code-switching patterns.

Future research should focus on developing models that integrate spelling correction and dialect identification to handle irregular social media input. Furthermore, expanding cross-lingual transfer learning—where Mandarin models are specifically adapted to Cantonese—and creating more open-license datasets will be essential for bridging the gap in the Transformers era.

Summary updated

Abstract

Despite being spoken by a large population of speakers worldwide, Cantonese is underresourced in terms of the data scale and diversity compared to other major languages.This limitation has excluded it from the current "pretraining and fine-tuning" paradigm that is dominated by Transformer architectures.In this paper, we provide a comprehensive review on the existing resources and methodologies for Cantonese Natural Language Processing, covering the recent progress in language understanding, text generation and development of language models.We finally discuss two aspects of the Cantonese language that could make it potentially challenging even for state-of-the-art architectures: colloquialism and multilinguality.
BibTeX
@misc{xiang2024cantonese,
  author = {Rong Xiang and Ming Liao and Jing Li},
  title = {{Cantonese Natural Language Processing in the Transformers Era}},
  year = {2024},
  doi = {10.18653/v1/2024.sighan-1.8},
}
Figure