Exploiting Cross-Lingual Subword Similarities in Low-Resource Document Classification

Mozhi Zhang, Yoshinari Fujinuma, Jordan Boyd-Graber

This paper presents a clever approach to a persistent problem in Natural Language Processing (NLP): how do we build effective text classifiers for languages that have almost no labeled data? The authors introduce CACO (Classification Aided by Convergent Orthography), a framework that bypasses the need for massive parallel corpora by looking at the building blocks of words—characters.

TL;DR

CACO leverages character-level similarities between related languages (like Spanish and Italian) to perform document classification in low-resource settings. By training a shared character-based embedder, the model can "guess" the meaning of words in a new language based on their orthographic similarity to a known language, proving significantly more data-efficient than traditional word-level methods.

What Problem Does This Paper Address?

Most Cross-Lingual Document Classification (CLDC) methods treat words as atomic units. To work, they require "bridges" between languages, such as high-coverage bilingual dictionaries, large parallel corpora (translated texts), or massive monolingual datasets to train Cross-Lingual Word Embeddings (CLWE).

For many of the world's languages—especially those in emergent crisis situations—these resources simply don't exist. However, languages don't exist in a vacuum. Related languages often share linguistic roots, cognates (e.g., religioso in both Spanish and Italian), and morphological patterns. The authors argue that these similarities are most visible at the subword or character level, a signal that traditional word-level models completely ignore.

Methods

The authors propose a bi-level neural architecture:

  1. Character-based Bi-LSTM Embedder: Instead of looking up a word in a fixed vocabulary, this component reads a word character-by-character. Because it uses a shared character vocabulary for both the source and target languages, a Spanish "a" is treated the same as an Italian "a." This allows the model to generalize from a known word to a similar-looking unknown word.
  2. Word-based DAN Classifier: A Deep Averaging Network (DAN) takes the word vectors produced by the embedder, averages them to represent the document, and passes them through non-linear layers to predict a label. The DAN is chosen for its simplicity and ability to generalize across varying word orders.

To boost performance, they introduce a multi-task objective that allows the model to incorporate whatever scraps of data might be available:

  • DICT: A small dictionary (as few as 100 pairs) to align embeddings of translation pairs.
  • MIM (Mimicking): Training the embedder to reconstruct pre-trained word embeddings in the source language.
  • Knowledge Distillation: Using a high-resource "teacher" model (e.g., in English) to guide the CACO model using small amounts of parallel text.

Datasets

The researchers evaluated CACO on two main benchmarks:

  • RCV2 (Reuters Multilingual Corpus): A topic identification dataset. They tested pairs within the North Germanic family (Danish, Swedish) and the Romance family (French, Italian, Spanish).
  • LORELEI: A "truly" low-resource dataset focused on disaster response (e.g., detecting "widespread crime"). They tested on Tigrinya and Amharic, two related Ethiopian languages with very limited digital resources.

Key Insights and Ideas

  • Orthography as a Bridge: The core "aha" moment is that shared script and similar spelling act as a natural, zero-cost alignment mechanism. If a model learns that "religioso" is a strong indicator for a "Social" topic in Spanish, it automatically knows the same for Italian without ever seeing an Italian document.
  • Data Efficiency: Character-level transfer is far more efficient than word-level transfer. CACO models using zero target language data often outperformed word-level models (CLWE) that were trained on millions of target language tokens.
  • Complementary Strengths: Character-level models are great at handling cognates and morphology, while word-level models (CLWE) are better at capturing semantic relationships between words that look different. Combining them (COM model) yielded the best results.
  • The Asymmetry of Transfer: Interestingly, transferring from Romance to Germanic languages worked better than the reverse. The authors suspect this is due to differences in how these language families handle morphology (word formation).

Key Results

  • Zero-Shot Success: On the RCV2 dataset, the basic CACO model (SRC) achieved an average accuracy of 50.0% across eight language pairs without using any target language data. This is competitive with a high-resource CLWE baseline (51.6%) that requires massive target corpora.
  • The Power of a Small Dictionary: Adding just 100 translation pairs (DICT) boosted CACO’s accuracy to 55.7%, significantly outperforming the baselines.
  • Best of Both Worlds: When CACO was combined with CLWE (the COM model), accuracy jumped to 64.5%, showing that subword and word-level features provide non-redundant information.
  • Low-Resource Reality: On the difficult LORELEI disaster response task, CACO was only slightly behind the CLWE baseline, despite the baseline having access to much larger external resources.

Limitations and Future Work

  • Script Constraints: The current framework relies on shared orthography. It wouldn't work directly between English (Latin script) and Russian (Cyrillic) without a pre-processing step like transliteration.
  • Language Distance: The effectiveness of CACO drops as languages become less related. Transferring from Danish to Spanish was significantly harder than transferring from Danish to Swedish.
  • Morphological Study: The authors suggest that a more systematic study of how specific morphological differences (like prefixing vs. suffixing) affect character-level transfer is a necessary next step.

Summary updated

Abstract

Text classification must sometimes be applied in a low-resource language with no labeled training data. However, training data may be available in a related language. We investigate whether character-level knowledge transfer from a related language helps text classification. We present a cross-lingual document classification framework (CACO) that exploits cross-lingual subword similarity by jointly training a character-based embedder and a word-based classifier. The embedder derives vector representations for input words from their written forms, and the classifier makes predictions based on the word vectors. We use a joint character representation for both the source language and the target language, which allows the embedder to generalize knowledge about source language words to target language words with similar forms. We propose a multi-task objective that can further improve the model if additional cross-lingual or monolingual resources are available. Experiments confirm that character-level knowledge transfer is more data-efficient than word-level transfer between related languages.
BibTeX
@inproceedings{zhang2020exploiting,
  author = {Mozhi Zhang and Yoshinari Fujinuma and Jordan Boyd-Graber},
  title = {{Exploiting Cross-Lingual Subword Similarities in Low-Resource Document Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6500},
}
Figure