UniPSDA: Unsupervised Pseudo Semantic Data Augmentation for Zero-Shot Cross-Lingual Natural Language Understanding

Dongyang Li, Taolin Zhang, Jiali Deng, Longtao Huang, Chengyu Wang, Xiaofeng He, Hui Xue

This paper introduces UniPSDA, a novel unsupervised framework for cross-lingual data augmentation. By using a hierarchical clustering approach to find semantically similar words across languages and a de-biased replacement strategy, the authors enable models to learn deep linguistic connections without requiring expensive human-annotated parallel corpora.

TL;DR

UniPSDA improves zero-shot cross-lingual understanding by replacing key sentence components (Subject-Verb-Object) with semantically similar "pseudo-semantic" embeddings discovered through a three-stage hierarchical clustering process. It outperforms existing unsupervised methods by maintaining sentential coherence and using optimal transport to align augmented data with original contexts.

What Problem Does This Paper Address?

Cross-lingual Natural Language Understanding (NLU) aims to transfer knowledge from resource-rich languages (like English) to resource-scarce ones. Traditionally, this requires supervised parallel data (e.g., bilingual dictionaries), which is expensive to produce.

Existing unsupervised methods often rely on "shallow" techniques like Easy Data Augmentation (EDA), which includes random word deletion or replacement. As shown in the figure below, these shallow methods risk destroying the sentence's meaning or grammatical structure. The challenge is to create an unsupervised method that performs "deep" semantic augmentation—swapping words for cross-lingual equivalents that actually make sense in context.

Figure 1: Examples of previous data augmentation techniques, including supervised methods that rely on parallel data, and unsupervised methods which carry the risk of losing sentential semantic coherence.

Methods

The UniPSDA framework consists of two primary modules:

1. Domino Unsupervised Cluster

To find words that mean the same thing across different languages, the authors propose a "Domino" clustering process. Instead of a single massive cluster, they use three sequential stages:

  • Single Language Stage: Groups words within one language based on their Pre-trained Language Model (PLM) embeddings and Part-of-Speech (POS) tags (e.g., ensuring a verb is only compared to other verbs).
  • Language Family Stage: Aggregates clusters from related languages (e.g., Romance languages) to learn regional linguistic relations.
  • Multi-Language Stage: Finally, it clusters across all language families to find global semantic equivalents.

2. Pseudo Semantic Data Augmentation

Once clusters are formed, the model identifies the Subject-Verb-Object (SVO) components of a source sentence using syntactic parsing. It then replaces the hidden states (embeddings) of these SVO words with randomly selected "candidate" embeddings from the same multi-lingual cluster.

To ensure these new "foreign" embeddings don't confuse the model, the authors introduce De-biasing Optimal Transport (OT) Affinity Regularization. This uses the Wasserstein distance and singular value decomposition (SVD) to minimize the representational gap between the original and the augmented sentence, ensuring the "pseudo-semantic" injection remains contextually stable.

Figure 2: Model Overview of UniPSDA. (Best viewed in color)

Datasets

The authors evaluated UniPSDA on several standard NLU benchmarks across diverse languages:

  • MLDoc: Cross-lingual text classification (8 languages).
  • OpeNER: Sentiment analysis (Catalan and Basque).
  • ACE2005: Information extraction/Relation extraction (English, Chinese, Arabic).
  • BiPaR: Cross-lingual question answering (English and Chinese).

Key Insights and Ideas

  • Hierarchy Matters: Flat clustering (like standard K-Means) often fails to capture the nuances of cross-lingual semantics. The "Domino" approach effectively bridges the gap from local dialects to global meanings.
  • SVO is the Core: Replacing the Subject, Verb, and Object is more effective than random replacement because these components carry the primary semantic weight of a sentence.
  • Embeddings over Strings: By replacing hidden states (pseudo-semantics) rather than raw text strings, the model can leverage the rich, context-aware knowledge already present in PLMs like m-BERT.
  • Clustering Quality: Visualizing the embeddings with t-SNE (see below) shows that the Domino Unsupervised Cluster produces much tighter, more meaningful groupings than naive K-Means.
t-SNE visualization comparing Domino Unsupervised Cluster (left) vs. K-Means (right). The Domino approach shows much clearer semantic separation.

Key Results

UniPSDA consistently improved performance over strong baselines like m-BERT and XLM-R:

  • Text Classification (MLDoc): Achieved an average accuracy of 79.3%, a significant jump over the 71.2% of base m-BERT. Notably, French accuracy improved by ~10% compared to previous unsupervised methods.
  • Sentiment Analysis: Reached a 86.0 F1 score, setting a new state-of-the-art for the OpeNER dataset.
  • Information Extraction: Improved the average F1 score on ACE2005 to 44.1, outperforming the previous best unsupervised method (DAP) by 0.3 points and m-BERT by 3.0 points.

Limitations and Future Work

The authors note that while UniPSDA is highly effective for an unsupervised method, it still falls slightly behind some supervised methods in specific tasks like English Question Answering. This is because supervised methods can leverage high-quality, human-aligned parallel data that UniPSDA explicitly avoids using.

Future work could involve expanding the "pseudo-semantic" replacement beyond the SVO structure to include more complex syntactic dependencies or exploring its application in generative cross-lingual tasks.

Summary updated

Abstract

Cross-lingual representation learning transfers knowledge from resource-rich data to resource-scarce ones to improve the semantic understanding abilities of different languages. However, previous works rely on shallow unsupervised data generated by token surface matching, regardless of the global context-aware semantics of the surrounding text tokens. In this paper, we propose an Unsupervised Pseudo Semantic Data Augmentation (UniPSDA) mechanism for cross-lingual natural language understanding to enrich the training data without human interventions. Specifically, to retrieve the tokens with similar meanings for the semantic data augmentation across different languages, we propose a sequential clustering process in 3 stages: within a single language, across multiple languages of a language family, and across languages from multiple language families. Meanwhile, considering the multi-lingual knowledge infusion with context-aware semantics while alleviating computation burden, we directly replace the key constituents of the sentences with the above-learned multi-lingual family knowledge, viewed as pseudo-semantic. The infusion process is further optimized via three de-biasing techniques without introducing any neural parameters. Extensive experiments demonstrate that our model consistently improves the performance on general zero-shot cross-lingual natural language understanding tasks, including sequence classification, information extraction, and question answering.
BibTeX
@inproceedings{li2024unipsda,
  author = {Dongyang Li and Taolin Zhang and Jiali Deng and Longtao Huang and Chengyu Wang and Xiaofeng He and Hui Xue},
  title = {{UniPSDA: Unsupervised Pseudo Semantic Data Augmentation for Zero-Shot Cross-Lingual Natural Language Understanding}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure