Abstract
Cross-lingual Natural Language Understanding (NLU) often relies on expensive parallel corpora or shallow unsupervised augmentation techniques that risk breaking sentence semantics. We present UniPSDA, an unsupervised framework that leverages a hierarchical "Domino" clustering process to discover semantic equivalents across languages. By targeting core Subject-Verb-Object (SVO) structures for replacement and employing Optimal Transport for de-biasing, UniPSDA significantly improves zero-shot performance across diverse NLU tasks without requiring human-annotated parallel data.
Introduction
The disparity in NLU performance between resource-rich languages like English and resource-scarce languages remains a significant challenge. While cross-lingual representation learning aims to bridge this gap, existing methods are often bottlenecked by the requirement for supervised parallel corpora. Unsupervised alternatives, such as Easy Data Augmentation (EDA), typically rely on "shallow" operations like random word deletion or replacement. These methods frequently disrupt the logical flow and global context of sentences.
UniPSDA addresses these limitations by introducing "deep" semantic augmentation. It identifies crucial sentence components and replaces them with semantically similar words discovered through an unsupervised hierarchical process, ensuring that the augmented data remains linguistically coherent and contextually relevant.
Methodology
The UniPSDA framework is built upon two primary pillars: the Domino Unsupervised Cluster and Pseudo Semantic Data Augmentation (PSDA).
Domino Unsupervised Cluster
Rather than performing a flat search for synonyms, UniPSDA utilizes a three-stage hierarchical clustering process termed "Domino Clustering":
- Single Language Clustering: Words are clustered within a single language using m-BERT embeddings and Part-of-Speech (POS) tags.
- Language Family Aggregation: These clusters are aggregated across specific language families (e.g., Romance languages) to capture shared semantic structures.
- Multi-Language Vocabulary: Finally, clusters are formed across multiple language families to create a global, unsupervised multi-language vocabulary.
Pseudo Semantic Data Augmentation (PSDA)
The augmentation process targets the structural "anchors" of a sentence—the Subject, Verb, and Object (SVO). Using syntactic parsing, the model identifies these components and replaces them with words from the corresponding Domino clusters, which may originate from different languages.
To maintain internal representation stability, UniPSDA employs Optimal Transport (OT) via the Sinkhorn algorithm. This de-biasing step minimizes the distance between original and augmented embeddings, further refined by Eigenvector and Distance Shrinkage techniques to ensure efficient and stable learning.
Experiments and Results
UniPSDA was evaluated on four diverse benchmarks: MLDoc (text classification), OpeNER (sentiment analysis), ACE2005 (information extraction), and BiPaR (reading comprehension).
The framework consistently outperformed strong baselines, including m-BERT and XLM-R. In text classification (MLDoc), UniPSDA achieved an average accuracy of 79.3%, compared to m-BERT's 71.2%. For sentiment analysis (OpeNER), it reached a new state-of-the-art macro F1 of 86.0%. Ablation studies confirmed the importance of the hierarchical clustering, as removing the Domino module led to a 4.4% drop in accuracy.
Conclusion
UniPSDA demonstrates that deep, structure-aware unsupervised augmentation can effectively bridge the cross-lingual gap in NLU. By focusing on SVO components and utilizing hierarchical semantic discovery, the model enriches training data while preserving meaning. Future work may explore extending this approach to generative tasks or integrating it into the pre-training phase of large-scale multilingual models.
Summary updated