Abstract
Cross-lingual Natural Language Inference (XNLI) presents a significant challenge in transferring logical reasoning across diverse languages. We present the Translation-based Matching Adversarial Network (TMAN), a framework designed to extract language-invariant features through adversarial training. TMAN utilizes a combination of source and translated target language data, processed in mixed mini-batches, to train a feature generator that competes against a language discriminator. This process forces the model to prioritize semantic logic over language-specific markers. Evaluated on the XNLI benchmark across 15 languages, TMAN consistently improves performance over baseline models, including mBERT and XLM, demonstrating its effectiveness in bridging the language gap for complex inference tasks.
Introduction
Natural Language Inference (NLI) is a core task in natural language understanding, requiring models to determine if a premise entails, contradicts, or is neutral toward a hypothesis. Extending this to cross-lingual scenarios (XNLI) is difficult because models fine-tuned on a high-resource language like English often fail to generalize their reasoning to low-resource target languages. Existing approaches typically rely on parallel corpora or large-scale pre-trained models, but these often struggle with language-specific overfitting. TMAN addresses these limitations by employing an adversarial objective that encourages the learning of language-agnostic representations.
Methodology
The TMAN framework integrates data augmentation with an adversarial training architecture to ensure robust cross-lingual transfer.
Data Augmentation and Mixed-Language Training
TMAN begins by translating source language training data into the target language. Rather than treating these as separate datasets, TMAN merges and shuffles them. A critical aspect of the training process is the use of mixed-language mini-batches, which forces the model to align features from different languages in real-time.
Adversarial Architecture
The model architecture consists of three main components:
- Feature Generator ($G$): An NLI model (such as BERT, XLM, or ESIM) that extracts a matching representation from the premise-hypothesis pair.
- NLI Discriminator ($Y$): A classifier that predicts the inference label.
- Language Discriminator ($D$): A classifier that attempts to distinguish between the source and target languages.
The Feature Generator is trained with a dual objective: to minimize the NLI classification loss while maximizing the loss of the Language Discriminator. This adversarial "tug-of-war" ensures that the generated features are optimized for logical inference but contain no discernible language-specific information.
Transformer Enhancement Layer
To further refine the performance of pre-trained models like BERT and XLM, TMAN introduces an additional Transformer encoder layer. This layer acts as a specialized task-specific tuner, adapting general contextual embeddings to the specific requirements of sentence-pair matching logic.
Experimental Results
The TMAN framework was evaluated on the XNLI benchmark, which includes 15 languages ranging from high-resource (e.g., English, Spanish) to low-resource (e.g., Swahili, Urdu).
Performance Gains
TMAN consistently outperformed vanilla baseline models:
- TMAN-XLM: Achieved 77.2% average accuracy, a 0.5% absolute gain over the original XLM.
- TMAN-BERT: Reached 73.7% average accuracy, a 1.7% improvement over multilingual BERT.
- TMAN-ESIM: Demonstrated a 1.7% boost for the classical ESIM architecture.
Analysis of Language Invariance
Ablation studies and visualization techniques confirmed that the adversarial objective successfully produces language-invariant features. By "confusing" the language discriminator, the model focuses exclusively on the semantic relationship between sentences. The framework also showed resilience to translation errors, maintaining high performance even in languages with lower-quality machine translation.
Conclusion and Future Work
TMAN provides an effective solution for cross-lingual NLI by forcing models to learn semantic logic independent of language. Through mixed-batch training and adversarial feature extraction, it achieves state-of-the-art results on the XNLI benchmark. Future research will focus on extending TMAN to multi-language training scenarios and incorporating external semantic resources like BabelNet to further improve cross-lingual reasoning capabilities.
Summary updated