Translation-Based Matching Adversarial Network for Cross-Lingual Natural Language Inference

Kunxun Qi, Jianfeng Du

Abstract

Cross-lingual Natural Language Inference (XNLI) presents a significant challenge in transferring logical reasoning across diverse languages. We present the Translation-based Matching Adversarial Network (TMAN), a framework designed to extract language-invariant features through adversarial training. TMAN utilizes a combination of source and translated target language data, processed in mixed mini-batches, to train a feature generator that competes against a language discriminator. This process forces the model to prioritize semantic logic over language-specific markers. Evaluated on the XNLI benchmark across 15 languages, TMAN consistently improves performance over baseline models, including mBERT and XLM, demonstrating its effectiveness in bridging the language gap for complex inference tasks.

Introduction

Natural Language Inference (NLI) is a core task in natural language understanding, requiring models to determine if a premise entails, contradicts, or is neutral toward a hypothesis. Extending this to cross-lingual scenarios (XNLI) is difficult because models fine-tuned on a high-resource language like English often fail to generalize their reasoning to low-resource target languages. Existing approaches typically rely on parallel corpora or large-scale pre-trained models, but these often struggle with language-specific overfitting. TMAN addresses these limitations by employing an adversarial objective that encourages the learning of language-agnostic representations.

Methodology

The TMAN framework integrates data augmentation with an adversarial training architecture to ensure robust cross-lingual transfer.

Data Augmentation and Mixed-Language Training

TMAN begins by translating source language training data into the target language. Rather than treating these as separate datasets, TMAN merges and shuffles them. A critical aspect of the training process is the use of mixed-language mini-batches, which forces the model to align features from different languages in real-time.

The TMAN framework overview, illustrating the data preparation pipeline and the adversarial training architecture.

Adversarial Architecture

The model architecture consists of three main components:

  1. Feature Generator ($G$): An NLI model (such as BERT, XLM, or ESIM) that extracts a matching representation from the premise-hypothesis pair.
  2. NLI Discriminator ($Y$): A classifier that predicts the inference label.
  3. Language Discriminator ($D$): A classifier that attempts to distinguish between the source and target languages.

The Feature Generator is trained with a dual objective: to minimize the NLI classification loss while maximizing the loss of the Language Discriminator. This adversarial "tug-of-war" ensures that the generated features are optimized for logical inference but contain no discernible language-specific information.

Transformer Enhancement Layer

To further refine the performance of pre-trained models like BERT and XLM, TMAN introduces an additional Transformer encoder layer. This layer acts as a specialized task-specific tuner, adapting general contextual embeddings to the specific requirements of sentence-pair matching logic.

Architecture of the Transformer enhancement layer added on top of lexicon encoders like BERT or XLM.

Experimental Results

The TMAN framework was evaluated on the XNLI benchmark, which includes 15 languages ranging from high-resource (e.g., English, Spanish) to low-resource (e.g., Swahili, Urdu).

Performance Gains

TMAN consistently outperformed vanilla baseline models:

  • TMAN-XLM: Achieved 77.2% average accuracy, a 0.5% absolute gain over the original XLM.
  • TMAN-BERT: Reached 73.7% average accuracy, a 1.7% improvement over multilingual BERT.
  • TMAN-ESIM: Demonstrated a 1.7% boost for the classical ESIM architecture.

Analysis of Language Invariance

Ablation studies and visualization techniques confirmed that the adversarial objective successfully produces language-invariant features. By "confusing" the language discriminator, the model focuses exclusively on the semantic relationship between sentences. The framework also showed resilience to translation errors, maintaining high performance even in languages with lower-quality machine translation.

Conclusion and Future Work

TMAN provides an effective solution for cross-lingual NLI by forcing models to learn semantic logic independent of language. Through mixed-batch training and adversarial feature extraction, it achieves state-of-the-art results on the XNLI benchmark. Future research will focus on extending TMAN to multi-language training scenarios and incorporating external semantic resources like BabelNet to further improve cross-lingual reasoning capabilities.

Summary updated

Abstract

Cross-lingual natural language inference is a fundamental task in cross-lingual natural language understanding, widely addressed by neural models recently. Existing neural model based methods either align sentence embeddings between source and target languages, heavily relying on annotated parallel corpora, or exploit pre-trained cross-lingual language models that are fine-tuned on a single language and hard to transfer knowledge to another language. To resolve these limitations in existing methods, this paper proposes an adversarial training framework to enhance both pre-trained models and classical neural models for cross-lingual natural language inference. It trains on the union of data in the source language and data in the target language, learning language-invariant features to improve the inference performance. Experimental results on the XNLI benchmark demonstrate that three popular neural models enhanced by the proposed framework significantly outperform the original models.
BibTeX
@inproceedings{qi2020translation,
  author = {Kunxun Qi and Jianfeng Du},
  title = {{Translation-Based Matching Adversarial Network for Cross-Lingual Natural Language Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6387},
}
Figure