SEQUENCE MIXUP FOR ZERO-SHOT CROSS-LINGUAL PART-OF-SPEECH TAGGING

Ramit Sawhney, Megh ThakkarICLR 2024 · OpenReview

TL;DR

This paper demonstrates that Sequence Mixup—an interpolative data augmentation technique—significantly boosts the performance of models in zero-shot cross-lingual part-of-speech (POS) tagging. By blending hidden representations of source language sentences during training, the model generalizes better to unseen target languages, especially when the languages are structurally dissimilar.

What Problem Does This Paper Address?

Part-of-speech (POS) tagging is a fundamental NLP task, but many of the world's languages lack the large, annotated datasets required to train high-performing neural models. Cross-lingual transfer learning attempts to solve this by training a model on a "source" language (like English or French) and applying it to a "target" language (like Urdu or Marathi).

However, models often overfit the specific syntactic patterns of the source language, leading to poor performance on target languages with different structures. The authors investigate whether "Mixup"—a technique originally popular in computer vision—can act as a regularizer to help models capture more universal linguistic features and improve zero-shot transfer.

Methods

The authors employ Sequence Mixup (Seq. Mixup). Unlike standard data augmentation that might swap words, Seq. Mixup operates in the latent space (the hidden layers of the model).

  1. Interpolation: Given two sequences from the source language, the model passes them through several encoder layers (using mBERT as the backbone).
  2. Mixing: At a specific hidden layer $k$, the representations of the two sequences are linearly combined using a ratio $\lambda$ (sampled from a Beta distribution).
  3. Label Mixing: The corresponding POS labels are also mixed using the same ratio.
  4. Training: The model continues processing this "synthetic" blended representation through the remaining layers.

This process forces the model to learn smoother decision boundaries, preventing it from becoming too "certain" about source-specific patterns and encouraging it to find features that might be shared across languages.

Datasets

The study uses the Universal Dependencies (UD) dataset, a gold standard for cross-lingual syntactic annotation. They evaluate their approach across ten diverse languages: Arabic (ar), Dutch (nl), French (fr), German (de), Hindi (hi), Indonesian (id), Italian (it), Marathi (mr), Vietnamese (vi), and Urdu (ur).

For each experiment, they use a modest training set of 800 sentences from the source language to simulate a constrained resource environment.

Key Insights and Ideas

  • Similarity Matters: The authors calculated a "dataset-level cosine similarity" between languages based on sentence embeddings. They found that Seq. Mixup provides the most significant gains when transferring between dissimilar languages (e.g., Hindi to Italian).
  • Mixup as a Bridge: By creating "virtual" training samples that exist between real data points in the latent space, Mixup helps the model explore linguistic structures it might not encounter in a standard source dataset, effectively "filling the gaps" between languages.
  • The Danger of Multi-Source Conflict: While mixing a single source language helps, training on a combination of very different source languages (like Arabic + French) to target a third (Italian) actually hurt performance. This suggests that "negative transfer" can occur if the interpolated representations become too noisy or linguistically incoherent.

Key Results

Single-Source Transfer Gains

In single-source transfer, Seq. Mixup consistently beat the standard mBERT baseline. The most impressive jumps occurred in low-similarity language pairs. For example, transferring from Hindi to Italian saw an F1-score improvement from 58.63 to 63.71 (+5.08 points). Even in higher similarity pairs, like Urdu to Arabic, scores improved from 44.61 to 47.38.

Single-source transfer results comparing mBERT and Sequence Mixup

Multi-Language Success and Failure

When source languages were similar to each other and the target (e.g., German + French to Dutch), the dual-source model outperformed the single-source model (86.11 vs 85.48 F1). However, combining structurally distant languages proved detrimental. Mixing Arabic and French to target Italian caused a massive drop in performance, plummeting to an F1 score of 28.00.

Dual-source transfer results showing the impact of combining source languages

Limitations and Future Work

  • Scale: As a "Tiny Paper," the experiments used relatively small subsets of data (800 sentences). It remains to be seen if these gains hold up with much larger datasets or more complex architectures.
  • Task Specificity: The study focused exclusively on POS tagging. Future work could explore if Sequence Mixup helps in more complex structural tasks like dependency parsing or semantic role labeling.
  • Language Selection: While the ten languages are diverse, the "negative transfer" results in the multi-source experiments suggest we need better heuristics for choosing which languages to "mix" to avoid confusing the model.

Summary updated

Abstract

There have been efforts in cross-lingual transfer learning for various tasks. We present an approach utilizing an interpolative data augmentation method, Mixup, to improve the generalizability of models for part-of-speech tagging trained on a source language, improving its performance on unseen target languages. Through experiments on ten languages with diverse structures and language roots, we put forward its applicability for downstream zero-shot cross-lingual tasks.
BibTeX
@inproceedings{sawhney2024sequence,
  author = {Ramit Sawhney and Megh Thakkar},
  title = {{SEQUENCE MIXUP FOR ZERO-SHOT CROSS-LINGUAL PART-OF-SPEECH TAGGING}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=H6gQ9iOS1X},
}
Figure