Replacing Linguists with Dummies: A Serious Need for Trivial Baselines in Multi-Task Neural Machine Translation

Ronald A. Cardenas Acosta, Daniel Kondratyuk, Ronald Cardenas, Ondrej BojarPrague Bull. Math. Linguistics 2019

Introduction

In the pursuit of human-like Machine Translation (MT), researchers frequently employ Multi-Task Learning (MTL) to imbue models with linguistic "intuition." A common approach involves training a model to simultaneously translate text and predict syntactic structures, such as Combinatory Categorial Grammar (CCG) supertags. While this often yields performance gains, it remains unclear whether these improvements stem from genuine syntactic understanding or secondary effects like regularization. This study investigates this phenomenon by introducing "trivial baselines"—replacing expert linguistic knowledge with random noise—to determine the true source of these gains.

Methodology

The researchers evaluated two primary neural architectures: the Recurrent Neural Network-based Seq2Seq and the Transformer. They explored two methods for coupling translation with a secondary tagging task:

  1. Multi-Decoder: A shared encoder with separate decoders for words and tags.
  2. Interleaved: A single decoder that predicts tags and words in a single alternating sequence.

The core of the experiment lies in the comparison of three tagging schemes:

  • CCG Supertags: Expert linguistic data representing syntactic roles.
  • Random Tags: Words assigned random IDs (0–499), serving as a "dummy" baseline.
  • Same Tags: Every word assigned the same tag, serving as a control.

Experiments were conducted on the WMT 2016 English-German corpus (~4.5M sentence pairs), with CCG tags generated via the EasySRL parser.

Results and Discussion

The results challenge the assumption that syntactic knowledge is the primary driver of MTL improvements. As shown in the training dynamics, models augmented with random tags performed remarkably similarly to those using expert CCG tags across various architectures.

Training dynamics showing BLEU scores for different tag schemes (Baseline, CCG, Random, Same) across architectures and coupling methods.

Quantitatively, the gap between "real knowledge" and "random noise" was often negligible. In the Seq2Seq Interleaved setup, random tags actually outperformed CCG tags (23.54 vs 22.96 BLEU). The "Same Tag" control consistently failed, indicating that the model requires a task of sufficient complexity to benefit from the multi-task setup, regardless of the task's semantic meaning.

Final BLEU scores comparing baseline models to those augmented with CCG, Random, and Same tag schemes.

These findings suggest that the secondary task likely acts as a regularizer, preventing overfitting and improving generalization. Furthermore, in interleaved setups, the performance boost may be partially attributed to increased "effective depth," as the decoder performs additional computation steps per word produced.

Conclusion

This research highlights a critical need for more rigorous baselines in NLP. While syntax-based MTL provides clear benefits, these benefits can often be replicated with random noise. The authors advocate for the standard inclusion of "dummy" baselines in future research to distinguish between genuine linguistic representation and simple architectural or regularization effects. Future work should explore these dynamics across more diverse language pairs and with higher-quality "gold" linguistic annotations.

Summary updated

BibTeX
@article{acosta2019replacing,
  author = {Ronald A. Cardenas Acosta and Daniel Kondratyuk and Ronald Cardenas and Ondrej Bojar},
  title = {{Replacing Linguists with Dummies: A Serious Need for Trivial Baselines in Multi-Task Neural Machine Translation}},
  journal = {Prague Bull. Math. Linguistics},
  year = {2019},
}
Figure