The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation

Evgeniia Tokarchuk, Vlad Niculae

Abstract

Continuous-Output Neural Machine Translation (CoNMT) typically relies on the assumption that target embeddings must be semantically structured to be effective. This research challenges this paradigm, demonstrating that static, random target embeddings can outperform laboriously pre-trained ones—especially on large-scale datasets and for rare vocabulary—by providing superior geometric dispersion in the vector space.

Introduction

Standard Neural Machine Translation (NMT) models treat language as a sequence of discrete choices, utilizing a large softmax layer to select tokens from a vocabulary. In contrast, Continuous-Output NMT (CoNMT) predicts a continuous vector and identifies the nearest word in the embedding space. While conventional wisdom suggests these embeddings should be semantically meaningful (e.g., "cat" and "kitten" being proximal), the authors hypothesize that such "clumping" may hinder the model's ability to distinguish between specific tokens, particularly rare ones.

Methodology

The study evaluates CoNMT models using a cosine distance objective. Three primary target embedding strategies are compared:

  1. Pre-trained: Embeddings extracted from a standard discrete NMT model.
  2. Random Uniform: Points sampled uniformly from the surface of a unit sphere.
  3. Random Hypercube: Points sampled from the corners of a hypercube.

These random embeddings are kept static during training. Additionally, a Combined Strategy is introduced, using a weighted average (90% pre-trained, 10% random) to leverage semantic structure for common words while utilizing random noise to disperse the representation of rare words.

Key Insights: Dispersion and Rare Words

The primary advantage of random embeddings is dispersion. In pre-trained spaces, rare words often collapse into the same regions as their frequent neighbors, leading to "tangled" representations. Random vectors, by nature, are more spread out, providing each word with a more distinct "territory" (Voronoi cell).

The Rare Word Advantage

Random embeddings significantly improve performance for rare tokens. Since random vectors are agnostic to semantics, they avoid the clumping effect that plagues pre-trained spaces, allowing for more accurate identification of low-frequency words.

F-measure across frequency buckets for different embedding strategies.

Beam Search Synergy

A notable finding is that random embeddings benefit significantly from beam search. While pre-trained embeddings can sometimes show performance degradation as beam size increases, the "disentangled" nature of random space allows beam search to effectively recover correct tokens even when initial predictions are slightly off-target.

Experimental Results

The models were tested on three WMT tasks: English→Turkish (low-resource), Romanian→English (mid-sized), and English→German (large-scale).

Performance comparison (BLEU and BERTScore) across different language pairs and embedding types.

As shown in the results, the Random Uniform model achieved 31.8 BLEU on the large-scale en-de task, significantly outperforming the Pre-trained baseline (29.2 BLEU). The Combined strategy proved most robust, matching or exceeding the best models across nearly all metrics and language pairs, notably achieving 32.0 BLEU on en-de.

Conclusion and Future Work

While random embeddings require a sufficient data volume to be effective—underperforming slightly in very low-resource scenarios like en-tr—they offer a powerful and computationally efficient alternative for large-scale CoNMT. Future research may explore different loss functions beyond cosine distance or the applicability of this approach to other generative tasks like summarization where semantic proximity might play a different role.

Summary updated

BibTeX
@inproceedings{tokarchuk2024unreasonable,
  author = {Evgeniia Tokarchuk and Vlad Niculae},
  title = {{The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.56},
}
Figure