Unfolding and Shrinking Neural Machine Translation Ensembles

Felix Stahlberg, Bill Byrne

Abstract

Ensembling multiple Neural Machine Translation (NMT) models significantly improves translation quality but introduces substantial computational overhead. This paper proposes a novel "unfold and shrink" pipeline that merges an ensemble into a single large network and subsequently prunes it to the size of a standalone model. The resulting system achieves ensemble-level accuracy with the inference speed and memory footprint of a single model.

Introduction

In Neural Machine Translation (NMT), ensembling—averaging predictions from multiple independently trained models—is the standard approach for achieving state-of-the-art BLEU scores. However, this comes with an "ensemble tax": translating a sentence requires $K$ separate forward passes, leading to $K$-fold increases in latency and memory consumption. This makes deployment on resource-constrained devices, such as smartphones, challenging.

While knowledge distillation is a common remedy, it requires a complex training pipeline to teach a "student" model from a "teacher" ensemble. This paper introduces a structural alternative: mathematically merging ensemble members and then pruning the resulting redundancy to maintain performance at a fraction of the cost.

Methodology

Unfolding

The first stage involves "unfolding" $K$ models with identical architectures into a single massive network. By stacking the weight matrices of individual models (horizontally or vertically, depending on the layer type), the authors create a single model that mathematically approximates the ensemble's behavior.

Illustration of unfolding two single networks into one combined network.

Even before size reduction, this unfolded model offers a GPU advantage. It reduces communication overhead and performs the computationally expensive softmax operation only once, rather than $K$ times.

Shrinking

The unfolded model is significantly larger than a standard network. To restore a manageable size, the authors employ three primary techniques:

  1. SVD for Embedding Layers: Singular Value Decomposition identifies low-rank approximations for large embedding matrices, exploiting their inherent redundancy.
  2. Data-Free Neuron Removal: This technique identifies neurons with similar weight patterns. Instead of simple deletion, a linear combination of remaining neurons "compensates" for the loss, minimizing the impact on output.
  3. Data-Bound Neuron Removal: An advanced variant that utilizes actual neuron activations during a brief training pass to identify and prune truly redundant neurons in practice.
Translation quality (BLEU) as a function of layer size reduction for various model components.

As shown in the figure above, different layers exhibit varying levels of sensitivity to shrinking. Embedding layers, in particular, can be significantly reduced with minimal impact on BLEU scores, often outperforming the baseline single system even at 20-30% of their original size.

Experimental Setup

The approach was evaluated on two major translation tasks:

  • Japanese-English (Ja-En): Using the ASPEC corpus (500K sentence pairs).
  • English-German (En-De): Using WMT datasets (news-test2014 for development, 2015/2016 for testing).

Key Results and Insights

The "unfold and shrink" method demonstrates that ensembles of models with the same topology contain massive amounts of overlapping information. Key findings include:

  • Performance Parity: On the Ja-En task, a shrunk model (3-Unfold-Normal) achieved a 2.2 BLEU point gain over a standard single model while maintaining identical size and speed.
  • Efficiency Gains: Compared to a traditional 3-model ensemble, the shrunk version provided a 3.4x speedup during CPU decoding.
  • Linear Compensation: Simply deleting neurons significantly degrades performance. The success of this method relies on mathematically redistributing the weights of deleted neurons into their neighbors to preserve the network signal.
  • Embedding Redundancy: Embedding layers are often over-parameterized and can be shrunk via SVD without losing translation quality.

Conclusion and Future Work

The proposed method effectively bridges the gap between high-performance ensembles and efficient standalone models. However, limitations remain. The current approach requires all ensemble members to share the same architecture, precluding the merging of diverse models (e.g., GRU-based and Transformer-based). Additionally, shrinking the "maxout" output layer remains a challenge. Future research will explore associations between units in different layer types to apply these techniques to more diverse ensembles.

Summary updated

BibTeX
@inproceedings{stahlberg2017unfolding,
  author = {Felix Stahlberg and Bill Byrne},
  title = {{Unfolding and Shrinking Neural Machine Translation Ensembles}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1208},
}
Figure