The Green KNIGHT: Green Machine Translation with Knowledge-Distilled, Narrow, Inexpensive, Greedy, Hybrid Transformers

Andreas Guta, Frithjof Petrick, Peter PolákEMNLP 2025

TL;DR

This paper introduces "Green KNIGHT," a set of practical optimizations for Neural Machine Translation (NMT) that drastically reduces inference speed and energy consumption with minimal quality loss. By combining well-understood techniques like greedy decoding, quantization, and architectural modifications, they achieve up to 140x speedups and 98.7% energy savings, making high-quality NMT more accessible and environmentally friendly.

What Problem Does This Paper Address?

State-of-the-art Neural Machine Translation (NMT) models, while achieving impressive fluency and adequacy, come with a significant computational cost. They require substantial GPU resources, leading to high inference latency and considerable energy consumption, contributing to carbon emissions. This makes advanced NMT less accessible and environmentally sustainable, especially for widespread deployment. The paper aims to bridge this gap by developing "green" NMT systems that are both efficient and practicable, without relying on custom hardware or complex, low-level code.

Methods

The Green KNIGHT approach is a collection of "recipes" that combine several inference-time optimizations and architectural modifications:

  • Greedy Search: Replacing beam search with greedy decoding (selecting the most probable token at each step) eliminates overhead and speeds up inference.
  • Quantization: Using 8-bit integer arithmetic for key operations (like matrix multiplications in feedforward blocks and attention mechanisms) significantly speeds up computation, especially on modern CPUs.
  • Architectural Optimizations:
    • Layer Reallocation & Reduction: Shifting decoder layers to the encoder and reducing the number of decoder layers (e.g., from 12/12 to 5/1) leverages encoder parallelism and streamlines the decoder, which is the primary bottleneck.
  • Decoder Width Compression: Reducing the hidden dimension of the decoder layers further decreases computational load.
  • Hybrid Decoders: Replacing the Transformer decoder with lightweight RNN modules (LSTM or SSRU), often enhanced with cross-attention to the encoder's hidden states ("interleaving"), is explored to reduce decoder complexity.
  • Knowledge Distillation (KD): Training smaller, optimized models (students) using a larger, high-quality baseline model (teacher) helps recover some of the quality lost during the optimization process.

Datasets

The paper uses several standard datasets for training and evaluation:

  • Unconstrained En→De and En→Ko: Trained on large amounts of publicly available data from OPUS. Evaluated on four test sets each, covering domains like movie subtitles, talks, news, and parliamentary speeches.
  • WMT 2014 En→De and En→Fr: Evaluated on the corresponding newstest2014 test sets.

Key Insights and Ideas

  • Decoder Bottleneck is Real: The paper confirms that the autoregressive nature of the Transformer decoder is the primary cause of high inference latency, often accounting for over 95% of the computation time.
  • Greedy Search + Quantization is Powerful: Even without architectural changes, simply applying greedy search and 8-bit quantization yields significant speedups (e.g., 14.5x) with only minor quality degradation (around 2.5% BLEU).
  • Architectural Shifts Pay Off: Reallocating decoder layers to the encoder and reducing decoder depth/width is crucial for substantial speedups, as it directly addresses the decoder bottleneck.
  • RNN Decoders Offer a Trade-off: While hybrid RNN-Transformer decoders can be efficient, their performance is sensitive to optimizations like quantization and greedy search. Interleaved LSTM decoders showed a good balance of quality and speed.
  • Knowledge Distillation Recovers Quality: KD is effective in mitigating quality loss, especially when combined with optimized architectures.

Key Results

The Green KNIGHT approach achieves remarkable improvements:

  • En→De (Unconstrained): Up to 117x CPU speedup and 98.2% energy savings with a 9% relative BLEU decrease.
  • WMT 2014 En→De and En→Fr: Up to 140x speedup and 98.7% energy savings, with quality drops within 10-12% relative BLEU.
  • CPU vs. GPU: The final optimized model achieves an 18x speedup on CPU compared to the baseline model running on GPU.
  • Overall Quality Preservation: The final models maintain at least 92.9% COMET and 88.0% BLEU scores relative to the baseline.

Limitations and Future Work

  • Human Evaluation: The evaluation relies solely on automatic metrics (BLEU, COMET, BLEURT). The authors acknowledge that these metrics may not perfectly reflect human judgment, and further investigation with human evaluation is needed.
  • Single Training Run: Results are based on a single training run per system, and variability from different training runs is not accounted for.
  • Hardware Specificity: While the methods are hardware-agnostic, the measurements were performed on a specific server setup. Different hardware configurations might yield varying results.
  • Low-Resource Languages: The study focuses on medium and high-resource language pairs. Investigating performance on low-resource settings is a potential future direction.
  • Source Language Bottleneck: While the encoder is less of a bottleneck, further investigation into its role, especially for different source languages, is suggested.

Summary updated

BibTeX
@inproceedings{guta2025green,
  author = {Andreas Guta and Frithjof Petrick and Peter Polák},
  title = {{The Green KNIGHT: Green Machine Translation with Knowledge-Distilled, Narrow, Inexpensive, Greedy, Hybrid Transformers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure