Introduction
Deep Neural Machine Translation (NMT) models, particularly those based on the Transformer architecture, have traditionally benefited from increased depth. However, researchers have identified a persistent "performance ceiling" where gains in translation quality (measured by BLEU scores) plateau or even decline once the encoder exceeds approximately 18 layers. This phenomenon, coupled with the high computational and memory costs of training deep models, limits the practical scalability of NMT systems.
This paper introduces Symbiosis Networks, a novel joint-training strategy designed to push deep models beyond these standard limits. By utilizing a smaller, parameter-sharing sub-network to regularize the deeper main network, the authors demonstrate that deep models can continue to improve in performance without increasing inference-time costs.
Symbiosis Networks: Architecture and Training
The core of the proposed framework is the interaction between two networks:
- M-Net (Main Network): The full-depth Transformer model intended for final deployment.
- S-Net (Sub-Network): A shallower version of the M-Net that shares its parameters, including the embedding layer, the decoder, and a specific subset of the M-Net’s encoder layers.
The Two-Stage Training Strategy
The "symbiotic" relationship is established through a two-stage training process:
- Joint Training: Both M-Net and S-Net are trained simultaneously using standard cross-entropy loss. This ensures that the shared parameters are optimized to perform well in both shallow and deep configurations.
- Margin Training: To prevent the deeper layers of the M-Net from becoming redundant, the authors introduce a Margin Regularization Loss. This hinge loss explicitly requires the M-Net to achieve a log-likelihood score that is at least a fixed margin ($\tau$) higher than the S-Net. Essentially, the deeper layers must "earn their keep" by providing a measurable performance boost over their shallower counterpart.
Experimental Results
The researchers evaluated Symbiosis Networks on standard NMT benchmarks, including WMT’14 English-German (EN↔DE) and English-French (EN→FR). The results consistently showed that this strategy outperforms classic training methods across various model depths.
Key Findings
- Breaking the Ceiling: As shown in Figure 1, while classic training performance drops after 21 encoder layers, the Symbiosis strategy allows a 30-layer model to continue improving, reaching the highest BLEU scores in the study.
- Depth Efficiency: A 12-layer model trained with the Symbiosis method outperformed an 18-layer model trained traditionally. This indicates that the strategy allows for higher quality translations with 33% fewer encoder layers, significantly improving efficiency.
- Inference Neutrality: Since the S-Net is only utilized during the training phase, the final deployed M-Net maintains the same parameter count and latency as a standard Transformer of equivalent depth.
Conclusion
Symbiosis Networks provide a robust solution to the performance plateau observed in deep NMT models. By leveraging internal sub-networks for self-regularization through margin loss, the method ensures that additional depth translates into actual performance gains. While the two-stage training process adds some complexity to the pipeline, the resulting improvements in BLEU scores and the ability to scale models more effectively make it a compelling approach for high-performance machine translation.
Summary updated