Abstract
Neural Machine Translation (NMT) systems often struggle to balance high-quality domain adaptation with the storage and latency constraints of multi-user cloud environments. This paper presents a method for creating efficient, personalized NMT models by representing user-specific adaptations as sparse "offsets" from a shared base model. By utilizing group lasso regularization and sparse output projections, we demonstrate that up to 75% of a Transformer model's parameters can remain frozen without sacrificing translation quality. Our results show that these compact models (storing ~9.2M parameters) achieve performance comparable to full-model adaptation while significantly reducing the storage footprint across multiple domains and language pairs.
Introduction
Professional translation workflows require NMT systems to adapt to specialized domains, such as legal or technical fields, to maintain terminology and stylistic consistency. In large-scale services, providing a unique, full-sized model (typically ~35M parameters) for every user is computationally and financially prohibitive. This research addresses the challenge of providing high-quality, personalized translation without the overhead of storing and loading full models for thousands of individual users.
Methodology
The proposed approach represents a personalized model as the sum of a shared base model and a user-specific offset: $$W = W_{base} + W_{offset}$$
To minimize the storage requirements for $W_{offset}$, two key techniques are employed:
- Group Lasso Regularization: Unlike standard L1 regularization, Group Lasso treats entire weight tensors as groups. This encourages the optimization process to drive entire matrices in the offset to zero. If an offset tensor is zero, the system defaults to the base model weights, eliminating the need for additional storage.
- Sparse Output Projection: The output projection layer is the largest component of the model. By updating only the weights corresponding to words present in the user's specific training data, the system achieves a highly sparse and efficient representation of this layer.
The framework supports both Batch Adaptation (pre-training on existing translation memories) and Incremental Adaptation (real-time learning from user corrections).
Experimental Setup
The method was evaluated on English→German datasets across four domains: Financial (User1), Technical Support (User2), Technical Documentation (Autodesk), and Semi-technical talks (IWSLT). To ensure generalizability, the researchers further validated the approach across six additional language pairs (English↔French, English↔Russian, and English↔Chinese) using production-grade data from various industrial domains.
Results and Discussion
The experimental results highlight a significant "80/20" rule in NMT parameter adaptation: a small subset of parameters is responsible for the majority of quality gains.
As shown in Figure 1, the "Lasso" method achieves translation quality nearly identical to the "Full Model" while requiring significantly less storage. For instance, on the User1 dataset, the compact Lasso-based model achieved a BLEU score of 51.3—surpassing the baseline (35.7) and matching the full model (50.6) while storing only 9.2M parameters.
A critical finding is the importance of the output projection layer. Updating this layer sparsely provides a more substantial boost to adaptation quality than updating internal encoder or decoder layers. Furthermore, the effectiveness of incremental adaptation was found to be highly dependent on the "repetition rate" of the text; domains with high internal consistency see the greatest benefits from real-time learning.
The cross-lingual evaluation (Figure 2) confirms the robustness of this approach. Across six language pairs, the compact model provided an average improvement of 14.3 BLEU points over the unadapted baseline, trailing the full-parameter model by a negligible 1.2 BLEU points.
Conclusion
This work demonstrates that highly efficient, personalized NMT is achievable through sparse parameter offsets. By leveraging Group Lasso and sparse projections, developers can provide high-quality domain adaptation with a fraction of the storage and latency costs associated with full-model fine-tuning. Future work may focus on automating hyperparameter tuning and adapting the sparsity constraints to evolving hardware architectures.
Summary updated