Introduction
Standard Neural Machine Translation (NMT) models are remarkably effective when processing clean, well-formatted text. However, their performance is notoriously fragile when faced with "multimodal noise"—errors introduced by upstream systems such as Automatic Speech Recognition (ASR), Optical Character Recognition (OCR), or the informal nature of User-Generated Content (UGC). For instance, an ASR system might transcribe "pour trier" as "pourtrier," or an OCR scan might misidentify "Il" as "Tl." These subtle perturbations often lead to significant drops in translation quality.
Traditional solutions involve fine-tuning models on specific noisy datasets or augmenting training with synthetic noise. While effective for specific scenarios, these methods often fail to generalize to new domains or novel noise types. This paper introduces a modular, extensible architecture designed to handle diverse noise sources through a "plug-and-play" adapter-based framework.
Methodology: Modular Robustness via Adapters
The core of the proposed approach is the use of Adapters—small, trainable bottleneck layers inserted into a frozen, pre-trained Transformer model. This design allows for a decoupled training strategy that separates domain-specific knowledge from noise-handling capabilities.
Two-Step Training Strategy
- Domain Adaptation: First, Domain Adapters (DA) are trained on clean, in-domain data to capture the specific linguistic style and vocabulary of the target task (e.g., TED talks or technical documentation).
- Noise Adaptation: Once the DAs are established, they are frozen alongside the base model. New Noise Adapters (NA) are then injected and trained specifically on noisy data representations (ASR-style, OCR-style, or UGC-style).
Runtime Composition and Fusion
When the source of noise is known at inference time, the system utilizes Dynamic Routing to select the appropriate NA. However, in many real-world scenarios, the noise source is unknown. To address this, the authors propose two transfer learning mechanisms to merge knowledge across multiple NAs:
- Fast Fusion (FF): A linear projection that aggregates outputs from all available noise adapters.
- Multi Fusion (MF): A sophisticated multi-head attention mechanism that dynamically weights the contributions of different noise adapters based on the specific input sequence.
Experimental Setup and Datasets
The framework was evaluated primarily using the mTEDx (Multilingual TEDx) corpus, covering 12 language pairs. To simulate real-world conditions, the authors generated noisy variants:
- Noisy ASR: Audio processed through a SpeechBrain ASR system.
- Noisy OCR: Transcripts rendered as images and re-extracted via CRAFT and CRNN models.
- Noisy UGC: Simulated keyboard typos using the "Butter Fingers" perturbation.
To test the "plug-and-play" compositionality, the authors also utilized CoVoST 2 as an additional clean domain, testing whether a Noise Adapter trained on mTEDx could be successfully paired with a Domain Adapter trained on CoVoST 2.
Key Results and Discussion
The experimental results demonstrate that decoupling domain and noise knowledge is highly effective. On noisy ASR data, the Domain + Noise Adapter configuration improved BLEU scores by an average of +6.0 points over a generic NMT model and +3.4 points over a model adapted only to the domain.
As shown in the training curves, the Multi Fusion (MF) approach significantly outperforms standard Adapter Fusion (AF). It effectively bridges the gap to an "Oracle" system (which knows the noise source perfectly), with only a negligible 0.16 BLEU point difference on average. Furthermore, the system exhibited strong compositionality: when applying an mTEDx-trained Noise Adapter to a new CoVoST 2 Domain Adapter, the model recovered nearly all performance lost to noise, dropping only 1.2 BLEU points compared to clean text, whereas the baseline dropped 13.9 points.
Conclusion
This research presents a scalable solution for multimodal NMT robustness. By utilizing a modular adapter architecture, the system can be easily extended to new domains or noise types without retraining the entire model. The introduction of Multi Fusion allows for high performance even when the source of noise is unknown, making it a practical choice for real-world translation pipelines. Future work may explore moving beyond text-to-text pipelines to integrate raw audio or image signals directly into the fusion process.
Summary updated