TL;DR
This paper introduces a novel, lightweight method to fine-tune frozen pre-trained encoder-decoder models for specific downstream metrics (like translation quality or speech recognition accuracy) without costly retraining. By training a small "proxy network" to predict these metrics, its gradients can be used to subtly perturb encoder outputs, leading to improved performance with minimal computational overhead.
What Problem Does This Paper Address?
Modern foundation models, like those used for machine translation (MT) and automatic speech recognition (ASR), achieve impressive results. However, they are typically trained using "teacher forcing" with a likelihood-based objective (predicting the next token in a reference sequence). This often doesn't directly align with the actual evaluation metrics used for these tasks, such as COMET for MT or Word Error Rate (WER) for ASR, which measure sequence-level quality. While methods like reinforcement learning or Minimum Bayes Risk decoding exist to bridge this gap, they are often computationally expensive, especially for large, pre-trained models. Retraining these models is also infeasible due to cost and instability. Existing inference-time methods, like ensembling, also add significant computational burden. The paper addresses the need for an efficient way to adapt these powerful, frozen models to optimize for specific, non-differentiable metrics without extensive retraining or ensemble methods.
Methods
The core of the proposed method is a Non-Autoregressive Proxy (NAP). This is a small, lightweight neural network trained to approximate a desired sequence-level metric (e.g., COMET score or WER). Crucially, this NAP is trained on the encoder outputs of the foundation model, not the full sequence generation.
The process involves:
- Training the NAP: The NAP is trained to predict the target metric (e.g., COMET score) given the encoder's output embeddings for a given input. This is done using a differentiable loss function (like Pearson correlation) that approximates the target metric.
- Differentiating through the NAP: Once trained, the NAP's gradients with respect to the encoder outputs are computed. These gradients indicate how to change the encoder outputs to increase the predicted metric score.
- Perturbing Encoder Outputs: These gradients are then used to create a small, sample-specific perturbation ($\delta$) to the encoder's output embeddings. This perturbation is applied during inference. The magnitude of this perturbation is controlled by a hyperparameter ($\alpha$).
This approach allows for sample-specific adjustments to the encoder's representation, guiding the decoder to produce outputs that better align with the desired metric, all while keeping the large foundation model frozen.
Datasets
- Machine Translation (MT):
- OPUS-100: Used for training and evaluating the Flan-T5 model on English-to-German (en→de) and English-to-Spanish (en→es) translation tasks.
- Automatic Speech Recognition (ASR):
- AMI meeting corpus: Used for training and evaluating the Whisper model on ASR.
- LibriSpeech: Used for evaluating Whisper on a simpler ASR benchmark.
No new benchmarks were introduced; the paper focuses on applying the method to existing datasets and models.
Key Insights and Ideas
- Lightweight Proxies for Metric Optimization: The surprising effectiveness of a small, auxiliary network (NAP) in capturing complex sequence-level metrics is a key insight. It demonstrates that even a simplified representation of the desired outcome can provide valuable guidance.
- Sample-Specific Perturbations: The method's ability to generate sample-specific perturbations is crucial. This means the adjustments are tailored to the individual input, allowing for more nuanced control than a global fine-tuning approach.
- Frozen Model Compatibility: The approach works with frozen, pre-trained foundation models, which is a significant advantage given the prohibitive cost of retraining these large systems.
- Generalizability Across Metrics and Domains: The paper shows that the NAP approach can be applied to different metrics (COMET, WER) and different modalities (MT, ASR), suggesting broad applicability. It also shows some robustness to the specific training data used for the NAP.
- Trade-off with Step Size ($\alpha$): The importance of finding an optimal perturbation step size ($\alpha$) is highlighted. Too small, and there's no effect; too large, and performance degrades, indicating a delicate balance is needed.
Key Results
- NMT Performance Improvement: For Flan-T5 on the OPUS-100 dataset, the proposed perturbation method led to consistent improvements in COMET scores across various translation directions and beam search configurations. For example, in en→de translation, the perturbation improved COMET scores from 64.3 to 66.0 for base models using greedy search.
- ASR Performance Improvement: For Whisper models on the AMI meeting corpus, the method also yielded improvements in Word Error Rate (WER). For instance, Whisper Tiny saw a reduction in WER from 24.0 to 23.5 with the optimal perturbation.
- Model Size Impact: Smaller models (e.g., Flan-T5 Small) benefited more from the perturbation approach than larger, more robust models, which showed smaller gains.
- SacreBLEU vs. COMET: While COMET scores improved, SacreBLEU scores remained largely unchanged. The authors hypothesize this is because the perturbations might introduce semantically closer words that don't necessarily align with n-gram overlap, or help overcome premature translation termination.
- Efficiency: The method adds only a negligible increase in runtime, making it highly efficient compared to ensemble methods or retraining.
Limitations and Future Work
- Encoder-Decoder Constraint: The current approach is limited to encoder-decoder architectures. Extending it to decoder-only models (like standard Large Language Models) is a significant area for future work.
- Effectiveness on High-Performing Models: For models that already perform very well on a benchmark (like Whisper on LibriSpeech), the gains from this method are marginal. This suggests the approach is more impactful when there's a clear gap between the model's current performance and the desired metric.
- Multimodal Models: The authors suggest future work could explore applying this to modal-specific encoders within multimodal language models.
- Other Attributes: The paper mentions the approach is generalizable and could be applied to other attributes like sentiment, which is an avenue for further exploration.
Summary updated