Abstract
Large-scale speech models, while robust, frequently fail to recognize or translate rare entities such as specific landmarks, technical terms, or personal names. Traditional fine-tuning to address this "long-tail" problem is often prohibitively expensive and leads to catastrophic forgetting of general capabilities. This study introduces a training-free approach utilizing task vectors—the parameter-wise difference between a specialized and a base model. By employing merging techniques such as Task Arithmetic (TA), TIES, and DARE, we demonstrate that rare word capabilities can be "plugged into" models like Whisper. Our results show that this method not only matches fine-tuned performance on rare words but also improves general translation scores by approximately 5 BLEU points while significantly mitigating catastrophic forgetting.
1. Introduction
The "long-tail" problem remains a significant hurdle for state-of-the-art speech models. While models like OpenAI’s Whisper exhibit impressive general-purpose performance, their accuracy drops sharply when encountering rare words—geographical locations, niche technical terminology, or specific names not well-represented in the training data.
The conventional remedy is fine-tuning on specialized datasets. However, this approach presents three critical challenges:
- Computational Cost: Fine-tuning massive architectures for every new vocabulary set is inefficient.
- Scalability: Merging multiple fine-tuned models is non-trivial.
- Catastrophic Forgetting: Specialization often degrades the model's performance on general tasks, eroding its "generalist" utility.
To overcome these limitations, we propose a paradigm shift using task vectors to incorporate new word-level knowledge without traditional retraining.
2. Methodology: Task Vector Arithmetic
The core of our approach lies in the extraction and manipulation of a Task Vector ($\tau$). A task vector represents the specific weight adjustments required to master a new task (or in this case, a new word). It is calculated as the element-wise difference between the parameters of a fine-tuned model ($\theta_i$) and the original pre-trained model ($\theta_0$):
$$\tau_i = \theta_i - \theta_0$$
Once extracted, these vectors can be combined and applied to the base model using various merging strategies:
- Task Arithmetic (TA): A straightforward summation of task vectors.
- TIES (Trim, Elect Sign, & Merge): A sophisticated method that trims low-magnitude parameter changes (noise) and resolves sign conflicts between different vectors.
- DARE (Drop and Rescale): A technique that randomly drops parameters and rescales the remaining ones to minimize redundancy and interference.
This allows for "word-level granularity," where specific capabilities (e.g., recognizing "Neuschwanstein Castle") can be added as modular components.
3. Experimental Setup
3.1 Synthetic Data Pipeline
Because rare word data is scarce, we developed a synthetic generation pipeline:
- Vocabulary Generation: GPT-4o was used to identify rare words across three domains: Landmarks (e.g., Berlin Wall), Locations (e.g., Avignon), and Individuals (e.g., Goethe).
- Contextual Synthesis: GPT-4o generated approximately 1,000 English sentences per word, which were then translated into Chinese.
- Audio Generation: We utilized CosyVoice2, a high-fidelity text-to-speech (TTS) engine, to produce the corresponding audio, resulting in a high-quality, multi-modal, bilingual dataset.
3.2 Benchmarking
Performance was evaluated against a general benchmark of 1,702 "normal" sentences to monitor for catastrophic forgetting and general translation quality.
4. Results and Discussion
4.1 Rare Word Accuracy
The task vector-based models (TA, TIES, DARE) achieved performance levels comparable to, and occasionally exceeding, fully fine-tuned models on the target rare words. This suggests that the essential "knowledge" of the rare word is effectively captured within the task vector.
4.2 General Performance and Synergy
A surprising finding was the improvement in general translation performance. Rather than merely preserving original skills, the merging process appeared to act as a form of regularized fine-tuning. The general translation score improved from a baseline of 35.66 BLEU (Raw Whisper-medium) to 40.77 BLEU using the TIES method.
4.3 Mitigating Catastrophic Forgetting
In Chinese speech recognition tasks, standard fine-tuning often led to a sharp increase in Character Error Rate (CER)—for instance, rising from 10% to 29% in specific tests. In contrast, the TIES merging method maintained a significantly lower CER (approximately 21%), demonstrating superior stability and retention of the base model's original capabilities.
5. Limitations and Future Work
While promising, the method faces a "crowding effect." As the number of task vectors increases (beyond 4 or 5), performance begins to degrade due to parameter interference, where different word vectors compete for the same weight space. Furthermore, the method's efficacy is tied to the quality of the TTS engine; inaccuracies in synthetic pronunciation directly impact the quality of the resulting task vector.
Future research will focus on scaling this arithmetic approach to accommodate thousands of rare words and developing more advanced conflict-resolution strategies to mitigate parameter interference.
6. Conclusion
Task vectors offer a modular, efficient, and scalable alternative to traditional fine-tuning for speech models. By treating model weights as arithmetic components, we can enhance specialized vocabulary recognition while simultaneously boosting general model performance, paving the way for more adaptable and robust speech AI.
Summary updated