MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders

Wenyu Zhang, Shuo Sun, Bin Wang, Xunlong Zou, Zhuohan Liu, Yingxu He, Geyu Lin, Nancy F. Chen, Ai Ti Aw

Abstract

The rapid advancements in large language models (LLMs) have significantly enhanced natural language processing capabilities, facilitating the development of AudioLLMs that process and understand speech and audio inputs alongside text. Existing AudioLLMs typically combine a pre-trained audio encoder with a pre-trained LLM, which are subsequently finetuned on specific audio tasks. However, the pre-trained audio encoder has constrained capacity to capture features for new tasks and datasets. To address this, we propose to incorporate mixtures of 'weak' encoders (MoWE) into the AudioLLM framework. MoWE supplements a base encoder with a pool of relatively light weight encoders, selectively activated based on the audio input to enhance feature extraction without significantly increasing model size. Our empirical results demonstrate that MoWE effectively improves multi-task performance, broadening the applicability of AudioLLMs to more diverse audio tasks.

Introduction

Current AudioLLMs typically follow a standard architectural recipe: a powerful pre-trained audio encoder (such as Whisper-large) is connected to a Large Language Model (LLM) via a modality adapter, and the entire system is then fine-tuned. However, this approach faces a significant bottleneck. A single encoder, often pre-trained on specific tasks like speech recognition, may lack the specialized knowledge or capacity to handle the nuances of diverse audio data, including environmental sounds, music, or emotional subtext.

Simply scaling up the encoder to be even larger is computationally expensive and often yields diminishing returns. MoWE-Audio addresses this by asking: Can we use a collection of smaller, specialized encoders to fill in the gaps left by the primary "strong" encoder? By introducing a "Mixture of Weak Encoders," the framework boosts performance across diverse speech and non-speech tasks without the computational burden of a massive monolithic encoder.

Methods

The MoWE (Mixture of Weak Encoders) framework supplements a "strong" base encoder with a pool of "weak" encoders—models with at least an order of magnitude fewer parameters (e.g., Whisper-tiny or HuBERT-base).

The AudioLLM framework and the MoWE module architecture.

Dual Routing Strategy

To maintain efficiency, MoWE employs two types of routers to select which weak encoders are activated:

  • Data-independent routing: Selects a fixed weak encoder regardless of the input to provide a stable, supplementary representation.
  • Data-dependent routing: Dynamically selects a weak encoder based on the specific features of the input audio, allowing for task-specific specialization (e.g., picking an emotion-tuned encoder for a crying sound).

Integration and Training

The embeddings from the base encoder and the selected weak encoders are concatenated and passed through a modality adapter and linear projection to align them with the LLM’s input space. The system uses LoRA (Low-Rank Adaptation) to efficiently fine-tune the LLM alongside the routers and adapters, utilizing a combined loss function that includes a "routing loss" to ensure the encoders are utilized effectively.

Experimental Setup

The researchers evaluated MoWE on five distinct audio and speech tasks across several datasets:

  • ASR (Automatic Speech Recognition): LibriSpeech-Clean.
  • ER (Emotion Recognition): MELD-Emotion.
  • AQA (Audio Question Answering): Clotho-AQA.
  • SQA (Speech Question Answering): Spoken-SQuAD.
  • AC (Audio Captioning): AudioCaps.

Generalization was also tested on out-of-distribution (OOD) datasets, including LibriSpeech-Other, SLUE-P2-SQA, and WavCaps.

Key Insights and Results

Empirical results show that small encoders (9M to 95M parameters) can provide critical features that even a massive 637M parameter Whisper-large model misses. The best performance was achieved by combining one fixed (independent) router with one dynamic (dependent) router.

Selection proportion of weak encoders across different datasets, demonstrating task-specific specialization.

As shown in the routing analysis, specialization naturally emerges: the data-dependent router learns to assign specific encoders to specific audio types. For example, Whisper-tiny encoders were frequently routed to speech tasks (ASR), while HuBERT-based encoders were preferred for non-speech audio and emotion recognition (ER).

Performance Gains

MoWE outperformed established models like SALMONN and Qwen-Audio in ER, AQA, and AC tasks. The framework showed consistent improvements when paired with different LLM backbones, including Llama-3-8B, Zephyr-7B, and Phi-3-Mini (3.8B). In the Llama-3 setup, MoWE improved the Model-as-Judge score for Emotion Recognition from 1.41 to 1.69 and boosted the METEOR score for Audio Captioning from 24.42 to 25.78.

Conclusion and Future Work

MoWE-Audio demonstrates that a mixture of small experts is a viable and efficient path toward scaling multimodal models. While MoWE excels on trained tasks, its performance on out-of-distribution datasets remains comparable to baselines, suggesting that multimodal capabilities are still heavily dependent on training data diversity.

Future research may explore even more diverse pools of encoders, such as music-specific or ultra-low-latency models, and potentially apply the "mixture of weak components" logic to the modality adapters themselves to further broaden AudioLLM applicability.

Summary updated

Abstract

The rapid advancements in large language models (LLMs) have significantly enhanced natural language processing capabilities, facilitating the development of AudioLLMs that process and understand speech and audio inputs alongside text. Existing AudioLLMs typically combine a pre-trained audio encoder with a pre-trained LLM, which are subsequently finetuned on specific audio tasks. However, the pre-trained audio encoder has constrained capacity to capture features for new tasks and datasets. To address this, we propose to incorporate mixtures of `weak' encoders (MoWE) into the AudioLLM framework. MoWE supplements a base encoder with a pool of relatively light weight encoders, selectively activated based on the audio input to enhance feature extraction without significantly increasing model size. Our empirical results demonstrate that MoWE effectively improves multi-task performance, broadening the applicability of AudioLLMs to more diverse audio tasks.
BibTeX
@inproceedings{zhang2025mowe,
  author = {Wenyu Zhang and Shuo Sun and Bin Wang and Xunlong Zou and Zhuohan Liu and Yingxu He and Geyu Lin and Nancy F. Chen and Ai Ti Aw},
  title = {{MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888128},
}
Figure