MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders

Wenyu Zhang, Shuo Sun, Bin Wang, Xunlong Zou, Zhuohan Liu, Yingxu He, Geyu Lin, Nancy F. Chen, Ai Ti Aw

TL;DR

MoWE-Audio introduces a "Mixture of Weak Encoders" to enhance Multimodal Large Language Models (AudioLLMs). By supplementing a single "strong" audio encoder with a pool of smaller, specialized "weak" encoders—activated via smart routing—the framework significantly boosts performance across diverse speech and non-speech tasks without the computational burden of a massive monolithic encoder.

What Problem Does This Paper Address?

Current AudioLLMs typically follow a standard recipe: take a powerful pre-trained audio encoder (like Whisper-large), connect it to a Large Language Model (LLM), and fine-tune the whole system.

However, this approach has a bottleneck. A single encoder, often pre-trained on specific tasks like speech recognition, may lack the "knowledge" or capacity to handle the nuances of diverse audio data, such as environmental sounds, music, or emotional subtext. Simply scaling up the encoder to be even larger is computationally expensive and often yields diminishing returns. The authors tackle this by asking: Can we use a collection of smaller, specialized encoders to fill in the gaps left by the primary "strong" encoder?

Methods

The authors propose the MoWE (Mixture of Weak Encoders) framework. Here’s how it works:

  1. The Architecture: The system uses a "strong" base encoder (Whisper-large) as the foundation. It is supplemented by a pool of "weak" encoders—models with at least an order of magnitude fewer parameters (e.g., Whisper-tiny or HuBERT-base).
  2. Dual Routing Strategy: To keep the model efficient, not all weak encoders are used at once. MoWE employs two types of routers:
  3. Data-independent routing: Selects a fixed weak encoder regardless of the input to provide a stable, supplementary representation.
  4. Data-dependent routing: Dynamically selects a weak encoder based on the specific features of the input audio, allowing for task-specific specialization (e.g., picking an emotion-tuned encoder for a crying sound).
  5. Integration: The embeddings from the base encoder and the selected weak encoders are concatenated and passed through a modality adapter and linear projection to align them with the LLM’s input space.
  6. Training: The system uses LoRA (Low-Rank Adaptation) to efficiently fine-tune the LLM alongside the routers and adapters, using a combined loss function that includes a "routing loss" to ensure the encoders are used effectively.

Datasets

The researchers evaluated MoWE on five distinct audio and speech tasks:

  • ASR (Automatic Speech Recognition): LibriSpeech-Clean.
  • ER (Emotion Recognition): MELD-Emotion.
  • AQA (Audio Question Answering): Clotho-AQA.
  • SQA (Speech Question Answering): Spoken-SQuAD.
  • AC (Audio Captioning): AudioCaps.

They also tested generalization on out-of-distribution (OOD) datasets, including LibriSpeech-Other, SLUE-P2-SQA, and WavCaps.

Key Insights and Ideas

  • The Power of "Weakness": Small encoders (9M to 95M parameters) can provide critical features that even a massive 637M parameter Whisper-large model misses.
  • Specialization Emerges: The data-dependent router naturally learns to assign specific encoders to specific types of audio. For example, Whisper-tiny encoders were frequently routed to speech tasks, while HuBERT-based encoders were preferred for non-speech audio and emotion recognition.
  • Routing Synergy: The best performance came from combining one fixed (independent) router with one dynamic (dependent) router. Using two dynamic routers actually performed worse, likely because it introduced too much instability in the embeddings during training.
  • Efficiency: Adding the MoWE module only slightly reduced inference throughput (from 1.92 to 1.83 samples per second), proving that a mixture of small experts is a viable path toward scaling multimodal models.

Key Results

  • State-of-the-Art Performance: MoWE outperformed established models like SALMONN and Qwen-Audio in Emotion Recognition (ER), Audio Question Answering (AQA), and Audio Captioning (AC).
  • Cross-LLM Success: The framework showed consistent improvements when paired with different LLM backbones, including Llama-3-8B, Zephyr-7B, and the smaller Phi-3-Mini (3.8B).
  • Quantitative Gains: In the Llama-3 setup, MoWE improved the Model-as-Judge score for Emotion Recognition from 1.41 to 1.69 and boosted the METEOR score for Audio Captioning from 24.42 to 25.78.

Limitations and Future Work

  • OOD Generalization: While MoWE excels on trained tasks, its performance on out-of-distribution datasets was comparable to the baseline. The authors note that multimodal LLM capabilities are still heavily dependent on the diversity of the training data.
  • Future Directions: The authors suggest that exploring even more diverse pools of encoders (perhaps including music-specific or ultra-low-latency models) could further broaden the applicability of AudioLLMs. There is also potential to apply this "mixture of weak components" logic to the modality adapters themselves.

Summary updated

Abstract

The rapid advancements in large language models (LLMs) have significantly enhanced natural language processing capabilities, facilitating the development of AudioLLMs that process and understand speech and audio inputs alongside text. Existing AudioLLMs typically combine a pre-trained audio encoder with a pre-trained LLM, which are subsequently finetuned on specific audio tasks. However, the pre-trained audio encoder has constrained capacity to capture features for new tasks and datasets. To address this, we propose to incorporate mixtures of ‘weak’ encoders (MoWE) into the AudioLLM framework. MoWE supplements a base encoder with a pool of relatively lightweight encoders, selectively activated based on the audio input to enhance feature extraction without significantly increasing model size. Our empirical results demonstrate that MoWE effectively improves multi-task performance, broadening the applicability of AudioLLMs to more diverse audio tasks.
BibTeX
@inproceedings{zhang2025mowe,
  author = {Wenyu Zhang and Shuo Sun and Bin Wang and Xunlong Zou and Zhuohan Liu and Yingxu He and Geyu Lin and Nancy F. Chen and Ai Ti Aw},
  title = {{MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888128},
}

Loading related content...

Figure