LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors

Rao Ma, Tongzhou Chen, Kartik Audhkhasi, Bhuvana RamabhadranEMNLP 2025

TL;DR

This paper introduces LegoSLM, a novel method for seamlessly integrating pre-trained speech encoders and Large Language Models (LLMs). By leveraging Connectionist Temporal Classification (CTC) posteriors, LegoSLM achieves competitive performance on ASR and Speech Translation tasks while offering enhanced modularity and flexibility, allowing speech encoders to be swapped without retraining the LLM.

What Problem Does This Paper Address?

The rapid advancement of pre-trained speech encoders and LLMs has opened doors for powerful spoken language processing. However, effectively combining these two powerful models has been challenging. Existing methods like Automatic Speech Recognition (ASR) error correction (AEC) or speech prompting often suffer from suboptimal performance, inflexibility, or information loss. AEC systems, for instance, rely on decoded hypotheses, which can lose crucial information. Speech prompting methods, while preserving more information, often create a tight coupling between a specific speech encoder and the LLM, limiting their adaptability. This paper aims to bridge this gap by proposing a method that allows for flexible and efficient integration of speech encoders and LLMs, preserving information and enabling modularity.

Methods

The core of LegoSLM lies in using CTC posteriors as an intermediate representation to connect the speech encoder and the LLM. Here's a breakdown of the approach:

  1. Speech Encoder Fine-tuning: A pre-trained speech encoder (like USM) is fine-tuned with a CTC head. Crucially, the output vocabulary of this CTC head includes all tokens from the LLM's vocabulary plus a special blank token. This fine-tuning is done using ASR training data.
  2. LLM Adaptation: The speech encoder is then frozen. The LLM is adapted by using the CTC posteriors generated by the speech encoder. Specifically, these posteriors are used to compute a weighted sum of the LLM's input embedding table, effectively reconstructing "pseudo-speech embeddings." These embeddings are then concatenated with the LLM's text embeddings for further processing.
  3. Modularity: A key advantage is that after the LLM is adapted, the speech encoder can be swapped out for a different one, as long as it uses the same CTC vocabulary. This is achieved in a "zero-shot" manner, without retraining the LLM.
  4. AM/LM Spectrum Control: To balance the influence of the speech encoder and the LLM during decoding, a temperature parameter ($\tau$) is introduced in the CTC softmax layer. This allows for fine-grained control over how much emphasis is placed on acoustic information versus language model knowledge.
  5. Model Variants: The paper also explores variants like LegoTopS and LegoTopP, which consider only the top-K CTC predictions to reduce computational complexity and potentially improve efficiency.

Datasets

The experiments utilize several standard datasets:

  • For ASR:
    • MLS (Multilingual LibriSpeech): Used for training multilingual ASR models and evaluating on English subsets (MLS_en) and other languages.
  • LibriSpeech: Used for training English-only ASR models.
  • SpeechStew: A collection of various public ASR datasets, combined with MLS_en for a larger English ASR training set.
  • For Speech Translation (AST):
    • CoVoST 2: A diversified speech translation dataset used for training and evaluation across multiple language pairs (e.g., fr→en, de→en, en→de).

Key Insights and Ideas

  • CTC Posteriors as a Universal Bridge: The paper demonstrates that CTC posteriors are a surprisingly effective intermediate representation for bridging speech encoders and LLMs. This is a more information-rich signal than simple ASR hypotheses used in AEC.
  • True Modularity Achieved: The zero-shot system combination experiments highlight that LegoSLM truly decouples the speech encoder and LLM. This is a significant advantage for real-world applications where speech encoders might need to be updated or replaced.
  • Temperature Control for Fine-tuning: The AM/LM spectrum control using a temperature parameter provides a principled way to balance acoustic and linguistic information, leading to improved performance and adaptability across different training setups.
  • Privacy Preservation: Using CTC posteriors, which are derived from the speech encoder but not the raw audio, offers a degree of privacy preservation compared to methods that might directly process continuous audio features.

Key Results

  • ASR Performance: LegoSLM achieved competitive ASR performance, with a WER of 6.1 on the MLS_en test set. The LegoSLM* variant (which reduces the influence of the <blk> token) achieved a 37% WERR over the USM-CTC baseline on 8 MLS testsets. It also showed strong performance on multilingual tasks.
  • Speech Translation Performance: LegoSLM outperformed AEC and speech prompt methods on AST tasks, demonstrating its ability to handle semantic understanding and translation effectively.
  • Zero-Shot System Combination: The method showed robust performance when swapping speech encoders, outperforming speech prompt methods which failed to transfer.
  • Efficiency: LegoSLM introduces minimal overhead during inference compared to continuous speech encoder outputs.
  • Temperature Tuning: Optimal temperature values were found to improve performance, with values below 1.0 favoring the speech encoder when it was trained on more data, and values above 1.0 favoring the LLM when it had more data.

Limitations and Future Work

  • Speech Encoder Architectures: The current work primarily uses CTC-based speech encoders (like USM). The authors suggest extending the approach to other architectures such as RNN-T or LAS models.
  • LLM Adaptation Methods: While fine-tuning the LLM weights was used, the paper acknowledges that parameter-efficient tuning methods like LoRA are common for LLMs and could be explored in future work.
  • Broader Applications: The authors believe LegoSLM has potential for broader applications beyond ASR and AST, such as speech summarization and spoken language understanding.
  • Foundation Models: Future work aims to expand the analysis to other large-scale foundation models to draw broader conclusions.

Summary updated

BibTeX
@inproceedings{ma2025legoslm,
  author = {Rao Ma and Tongzhou Chen and Kartik Audhkhasi and Bhuvana Ramabhadran},
  title = {{LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure