Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison

Tsz Kin Lam, Marco Gaido, Sara Papi, Luisa Bentivogli, Barry Haddow

Abstract

The rise of Large Language Models (LLMs) has shifted the paradigm of Speech-to-Text (S2T) research toward "dense feature prepending" (DFP), where audio features are directly concatenated to text prompts. This paper presents a controlled, empirical comparison between DFP and the classical cross-attention architecture. Results indicate that while both methods achieve similar translation and recognition quality, DFP is substantially less efficient, requiring more memory and offering slower inference speeds.

Introduction

Modern S2T systems often adapt decoder-only LLMs by "injecting" speech through prepending audio features. While effective for leveraging massive pre-trained models, it remains unclear if DFP is an architectural improvement or simply a beneficiary of scale. Existing literature often lacks fair comparisons due to varying pre-trained backbones and parameter counts. This study levels the playing field by training models from scratch to isolate the impact of the connection mechanism itself.

Architectural Configurations

The study evaluates three primary architectures to determine which is truly more effective for Automatic Speech Recognition (ASR) and Speech Translation (ST):

  1. Cross-Attention: The standard encoder-decoder setup where a speech encoder interacts with the decoder via a dedicated cross-attention layer.
  2. Decoder-only: A pure LLM-style approach where downsampled speech features are prepended to text embeddings without a deep speech encoder.
  3. Decoder-prepend: A hybrid model using a deep speech encoder whose output is prepended to the decoder's input sequence.
Comparison of (a) Cross-Attention, (b) Decoder-Only, and (c) Decoder-Prepend architectures.

The researchers also explored auxiliary techniques such as CTC compression and Sequence-level Knowledge Distillation (seqKD) to enhance performance across these configurations.

Key Findings

Quality Parity

When controlling for parameters and data, Decoder-prepend and Cross-Attention perform almost identically. On the CoVoST2 benchmark, both architectures achieved approximately 29.7 BLEU using Conformer encoders. This suggests that the "modern" prepending method does not inherently improve speech understanding over traditional cross-attention. In contrast, the "pure" decoder-only approach consistently lagged by roughly 2 points in WER and BLEU, highlighting the continued necessity of a dedicated speech encoder.

The Efficiency Gap

The most significant difference lies in resource consumption. DFP forces the decoder to perform self-attention over a combined sequence of speech and text, leading to quadratic computational growth. Cross-attention models proved to be the clear winners in efficiency, being approximately 4% faster and using 1.5x to 3x less GPU memory.

Table showing the speed and memory ratios for different architectures, highlighting the efficiency of cross-attention.

The Role of Masking and CTC

The study revealed a "causality paradox": while causal masking (restricting frames from seeing future context) hurts pure decoder-only models, it slightly benefits Decoder-prepend models. This implies that a separate encoder handles temporal context sufficiently, allowing the decoder to focus on mapping. Additionally, DFP appears to benefit more from CTC-based sequence compression, making it a potentially robust choice when extreme audio compression is required.

Conclusion

While prepending is a convenient method for adapting existing LLMs, it is not an architectural upgrade for S2T tasks. For systems built from the ground up, cross-attention remains the more efficient and performant choice. Future research should investigate if these dynamics shift at the scale of 70B+ parameter models or in multi-task scenarios like simultaneous translation.

Summary updated

Abstract

Following the remarkable success of Large Language Models (LLMs) in NLP tasks, there is increasing interest in extending their capabilities to speech -- the most common form of communication. The most widespread approach to integrating speech into LLMs is dense feature prepending (DFP), which prepends the projected speech representations to the textual representations, allowing end-to-end training with a speech encoder. This raises questions about the need for a sophisticated speech encoder for DFP and how its performance compares with a standard encoder-decoder (i.e., cross-attention) architecture. We compare DFP and cross-attention under a variety of configurations, such as CTC compression, sequence-level knowledge distillation, on monolingual, bilingual, and multilingual models. To perform a controlled architectural comparison, we train all models from scratch rather than using large pretrained models and use comparable data and parameter settings, testing speech-to-text recognition (ASR) and translation (ST) on MuST-C v1.0 and CoVoST2 datasets. Despite the wide adoption of DFP, our results do not indicate a clear advantage of DFP over cross-attention.
BibTeX
@inproceedings{lam2025prepending,
  author = {Tsz Kin Lam and Marco Gaido and Sara Papi and Luisa Bentivogli and Barry Haddow},
  title = {{Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.153},
}
Figure