Abstract
The rise of Large Language Models (LLMs) has shifted the paradigm of Speech-to-Text (S2T) research toward "dense feature prepending" (DFP), where audio features are directly concatenated to text prompts. This paper presents a controlled, empirical comparison between DFP and the classical cross-attention architecture. Results indicate that while both methods achieve similar translation and recognition quality, DFP is substantially less efficient, requiring more memory and offering slower inference speeds.
Introduction
Modern S2T systems often adapt decoder-only LLMs by "injecting" speech through prepending audio features. While effective for leveraging massive pre-trained models, it remains unclear if DFP is an architectural improvement or simply a beneficiary of scale. Existing literature often lacks fair comparisons due to varying pre-trained backbones and parameter counts. This study levels the playing field by training models from scratch to isolate the impact of the connection mechanism itself.
Architectural Configurations
The study evaluates three primary architectures to determine which is truly more effective for Automatic Speech Recognition (ASR) and Speech Translation (ST):
- Cross-Attention: The standard encoder-decoder setup where a speech encoder interacts with the decoder via a dedicated cross-attention layer.
- Decoder-only: A pure LLM-style approach where downsampled speech features are prepended to text embeddings without a deep speech encoder.
- Decoder-prepend: A hybrid model using a deep speech encoder whose output is prepended to the decoder's input sequence.
The researchers also explored auxiliary techniques such as CTC compression and Sequence-level Knowledge Distillation (seqKD) to enhance performance across these configurations.
Key Findings
Quality Parity
When controlling for parameters and data, Decoder-prepend and Cross-Attention perform almost identically. On the CoVoST2 benchmark, both architectures achieved approximately 29.7 BLEU using Conformer encoders. This suggests that the "modern" prepending method does not inherently improve speech understanding over traditional cross-attention. In contrast, the "pure" decoder-only approach consistently lagged by roughly 2 points in WER and BLEU, highlighting the continued necessity of a dedicated speech encoder.
The Efficiency Gap
The most significant difference lies in resource consumption. DFP forces the decoder to perform self-attention over a combined sequence of speech and text, leading to quadratic computational growth. Cross-attention models proved to be the clear winners in efficiency, being approximately 4% faster and using 1.5x to 3x less GPU memory.
The Role of Masking and CTC
The study revealed a "causality paradox": while causal masking (restricting frames from seeing future context) hurts pure decoder-only models, it slightly benefits Decoder-prepend models. This implies that a separate encoder handles temporal context sufficiently, allowing the decoder to focus on mapping. Additionally, DFP appears to benefit more from CTC-based sequence compression, making it a potentially robust choice when extreme audio compression is required.
Conclusion
While prepending is a convenient method for adapting existing LLMs, it is not an architectural upgrade for S2T tasks. For systems built from the ground up, cross-attention remains the more efficient and performant choice. Future research should investigate if these dynamics shift at the scale of 70B+ parameter models or in multi-task scenarios like simultaneous translation.
Summary updated