Abstract
As Large Language Models (LLMs) evolve into multimodal systems, integrating speech features has become a primary research focus. Two dominant architectural patterns have emerged: the traditional encoder-decoder approach using cross-attention and the more recent "Dense Feature Prepending" (DFP) method common in decoder-only LLMs. This paper provides a controlled empirical comparison of these methods, evaluating their performance, efficiency, and architectural requirements. Our findings indicate that while DFP is popular for its simplicity, it offers no performance advantage over cross-attention and introduces significant overhead in memory usage and generation latency.
Introduction
The dominance of decoder-only Large Language Models in natural language processing has led to a surge in "multimodal" adaptations. In the realm of Speech-to-Text (S2T), the most prevalent integration method is Dense Feature Prepending (DFP). This approach involves projecting speech features into the LLM's embedding space and prepending them directly to the text prompt, effectively treating speech as a sequence of continuous tokens.
Historically, S2T systems relied on encoder-decoder architectures where the two modalities interacted via a dedicated cross-attention mechanism. Because modern S2T research often utilizes massive, disparate pretrained models, it has been difficult to isolate whether the success of DFP is due to the architecture itself or the scale of the underlying LLMs. This study addresses this gap by conducting a "clean" head-to-head comparison using models trained from scratch under identical conditions.
Architectural Paradigms
To evaluate the impact of modality integration, the researchers compared three distinct architectural flavors:
- Cross-Attention (Encoder-Decoder): The classical approach where a dedicated decoder interacts with the output of a speech encoder through a cross-attention mechanism. This keeps the text and speech sequences distinct during processing.
- Decoder-Only (Shallow Prepend): A simplified approach where raw speech features—subjected only to basic downsampling—are prepended directly to text embeddings. This model lacks a deep, specialized speech encoder.
- Decoder-Prepend (Deep Prepend): A hybrid approach where a deep speech encoder processes the audio first. The resulting high-level features are then prepended to the text embeddings for the decoder to process as a single sequence.
Additionally, the study investigated the role of Audio Causal Masking. This tested whether speech frames should be restricted to "seeing" only preceding frames (causal) or allowed to attend to the entire audio segment (non-causal) during the decoding process.
Experimental Setup
The models were benchmarked on two primary datasets to ensure multilingual and task-specific robustness:
- MuST-C v1.0: English speech translated into eight languages, alongside English Automatic Speech Recognition (ASR).
- CoVoST2: A large-scale multilingual dataset featuring 21 non-English languages translated into English.
To ensure a fair comparison, all models were sized between 65 million and 150 million parameters and trained from scratch.
Key Insights
The Necessity of the Speech Encoder
The results demonstrate that the "Decoder-Only" approach (lacking a deep speech encoder) performs significantly worse than the other two architectures. This suggests that raw audio features are too complex for a standard text decoder to process effectively without sophisticated preprocessing. A dedicated encoder is essential for extracting the phonetic and linguistic nuances required for high-quality S2T.
Performance Parity
Contrary to the hype surrounding decoder-only multimodal models, DFP does not provide a quality boost. In the MuST-C English-Spanish benchmark, Cross-Attention achieved a 30.2 BLEU score, while Decoder-Prepend reached 30.7 BLEU. In ASR tasks, the performance was nearly identical, with Word Error Rates (WER) hovering between 10.3 and 10.4. In short, the choice of integration method has a negligible impact on final translation or transcription accuracy.
The Efficiency Tax
While performance is comparable, the efficiency gap is stark. Prepending speech features significantly extends the input sequence length. Because Transformer attention complexity grows quadratically with sequence length, DFP models are inherently slower and more resource-intensive.
Compared to the Cross-Attention baseline, the Decoder-Prepend model was 4% slower and required 59% more GPU memory. The Decoder-Only model fared even worse, consuming three times the memory of the cross-attention baseline. For deployment scenarios, cross-attention remains the superior choice for throughput and cost-effectiveness.
The Role of Causality
The impact of causal masking depends heavily on the architecture. In Decoder-Only models, applying causal masking to speech frames significantly hurts performance. However, in Decoder-Prepend models—which benefit from a separate encoder—the system actually sees a slight performance benefit from causal masking in the decoder stage.
Conclusion
This empirical study clarifies the trade-offs in multimodal LLM design. While Dense Feature Prepending is a straightforward way to extend LLMs to speech, it is not a "better" architecture in terms of accuracy and carries a heavy penalty in computational efficiency. Cross-attention remains a highly competitive and more efficient alternative for Speech-to-Text tasks.
Future research should investigate whether these dynamics hold at the scale of multi-billion parameter models. Additionally, exploring advanced sequence compression or linear-attention mechanisms may help bridge the efficiency gap currently inherent in prepending-based architectures.
Summary updated