Abstract
End-to-end speech-to-text (S2T) translation has emerged as the preferred architecture over traditional cascaded systems. However, these models—often composed of pre-trained acoustic encoders and linguistic decoders—face a fundamental challenge in fusing audio and text modalities. This paper investigates the "linguistic gap" in S2T encoders and introduces PromptST, a lightweight abstract prompt learning strategy. By inserting trainable soft prompts into the upper encoder layers, PromptST boosts the model's ability to capture complex syntax and semantics. Experimental results on the CoVoST-2 benchmark show an average improvement of +1.1 BLEU, demonstrating the effectiveness of this "parallel" augmentation approach.
Introduction: The Modality Fusion Mystery
Modern end-to-end S2T models typically leverage a pre-trained Automatic Speech Recognition (ASR) encoder and a pre-trained Machine Translation (MT) decoder. While this hybrid approach is effective, the encoder is primarily optimized for acoustic features (phonemes and sounds), which may not align with the high-level linguistic structures (syntax and semantics) required for high-quality translation. The authors suspected that this discrepancy limits the performance of end-to-end systems compared to their text-based counterparts.
Probing the Linguistic Plateau
To understand how these models process information, researchers developed Speech-Senteval, a 10GB benchmark designed to probe linguistic knowledge across different model layers. The probing tasks range from basic word content to complex sentence structure.
The analysis revealed a striking "Linguistic Plateau." Unlike standard text-to-text (T2T) models, where linguistic comprehension increases with depth, S2T encoders often see a stagnation or decline in their ability to represent syntactic and semantic information in the upper layers.
As shown in Figure 1, while T2T models (right column) show consistent gains in accuracy as depth increases, S2T models (left column) struggle to maintain this trajectory in the "Top" layers, particularly for syntactic and semantic tasks. This confirms that the encoder is born from ASR and remains focused on "hearing" rather than "comprehending."
PromptST: Bridging the Gap with Abstract Prompts
To address this plateau without the computational cost of full retraining, the authors proposed PromptST. This method utilizes Abstract Prompt Learning, where small, trainable "soft prompts" (continuous vectors) are inserted into the attention modules of the encoder's upper half.
As illustrated in Figure 4, these prompts act as an auxiliary workspace. They help the encoder focus on abstract linguistic features while preserving the acoustic knowledge established in the lower layers. The authors argue that this approach is like a parallel circuit: it provides auxiliary capacity to the model without disturbing the original information flow, unlike standard "series" adapters which can sometimes bottleneck data.
Experimental Results
PromptST was evaluated primarily on the CoVoST-2 dataset across four language pairs: English to German (De), Catalan (Ca), Arabic (Ar), and Turkish (Tr).
- Translation Performance: PromptST consistently outperformed strong baselines. On the CoVoST-2 test set, it achieved an average improvement of +1.1 BLEU over the base model and +0.4 BLEU over a model that was simply trained for longer ("Continue Train").
- Robustness: The strategy proved particularly effective in "extremely low-resource" settings (less than 100 hours of data), showing that the prompts help the model make the most of limited linguistic signals.
- Error Reduction: The models showed a significant average decline of 3.5 points in Translation Error Rate (TER) across all tested languages.
- Qualitative Gains: Analysis indicated that PromptST is particularly good at translating high-frequency words and shorter sentences—the essential components of daily oral communication.
Conclusion and Future Work
PromptST demonstrates that the linguistic limitations of ASR-based encoders can be mitigated through targeted, lightweight prompting. By treating the prompts as a parallel circuit, the model gains the capacity to process abstract linguistic information without sacrificing its acoustic foundations. Future research will explore the acoustic side of the modality discrepancy and investigate the potential of parallel adapter structures to further enhance end-to-end speech translation.
Summary updated