PromptST: Abstract Prompt Learning for End-to-End Speech Translation

Tengfei Yu, Liang Ding, Xuebo Liu, Kehai Chen, Meishan Zhang, Dacheng Tao, Min Zhang

Abstract

End-to-end speech-to-text (S2T) translation has emerged as the preferred architecture over traditional cascaded systems. However, these models—often composed of pre-trained acoustic encoders and linguistic decoders—face a fundamental challenge in fusing audio and text modalities. This paper investigates the "linguistic gap" in S2T encoders and introduces PromptST, a lightweight abstract prompt learning strategy. By inserting trainable soft prompts into the upper encoder layers, PromptST boosts the model's ability to capture complex syntax and semantics. Experimental results on the CoVoST-2 benchmark show an average improvement of +1.1 BLEU, demonstrating the effectiveness of this "parallel" augmentation approach.

Introduction: The Modality Fusion Mystery

Modern end-to-end S2T models typically leverage a pre-trained Automatic Speech Recognition (ASR) encoder and a pre-trained Machine Translation (MT) decoder. While this hybrid approach is effective, the encoder is primarily optimized for acoustic features (phonemes and sounds), which may not align with the high-level linguistic structures (syntax and semantics) required for high-quality translation. The authors suspected that this discrepancy limits the performance of end-to-end systems compared to their text-based counterparts.

Probing the Linguistic Plateau

To understand how these models process information, researchers developed Speech-Senteval, a 10GB benchmark designed to probe linguistic knowledge across different model layers. The probing tasks range from basic word content to complex sentence structure.

The analysis revealed a striking "Linguistic Plateau." Unlike standard text-to-text (T2T) models, where linguistic comprehension increases with depth, S2T encoders often see a stagnation or decline in their ability to represent syntactic and semantic information in the upper layers.

Linguistic accuracy across encoder depths for S2T vs. T2T models.

As shown in Figure 1, while T2T models (right column) show consistent gains in accuracy as depth increases, S2T models (left column) struggle to maintain this trajectory in the "Top" layers, particularly for syntactic and semantic tasks. This confirms that the encoder is born from ASR and remains focused on "hearing" rather than "comprehending."

PromptST: Bridging the Gap with Abstract Prompts

To address this plateau without the computational cost of full retraining, the authors proposed PromptST. This method utilizes Abstract Prompt Learning, where small, trainable "soft prompts" (continuous vectors) are inserted into the attention modules of the encoder's upper half.

Architecture of the PromptST model with layer-specific prompts.

As illustrated in Figure 4, these prompts act as an auxiliary workspace. They help the encoder focus on abstract linguistic features while preserving the acoustic knowledge established in the lower layers. The authors argue that this approach is like a parallel circuit: it provides auxiliary capacity to the model without disturbing the original information flow, unlike standard "series" adapters which can sometimes bottleneck data.

Experimental Results

PromptST was evaluated primarily on the CoVoST-2 dataset across four language pairs: English to German (De), Catalan (Ca), Arabic (Ar), and Turkish (Tr).

  • Translation Performance: PromptST consistently outperformed strong baselines. On the CoVoST-2 test set, it achieved an average improvement of +1.1 BLEU over the base model and +0.4 BLEU over a model that was simply trained for longer ("Continue Train").
  • Robustness: The strategy proved particularly effective in "extremely low-resource" settings (less than 100 hours of data), showing that the prompts help the model make the most of limited linguistic signals.
  • Error Reduction: The models showed a significant average decline of 3.5 points in Translation Error Rate (TER) across all tested languages.
  • Qualitative Gains: Analysis indicated that PromptST is particularly good at translating high-frequency words and shorter sentences—the essential components of daily oral communication.

Conclusion and Future Work

PromptST demonstrates that the linguistic limitations of ASR-based encoders can be mitigated through targeted, lightweight prompting. By treating the prompts as a parallel circuit, the model gains the capacity to process abstract linguistic information without sacrificing its acoustic foundations. Future research will explore the acoustic side of the modality discrepancy and investigate the potential of parallel adapter structures to further enhance end-to-end speech translation.

Summary updated

Abstract

An end-to-end speech-to-text (S2T) translation model is usually initialized from a pre-trained speech recognition encoder and a pre-trained text-to-text (T2T) translation decoder. Although this straightforward setting has been shown empirically successful, there do not exist clear answers to the research questions: 1) how are speech and text modalities fused in S2T model and 2) how to better fuse the two modalities? In this paper, we take the first step toward understanding the fusion of speech and text features in S2T model. We first design and release a 10GB linguistic probing benchmark, namely Speech-Senteval, to investigate the acoustic and linguistic behaviors of S2T models. Preliminary analysis reveals that the uppermost encoder layers of the S2T model can not learn linguistic knowledge efficiently, which is crucial for accurate translation. Based on the finding, we further propose a simple plug-in prompt-learning strategy on the uppermost encoder layers to broaden the abstract representation power of the encoder of S2T models. We call such a prompt-enhanced S2T model PromptST. Experimental results on four widely-used S2T datasets show that PromptST can deliver significant improvements over a strong baseline by capturing richer linguistic knowledge. Benchmarks, code, and scripts are freely available at https://github.com/ytf-philp/PromptST.
BibTeX
@inproceedings{yu2023promptst,
  author = {Tengfei Yu and Liang Ding and Xuebo Liu and Kehai Chen and Meishan Zhang and Dacheng Tao and Min Zhang},
  title = {{PromptST: Abstract Prompt Learning for End-to-End Speech Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.627},
}
Figure