NAIST Simultaneous Speech-to-Text Translation System for IWSLT 2022

Ryo Fukuda, Yuka Ko, Yasumasa Kano, Kosuke Doi, Hirotaka Tokuyama, Sakriani Sakti, Katsuhito Sudoh, Satoshi NakamuraACL 2022

TL;DR

This paper presents NAIST's approach to simultaneous speech-to-text translation for IWSLT 2022, focusing on end-to-end systems for English-to-German and English-to-Japanese. They introduce an adaptive segmentation policy based on "Bilingual Prefix Alignment" to improve translation quality and reduce latency, outperforming traditional wait-k baselines.

What Problem Does This Paper Address?

Simultaneous speech translation (SimulST) aims to translate spoken language in real-time, presenting a significant challenge due to the need to balance translation accuracy with minimal delay. Traditional approaches often rely on fixed policies for deciding when to read incoming speech and when to output translated text, which can lead to suboptimal decisions, especially with languages that have different word orders. This paper tackles the problem of creating more adaptive and effective SimulST systems, particularly for language pairs with structural differences like English and Japanese.

Methods

The core of NAIST's system is an end-to-end Simultaneous Machine Translation (SimulMT) model. They employ two key innovations:

  • Bilingual Prefix Alignment: This technique extracts pairs of source language speech prefixes and their corresponding target language translations from training data. This is crucial for fine-tuning their offline speech translation (ST) models.
  • Adaptive Segmentation Policy: Instead of fixed rules, they train a "boundary predictor" that adaptively decides when to segment incoming speech. This predictor uses the bilingual prefix alignment data to learn optimal READ/WRITE actions, aiming to be more effective than fixed policies, especially for languages like English and Japanese.

The system is built using the Transformer architecture and fine-tuned in two stages: first on full segment pairs, then on the extracted bilingual prefix pairs.

Datasets

  • MuST-C v2: This multilingual speech translation corpus, extracted from TED talks, was used for training the offline ST models. The En-De dataset contained approximately 250k segments, and the En-Ja dataset had about 330k segments.
  • IWSLT 2022 Evaluation Campaign: The systems were evaluated on the tst-COMMON dataset for En-De and En-Ja.

Key Insights and Ideas

  • Adaptive policies are crucial for distant language pairs: The paper highlights that while adaptive policies offer improvements, their effectiveness is particularly pronounced for language pairs with significant structural differences (like English and Japanese), where fixed policies struggle.
  • Bilingual Prefix Alignment is key for fine-tuning: Fine-tuning with these carefully extracted prefix pairs significantly improved performance, especially for English-to-German.
  • Data filtering is essential for Japanese: For English-to-Japanese, simply fine-tuning with all prefix pairs led to under-translation. Filtering out "unbalanced" pairs (where source speech prefixes were long but target translations were short) was critical to improving performance. This suggests that the alignment between source speech and target text is more complex for SOV languages like Japanese compared to SVO languages like English and German.
  • Latency-accuracy trade-off: The results consistently show a trade-off between latency (Average Lagging) and accuracy (BLEU), with different thresholds for the boundary predictor allowing for adjustments based on desired performance characteristics.

Key Results

  • Outperformed baselines: NAIST's proposed models consistently outperformed wait-k baselines across various latency regimes for both En-De and En-Ja.
  • Significant BLEU improvement in En-De: The system achieved a notable improvement of over 10 BLEU points in the low-latency regime for English-to-German.
  • En-Ja challenges: While improvements were seen in En-Ja, they were smaller than in En-De. The fine-tuned model initially performed worse than non-fine-tuned models, but data filtering significantly improved its performance, suggesting the importance of data quality for this language pair.
  • Boundary predictor accuracy: The boundary predictor achieved accuracies below 68% on the tst-COMMON dataset, indicating the inherent difficulty of binary classification at the acoustic frame level, especially for En-Ja where recall was extremely low.

Limitations and Future Work

  • Boundary Predictor Performance: The authors acknowledge the difficulty of binary classification for the boundary predictor, particularly for En-Ja, where recall was very low. This suggests room for improvement in adaptive segmentation.
  • Fine-tuning for En-Ja: The initial poor performance of the fine-tuned model for En-Ja, even before filtering, points to the complexity of aligning different language structures and the need for more sophisticated fine-tuning strategies.
  • Future Work: The paper implicitly suggests further research into improving boundary prediction accuracy and developing more robust fine-tuning methods, especially for language pairs with significant structural differences. The success of data filtering for En-Ja also opens avenues for exploring more advanced data selection and augmentation techniques.

Summary updated

BibTeX
@inproceedings{fukuda2022naist,
  author = {Ryo Fukuda and Yuka Ko and Yasumasa Kano and Kosuke Doi and Hirotaka Tokuyama and Sakriani Sakti and Katsuhito Sudoh and Satoshi Nakamura},
  title = {{NAIST Simultaneous Speech-to-Text Translation System for IWSLT 2022}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure