TL;DR
This paper investigates how realistic errors from Automatic Speech Recognition (ASR) systems affect sentence embeddings, which are crucial for many Natural Language Processing (NLP) tasks. They introduce a novel simulator to inject ASR-like word substitution errors and find that while pre-trained neural encoders remain robust, simpler averaging methods degrade significantly.
What Problem Does This Paper Address?
Many NLP applications rely on converting text into numerical representations called "embeddings" that capture semantic meaning. These embeddings are fundamental for tasks like sentiment analysis, chatbots, and information retrieval. However, a critical assumption in most current methods is that the input text is perfectly transcribed. This is often not the case in real-world scenarios, especially when dealing with transcribed speech from Automatic Speech Recognition (ASR) systems, which are prone to word substitution errors. The Word Error Rate (WER) quantifies these errors, and typical ASR systems have WERs ranging from 10% to 35%. This paper addresses the gap in understanding how these realistic ASR errors impact the performance of various sentence embedding techniques, particularly in semantic similarity tasks.
Methods
The authors propose a new ASR error simulator to introduce plausible word substitution errors into text. This simulator considers both the phonemic distance (how similar words sound) and the semantic distance (how similar words' meanings are) between the original word and potential replacements. This is achieved by using phonetic transcriptions (ARPABET) and GloVe word embeddings. To manage the computational complexity of calculating pairwise word distances for large vocabularies, they limit their analysis to the 1000 most semantically similar words.
They then evaluate five different sentence embedding methods:
- Simple Unweighted Average: A basic method that averages word vectors.
- Smooth Inverse Frequency (SIF): Weights words based on their frequency, down-weighting common words and up-weighting rarer ones, and removes the first principal component of the embedding matrix to reduce noise.
- Unsupervised Smooth Inverse Frequency (uSIF): A refinement of SIF that makes it fully unsupervised by automatically computing hyperparameters and removing the first m principal components.
- Low-Rank Subspace: Represents sentences as an N-dimensional subspace rather than a single vector, using Principal Component Analysis (PCA).
- InferSent: A pre-trained neural network encoder trained on a Natural Language Inference (NLI) dataset, designed for universal sentence representations.
Datasets
The paper uses two standard datasets for evaluating semantic textual similarity (STS):
- SICK (Sentences Involving Compositional Knowledge): A dataset for evaluating compositional distributional semantics.
- STS-benchmark: A widely used benchmark for semantic textual similarity tasks.
These datasets were used in their original form and then corrupted with simulated ASR errors at varying Word Error Rates (WERs).
Key Insights and Ideas
- Pre-trained neural encoders are surprisingly robust: Methods like InferSent, which are trained on large datasets for related tasks, show remarkable resilience to ASR errors. This suggests that their learned representations are less sensitive to individual word substitutions.
- Simple averaging methods are fragile: Basic techniques like averaging word vectors perform well on clean text but degrade rapidly as ASR errors increase. This highlights the importance of more sophisticated embedding strategies for real-world applications.
- De-noising techniques offer a trade-off: SIF and uSIF, which employ de-noising by removing principal components, show a steeper initial decline in performance with errors but end up performing better than simple averages at higher WERs. This suggests that their de-noising mechanisms can mitigate some of the noise introduced by ASR errors.
- Phonetic and semantic similarity are key for error simulation: The authors' simulator's ability to incorporate both sound and meaning similarity makes the injected errors more realistic and thus the evaluation more meaningful.
Key Results
- When ASR errors are introduced (WER from 0% to 50%), all sentence embedding methods show a steady decline in similarity scores.
- Simple unweighted averaging (both with and without stop word removal) showed the least significant decline, maintaining a similarity score of around 0.74-0.77 at a 50% WER.
- SIF and uSIF showed a steeper decline, with similarity scores around 0.59-0.63 at 50% WER.
- InferSent consistently performed the best, showing the strongest performance on both clean and corrupted text, indicating its high robustness.
- The low-rank subspace method showed moderate performance, with its effectiveness varying slightly between the two datasets.
- In terms of Pearson Correlation Coefficient (PCC) on STS tasks, InferSent was the top performer. While simple averages were least impacted by errors in terms of absolute PCC drop, they performed worst at high WERs compared to more advanced methods. SIF and uSIF, despite being more impacted by errors, performed better than averages at high WERs.
Limitations and Future Work
- The current simulator only considers word substitution errors, not insertions or deletions, which are also common in ASR.
- The authors acknowledge that the low-rank subspace method's performance might be improved by tuning the subspace rank for specific datasets like SICK.
- Future work could involve extending the simulator to include other ASR error types and evaluating a wider range of sentence embedding methods and NLP tasks. Further investigation into why pre-trained encoders are so robust could also be beneficial.
Summary updated