TL;DR
This paper investigates how acoustic cues like constrictions, bursts, and friction in word-final /t/ sounds influence both human perception and automatic speech recognition (ASR) systems. The findings reveal that while humans and ASR systems rely on similar acoustic cues, humans are more sensitive to subtle variations, and ASR systems can be misled by cues like voicing. This work offers insights for improving ASR by better modeling human speech perception.
What Problem Does This Paper Address?
Conversational speech is often characterized by reduced pronunciations compared to their citation forms. For instance, word-final /t/ sounds in Dutch can be significantly altered or even absent. This variability poses a challenge for both human listeners trying to understand speech and for Automatic Speech Recognition (ASR) systems aiming to transcribe it accurately. While previous research has explored segmental-level reductions (like complete deletion of a sound), this paper focuses on the sub-segmental properties – the finer acoustic details within a sound – that contribute to whether a word-final /t/ is perceived as present or absent by humans and how ASR systems classify it. Understanding these sub-segmental cues is crucial for developing more robust models of both human and machine speech processing.
Methods
The researchers employed a multi-faceted approach involving both human listeners and an ASR system.
- Human Perception Annotation: Two phoneticians, native Dutch speakers, listened to recorded conversational Dutch speech and classified word-final /t/ sounds into categories like "strong," "weak," "assimilated," or "absent."
- Sub-segmental Property Annotation: The same phoneticians meticulously annotated the acoustic properties of these /t/ sounds, focusing on:
- Constriction: Presence or absence, and whether it was voiced or unvoiced.
- Burst: Presence, absence, or whether it was a single or multiple burst.
- Friction: Whether it was smooth or abrupt (for /t/ followed by an alveolar consonant), and if it started simultaneously with the burst.
- Alveolar Friction: Presence or absence (for /t/ not followed by an alveolar obstruent).
- ASR System: A standard ASR system, using forced alignment with acoustic phone models, was used to classify the word-final /t/s as present or absent.
- Statistical Modeling: Mixed-effects logistic regressions were used to determine which sub-segmental properties significantly predicted human perceptual presence and ASR classification.
Datasets
The study utilized the ERNESTUS CORPUS OF SPONTANEOUS DUTCH, a collection of ten spontaneous dialogues. From this corpus, 486 word tokens (representing 141 word types) ending in /t/ in their citation form were extracted. These tokens were selected based on specific segmental contexts (preceded by a vowel or /n/, followed by a vowel, fricative, or plosive). The corpus contained both function and content words, and speakers were from a homogeneous group.
Key Insights and Ideas
- Shared Cues, Different Sensitivity: Both humans and the ASR system rely on the presence of a constriction, a burst, and alveolar friction to identify word-final /t/. However, humans are more attuned to subtle variations like weak bursts and smoothly starting friction, which the ASR system struggles with.
- Voicing is a Red Herring for ASR: Surprisingly, the ASR system was more likely to classify voiced /t/ sounds as absent. This is attributed to the ASR's acoustic models being trained on read speech, where voiced final /t/ (often realized as /d/) is less common. This suggests that voicing is a misleading cue for the current ASR system.
- Constriction is King (for Humans): The presence of a constriction was a significant predictor for human perception of /t/ presence, distinguishing between both strong/weak vs. absent, and weak vs. absent realizations.
- Abrupt Friction Aids Weakness: For tokens classified as "weakly" present, an abruptly starting friction was more likely than a smoothly starting one, suggesting a specific acoustic signature for this type of reduction.
Key Results
- Human Perception: The presence of a constriction and a burst (single or multiple) significantly predicted the perceptual presence of word-final /t/. Alveolar friction also played a significant role.
- ASR Classification: Similar to humans, the presence of a constriction and a burst favored /t/ being labeled as present. However, voiced /t/s were more often classified as absent by the ASR. Alveolar friction was also a significant predictor for ASR presence.
- Agreement: The agreement between human perceptual classification and ASR classification was moderate, with higher agreement for clearly "strong" and "absent" tokens (around 80%) and lower agreement for "weak" and "assimilated" tokens (around 66-73%).
- ASR vs. Human: The ASR system classified more /t/s as absent (31.7%) compared to human phoneticians (21.8%), potentially due to the ASR's training on read speech and its minimum segment length requirement.
Limitations and Future Work
- Naive Listeners: The study used expert phoneticians for annotation. The authors suggest that future work should involve naive listeners to see if these findings hold for less experienced listeners.
- ASR Model Specificity: The ASR system used was a standard one. Future work could explore how different ASR architectures or training data might better capture sub-segmental cues.
- Improving ASR: The paper suggests that ASR systems could be improved by first identifying potential /t/ intervals based on canonical transcriptions and then applying more detailed analysis informed by human perception of sub-segmental properties. This could involve better modeling of fine acoustic cues like weak bursts and friction onset.
Summary updated