Acoustic-phonetic approach for automatic evaluation of spoken grammar

Om D. Deshmukh, Ashish VermaInterspeech 2008

TL;DR

This paper introduces a novel approach to automatically evaluate spoken grammar by integrating acoustic-phonetic features with existing speech recognition systems. This method significantly improves the accuracy of identifying grammatical errors in spoken language, a crucial step for developing more effective language learning tools and efficient assessment systems.

What Problem Does This Paper Address?

Evaluating spoken grammar is a key component of assessing overall spoken language proficiency, particularly for non-native speakers. Current automated systems often struggle with this task for a few reasons: 1) Automatic Speech Recognition (ASR) systems, while improving, still have limitations in accurately transcribing spontaneous speech. 2) Language Models (LMs) used in ASR tend to penalize grammatically incorrect sentences, making it harder for the ASR to even recognize them in the first place. This means that existing systems might miss grammatical errors or misinterpret them. The paper aims to bridge this gap by leveraging acoustic-phonetic information to refine ASR outputs and more reliably detect grammatical inaccuracies.

Methods

The core of the proposed approach lies in enhancing the output of a standard ASR system. Instead of solely relying on the ASR's transcription, the system uses acoustic-phonetic features to re-evaluate the speech signal.

Here's a breakdown of the key techniques:

  • Prompt-Response Paradigm: The system presents a prompt (which may contain a grammatical error) to the candidate. The candidate is expected to identify the error and provide a grammatically correct response.
  • Restricted Language Model (LM): The ASR system uses a specialized LM that includes common grammatically correct and incorrect variations of the expected response. This helps guide the ASR towards plausible outputs.
  • Acoustic Parameter Extraction: The paper introduces a novel acoustic parameter called Onset Coherence (OC). OC aims to distinguish between the onsets of fricatives, stops, and vowels without needing to compute periodicity information. This is achieved by analyzing energy differences across frequency channels at a regular frame rate.
  • Other acoustic parameters used include Zero Crossing Rate (ZCR), which helps distinguish frication, and Spectral Energy Ratio (SER), which differentiates vocalic from non-vocalic regions.
  • Temporal Region Identification: The system identifies potential areas of confusion in the speech signal based on the ASR's word-level alignments and the known acoustically similar sentences in the LM.
  • Re-analysis and Refinement: The acoustic parameters are computed over these identified temporal regions. This information is then used to refine the ASR's output, either by confirming the recognized word or by correcting it based on the acoustic evidence. For instance, if the ASR misinterprets "kill" as "killed," acoustic analysis of the relevant segment can help determine if the /d/ sound is actually present.

Datasets

The paper describes experiments conducted on a dataset derived from real-life spoken grammar assessment tests.

  • Database: Candidates were presented with 20 prompts randomly selected from a larger set of 204 prompts designed to discriminate between good and bad grammar.
  • Human Evaluation: Responses were independently evaluated by four human experts, achieving an inter-human agreement of 82%. Only utterances with unanimous human ratings were used for the experiments.
  • Test Data: The evaluation focused on 17 unique prompts and a total of 182 utterances where sentences in the prompt-specific LMs were acoustically similar and dissimilarities were localized.

Key Insights and Ideas

  • Acoustic-Phonetic Features are Crucial for Grammar: The paper strongly suggests that simply relying on ASR transcriptions is insufficient for accurate spoken grammar evaluation. Subtle acoustic cues, like the presence or absence of a final /d/ or /s/, carry vital grammatical information.
  • Novelty of Onset Coherence (OC): The introduction of OC as a parameter to differentiate onsets of different sound classes (fricatives, stops, vowels) without complex periodicity analysis is a significant contribution. This parameter, along with ZCR and SER, provides a more nuanced understanding of the speech signal.
  • Targeted Re-analysis Improves Accuracy: By focusing acoustic analysis on specific temporal regions identified as potentially problematic by the ASR, the system achieves higher efficiency and accuracy compared to a brute-force approach.
  • Bridging the ASR-LM Gap: The method effectively addresses the inherent limitations of LMs in ASR by using acoustic evidence to override or confirm ASR decisions, especially in cases of acoustic ambiguity.

Key Results

The proposed acoustic-phonetic approach led to significant improvements:

  • Speech Recognition Accuracy: Increased by 5.8% (from 88.4% to 94.2%).
  • Spoken Grammar Evaluation Accuracy: Improved by 8.2% (from 74.4% to 82.6%).

This demonstrates that integrating acoustic information directly enhances the system's ability to correctly assess grammatical correctness.

Limitations and Future Work

  • Limitations:
    • The current approach is evaluated on a specific subset of prompts where acoustical similarities are localized. Its performance on more complex or globally different grammatical errors might need further investigation.
  • The paper acknowledges that ASR systems still have limitations, particularly with spontaneous speech.
  • Future Work:
    • The authors suggest extending the technique to detect Grammatically Incorrect Constructs (GICs) in free speech, which is a more challenging task as ASR systems heavily favor grammatically correct LMs.
  • They propose that if categories of grammatical errors are pre-defined, the proposed method could be used to automatically detect them.

Summary updated

BibTeX
@inproceedings{deshmukh2008acoustic,
  author = {Om D. Deshmukh and Ashish Verma},
  title = {{Acoustic-phonetic approach for automatic evaluation of spoken grammar}},
  booktitle = {Interspeech},
  year = {2008},
}
Figure