Guiding Students to Investigate What Google Speech Recognition Knows about Language

David S. Touretzky, Christina Gardner-McCuneAAAI 2023 · DOI

TL;DR

This paper introduces a novel approach to teaching K-12 students about the complexities of speech recognition and human language by using a browser-based tool called SpeechDemo. By engaging students in experiments with the Google Web Speech API, the authors aim to foster a deeper understanding of how AI systems process language and the inherent biases they may exhibit.

What Problem Does This Paper Address?

While children are increasingly interacting with speech recognition systems, they often lack an understanding of how these technologies function. This gap in knowledge prevents them from appreciating the intricate linguistic processes involved and the potential for AI decision-making to be influenced by biases. The paper addresses this by providing an educational framework and tools that allow students to actively investigate the capabilities and limitations of a real-world speech recognition system, specifically Google's Web Speech API. This empowers them to move beyond passive consumption of AI and become active explorers of its inner workings.

Methods

The core of the paper's methodology lies in the SpeechDemo tool, an open-source, browser-based application that leverages the Google Web Speech API. Unlike many demonstrations that only show the top-ranked transcription, SpeechDemo's key innovation is its ability to display multiple transcription hypotheses in rank order. This feature is crucial for revealing the inherent ambiguity in spoken language and how the AI system attempts to resolve it.

The paper also outlines a pedagogical approach that guides students through a series of experiments designed to probe different levels of linguistic knowledge:

  • Lexical Knowledge: Testing the system's vocabulary and its ability to handle non-words or slang.
  • Phonological Knowledge: Investigating how the system handles words that violate English phonotactic constraints and its mapping of non-words to plausible words.
  • Syntactic and Semantic Knowledge: Examining how context is used to disambiguate homophones and other ambiguous phrases.
  • Cultural Knowledge: Exploring how the system's understanding of idioms, sayings, and famous quotations influences transcription.
  • Biases: Investigating potential biases in the system, particularly concerning dialectical variations in speech.

Students are encouraged to design their own experiments and analyze the results, fostering critical thinking about AI and language.

Datasets

The paper does not explicitly mention specific datasets used for training the Google Web Speech API. However, it implies that the API is trained on massive corpora, and the experiments conducted by students are designed to reveal the behavior of this pre-trained system. The AI4K12 Guidelines are referenced as a framework for developing educational activities, but these are not datasets themselves.

Key Insights and Ideas

  • The Power of Multiple Hypotheses: Displaying multiple transcription candidates is essential for understanding the ambiguity of speech and how AI systems make decisions. This is a significant departure from typical demonstrations.
  • Language is More Than Just Words: Students can learn about phonology, syntax, semantics, and cultural nuances by simply interacting with a speech recognition tool.
  • AI Doesn't "Understand" Like Humans: The experiments reveal that Google's speech recognition, while powerful, relies on statistical inference rather than true semantic understanding, leading to occasional, sometimes humorous, errors.
  • Cultural Context Matters: Idioms, sayings, and even well-known quotes can significantly influence how speech recognition systems interpret utterances, highlighting the role of cultural knowledge.
  • The Pervasiveness of Bias: The paper touches upon the potential for speech recognition systems to exhibit biases, particularly against non-standard dialects, raising important questions about fairness and representation in AI.

Key Results

The paper reports on an initial implementation of the SpeechDemo tool in a middle school AI elective. The results indicate that:

  • Middle school students were able to successfully engage with the homophone disambiguation activities, demonstrating an understanding of the need for context.
  • Students enjoyed the process of "fooling Google" and making up their own sentences to test the system's limits.
  • A significant majority (88%) of students in a review session correctly identified sentences with homophone pairs as difficult for computers to understand.
  • Informal testing suggests Google's Web Speech API shows a strong preference for transcribing speech into formal Standard American English, even when dialectical variations are present.

Limitations and Future Work

  • Reproducibility: The authors acknowledge that as Google continuously improves its Web Speech API, the specific limitations revealed by some experiments may change, making perfect reproducibility difficult.
  • Worksheet Design: The initial implementation of the student worksheet had some issues with clarity and student adherence, which the authors plan to address in future iterations.
  • Scope of Bias Investigation: While the paper touches on biases, the authors state they have not conducted a rigorous investigation into this area.
  • Future Work: The authors plan to refine the activity worksheet and implement it in more classrooms. They also suggest that this "dissection" approach to teaching AI can be applied to other AI technologies beyond speech recognition.

Summary updated

BibTeX
@inproceedings{touretzky2023guiding,
  author = {David S. Touretzky and Christina Gardner-McCune},
  title = {{Guiding Students to Investigate What Google Speech Recognition Knows about Language}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26905},
}
Figure