Introduction
Transformer-based speech models have achieved remarkable performance, yet the internal mechanisms by which they integrate context to resolve linguistic ambiguities remain poorly understood. Unlike text models, where inputs are discrete tokens, speech models process continuous audio frames, making "context mixing" more complex. This study investigates how different architectures—specifically encoder-only versus encoder-decoder models—resolve French homophones, such as livre (book) versus livres (books), which sound identical but require grammatical cues for correct transcription.
Methodology
To quantify how information flows from "cue" words (e.g., determiners like le or les) to ambiguous target words, the researchers adapted several context-mixing metrics:
- Value Zeroing (VZ): Measures the change in a target representation when a specific input cue is zeroed out. This is the study's preferred metric for identifying functional dependencies.
- Attention Norm (AN): A refined version of attention that considers vector magnitudes, offering a more reliable signal than raw attention weights.
- Probing: Diagnostic classifiers were trained on frozen layer representations to determine at what stage the model "identifies" grammatical features like singular or plural.
- Input Ablation: The researchers physically silenced audio cues or removed text prefixes to observe the resulting drop in model confidence.
Data and Experimental Setup
The study utilized the French portion of the Common Voice corpus. Using dependency parsing, the authors extracted 1,000 pairs across three grammatical patterns: Determiner-Noun, Pronoun-Verb, and Subject-Verb. These patterns provide clear, localized dependencies where a specific cue word dictates the spelling of a homophonous target.
Results: The Architectural Divide
The findings reveal a fundamental divergence in processing strategies between model types.
Encoder-Only Models (XLSR)
In models like wav2vec 2.0 (XLSR), disambiguation occurs within the audio representation itself. Probing accuracy for grammatical number spikes in the middle layers (layers 10–15), suggesting a hierarchical processing structure where lower layers handle acoustics, middle layers integrate syntax, and top layers prepare for output.
Encoder-Decoder Models (Whisper)
In contrast, models like Whisper largely ignore grammatical cues within the encoder. The encoder remains primarily "acoustic," showing little specialized information about grammatical number. Instead, the "heavy lifting" is offloaded to the text-based decoder, which exhibits near-perfect probing accuracy from its very first layer.
Ablation and Context Sensitivity
Ablation experiments confirmed this labor division. For Whisper, silencing the audio cue (SC) had a negligible impact on transcription confidence. However, removing the text cue (ST) from the decoder prefix caused a massive drop in probability. Conversely, for encoder-only models, silencing the audio cue significantly reduced confidence, proving their reliance on integrated audio context.
Discussion and Limitations
The study highlights that raw attention weights are often uninformative, whereas Value Zeroing provides a clearer picture of functional dependencies. While these results clarify how models handle local homophony in French, real-world language often involves more complex, long-distance dependencies. Future research should investigate whether these architectural patterns persist across different languages, larger model scales, and varied tasks such as speech translation.
Summary updated