Analyzing analytical methods: The case of phonology in neural models of spoken language

Grzegorz Chrupala, Bertrand Higy, Afra Alishahi

Abstract

Given the fast development of analysis techniques for NLP and speech processing systems, few systematic studies have been conducted to compare the strengths and weaknesses of each method. As a step in this direction we study the case of representations of phonology in neural network models of spoken language. We use two commonly applied analytical techniques, diagnostic classifiers and representational similarity analysis, to quantify to what extent neural activation patterns encode phonemes and phoneme sequences. We manipulate two factors that can affect the outcome of analysis. First, we investigate the role of learning by comparing neural activations extracted from trained versus randomly-initialized models. Second, we examine the temporal scope of the activations by probing both local activations corresponding to a few milliseconds of the speech signal, and global activations pooled over the whole utterance. We conclude that reporting analysis results with randomly initialized models is crucial, and that global-scope methods tend to yield more consistent results and we recommend their use as a complement to local-scope diagnostic methods.

Introduction

As neural networks for speech and language processing become more complex, researchers have developed a suite of "probing" tools to understand what these black boxes are actually learning. However, the reliability of these analytical tools themselves remains a subject of debate. A primary concern in the field of interpretability is whether these auxiliary tasks measure the model's actual learning or are merely picking up on the inherent statistical properties of the input signal. If a randomly initialized (untrained) model performs well on a diagnostic task, then a high score on a trained model might not signify genuine learning—it might just be a baseline property of the architecture. This paper systematically compares different analytical frameworks to establish best practices for interpreting neural representations of speech.

Methodology

The authors compare two primary analytical techniques:

  1. Diagnostic Classifiers (Probing): Training a simple linear model to predict a label (like a phoneme) from the network's activations.
  2. Representational Similarity Analysis (RSA): A technique that compares the "similarity structure" of neural activations to the similarity of the actual labels (e.g., phonetic distance).

They manipulate two key variables:

  • The Effect of Learning: Comparing activations from fully trained models against randomly initialized versions of the same architecture.
  • Temporal Scope: Examining Local representations (individual frames of speech) versus Global representations (activations pooled over an entire utterance using mean or attention-based pooling).

The study utilizes two major speech datasets to train and evaluate three different model architectures: Transformer-ASR (trained on Librispeech), and RNN-VGS and RNN-ASR (trained on Flickr8K Audio Caption Corpus).

Results and Discussion

The "Random RNN" Trap and Transformer Behavior

A key insight of the study is the "Random RNN" trap: in RNNs, even random weights can create complex, non-random activation patterns as information flows deeper into the network. This makes it easy for a diagnostic classifier to "decode" phonemes from an untrained RNN, potentially leading researchers to overestimate what the model has learned. Conversely, randomly initialized Transformers do not show an increase in decodability with depth, as their connectivity prevents information from "accumulating" in the same way without trained weights.

Comparison of diagnostic classifiers and RSA across layers for RNN-ASR, showing local vs. global pooling and trained vs. random models.
Analysis results for Transformer-ASR, illustrating the different behavior of randomly initialized Transformer layers compared to RNNs.

Global vs. Local Scope

Methods that look at the whole utterance (Global) are much better at distinguishing between trained and random models than those that look at individual frames (Local). Global-scope methods yielded much more consistent results across different analytical techniques. Furthermore, RSA proved to be more sensitive to the actual effects of training at the local level. While a classifier could potentially find patterns in random noise, RSA's requirement for a specific similarity structure made it harder to achieve high scores without genuine learning.

Phonological Localization

In trained models, phonological information tends to be strongest in the "pre-final" layers—not the very first or very last layers. This suggests that the network refines these representations in its middle stages before transforming them for the final task objective.

Conclusion

The authors conclude that reporting analysis results with randomly initialized models is crucial for any interpretability study to establish a valid baseline. Global-scope methods tend to yield more consistent results and are recommended as a complement to local-scope diagnostic methods. While attention-based pooling can be erratic, simple mean pooling is often a more robust choice for researchers. Future work should explore whether these findings hold for higher-level linguistic features like syntax or semantics.

Code repository: github.com/gchrupala/analyzing-analytical-methods

Summary updated

Abstract

Given the fast development of analysis techniques for NLP and speech processing systems, few systematic studies have been conducted to compare the strengths and weaknesses of each method. As a step in this direction we study the case of representations of phonology in neural network models of spoken language. We use two commonly applied analytical techniques, diagnostic classifiers and representational similarity analysis, to quantify to what extent neural activation patterns encode phonemes and phoneme sequences. We manipulate two factors that can affect the outcome of analysis. First, we investigate the role of learning by comparing neural activations extracted from trained versus randomly-initialized models. Second, we examine the temporal scope of the activations by probing both local activations corresponding to a few milliseconds of the speech signal, and global activations pooled over the whole utterance. We conclude that reporting analysis results with randomly initialized models is crucial, and that global-scope methods tend to yield more consistent results and we recommend their use as a complement to local-scope diagnostic methods.
BibTeX
@inproceedings{chrupala2020analyzing,
  author = {Grzegorz Chrupala and Bertrand Higy and Afra Alishahi},
  title = {{Analyzing analytical methods: The case of phonology in neural models of spoken language}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.381},
}
Figure