Intrinsic spectral analysis for zero and high resource speech recognition

Aren Jansen, Samuel Thomas, Hynek HermanskyInterspeech 2012

TL;DR

This paper introduces a novel nonlinear extension to Intrinsic Spectral Analysis (ISA), a method that learns speech features by uncovering the underlying manifold structure of acoustic data. This new approach significantly improves performance in both "zero-resource" speech recognition (where no transcribed data is available) and high-resource scenarios, demonstrating its versatility and effectiveness.

What Problem Does This Paper Address?

Traditional speech recognition systems heavily rely on vast amounts of transcribed data, which are unavailable for many languages and domains. This creates a significant performance gap when applying these systems to "low-resource" settings. While semi-supervised learning has been explored, this paper focuses on improving the "front-end" of speech recognition – the feature extraction process. Specifically, it addresses how to extract more robust and discriminative acoustic features from untranscribed speech, particularly in the extreme "zero-resource" case where only the raw audio is available. The goal is to create features that are less sensitive to speaker and channel variations and better capture the intrinsic phonetic structure of speech, making them useful even without prior linguistic knowledge.

Methods

The core of this paper is the evaluation of a nonlinear out-of-sample extension for Intrinsic Spectral Analysis (ISA). ISA is based on the idea that speech sounds lie on a low-dimensional manifold within a high-dimensional acoustic space. Manifold learning algorithms are used to uncover this structure.

The paper utilizes the Laplace-Beltrami operator as a theoretical foundation, which measures the smoothness of functions defined on a manifold. In practice, this is approximated using a graph Laplacian constructed from unlabeled acoustic feature vectors. The key innovation here is the use of a nonlinear out-of-sample extension for ISA. Previously, ISA often relied on linear approximations, which struggled with manifolds that had high extrinsic curvature. This paper employs a Radial Basis Function (RBF) kernel within a manifold regularization framework to learn nonlinear projection maps. This allows ISA to capture more complex manifold structures.

The process involves:

  1. Constructing a graph Laplacian from unlabeled acoustic data.
  2. Solving a generalized eigenvalue problem to obtain basis functions.
  3. Projecting novel feature vectors onto these learned intrinsic basis functions using the RBF kernel.

This nonlinear approach is then applied to both unsupervised and supervised speech recognition tasks.

Datasets

  • TIMIT Corpus: This is the primary dataset used for evaluation. It's a widely used corpus for speech research, containing read American English speech with phonetic transcriptions.
  • Used for: Evaluating both unsupervised spoken term discovery and supervised phonetic recognition.
  • Specific subsets/uses:
  • For spoken term discovery, word examples of at least 0.35s duration and 6 characters were extracted (approx. 11k examples across 3,745 word types).
  • For phonetic recognition, the standard 39 phoneme classes were used.

Key Insights and Ideas

  • Nonlinearity is Crucial: The paper strongly suggests that the speech manifold has significant extrinsic curvature, and a nonlinear ISA approach is essential to effectively capture its structure, leading to substantial gains over linear ISA.
  • Zero-Resource Powerhouse: Nonlinear ISA demonstrates remarkable performance in the zero-resource spoken term discovery task, significantly outperforming traditional front-ends like PLP and MFCC, and narrowing the gap with fully supervised methods. This highlights its potential for languages with limited or no transcribed data.
  • Complementary to Supervised Methods: Even in high-resource settings, nonlinear ISA features are not only competitive with but also complementary to traditional features like MFCC and PLP when combined using evidence theory. This suggests ISA can enhance existing supervised systems.
  • Intrinsic Structure Matters: The success of ISA, even when using simple graph construction (e.g., cosine distance for similarity), indicates that the topological structure of unlabeled data alone is sufficient to learn representations that encode crucial information for speech recognition.
  • Data-Driven Advantage: ISA offers a data-driven approach that can potentially be applied "as-is" to new languages or tasks without needing to redesign signal processing algorithms, a significant advantage over traditional methods.

Key Results

  • Spoken Term Discovery (Zero-Resource):
    • Nonlinear ISA-LogMelSpec (cosine) achieved an Average Precision (AP) of 48.5%.
  • This is a significant improvement over Linear ISA-LogMelSpec (25.6%) and traditional MFCC (33.8%) and PLP (34.8%).
  • It represents a nearly 15% absolute improvement in cross-speaker word matching AP compared to PLP/MFCC, closing about a third of the gap to supervised posteriors (75.4%).
  • Cosine distance for graph construction was found to be superior to Euclidean distance.
  • Phonetic Recognition (High-Resource):
    • Nonlinear ISA-LogMelSpec achieved 76.0% accuracy.
  • Nonlinear ISA-PLP achieved 76.7% accuracy.
  • These results match the performance of traditional MFCC (76.8%) and PLP (77.0%) front-ends.
  • When combined with PLP using Dempster-Shafer theory, ISA features led to improved accuracy (DS: PLP + Nonlinear ISA-LogMelSpec: 78.5%, DS: PLP + Nonlinear ISA-PLP: 78.0%), demonstrating complementarity.

Limitations and Future Work

  • Parameter Tuning: While the authors found parameter values (κ, ξ, σ) that generally worked well, performance was moderately dependent on κ (number of neighbors) and σ (RBF kernel width). They note that ideally, this dependence would be weak, but in practice, some cross-validation might be needed.
  • Computational Cost: Constructing the nearest neighbor graph and solving the generalized eigenvalue problem can be computationally intensive for very large datasets, although the paper notes this is an offline process and subsequent projection is linear time.
  • Future Work: The paper suggests that the data-driven nature of ISA, without requiring task-specific algorithm redesign, makes it a promising avenue for new languages and non-speech applications. The success in both zero and high-resource settings indicates its broad applicability.

Summary updated

BibTeX
@inproceedings{jansen2012intrinsic,
  author = {Aren Jansen and Samuel Thomas and Hynek Hermansky},
  title = {{Intrinsic spectral analysis for zero and high resource speech recognition}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure