Abstract
Previous work has shown that it is possible to improve speech recognition by learning acoustic features from paired acoustic-articulatory data, for example by using canonical correlation analysis (CCA) or its deep extensions. One limitation of this prior work is that the learned feature models are difficult to port to new datasets or domains, and articulatory data is not available for most speech corpora. In this work we study the problem of acoustic feature learning in the setting where we have access to an external, domain-mismatched dataset of paired speech and articulatory measurements, either with or without labels. We develop methods for acoustic feature learning in these settings, based on deep variational CCA and extensions that use both source and target domain data and labels. Using this approach, we improve phonetic recognition accuracies on both TIMIT and Wall Street Journal and analyze a number of design choices.
Introduction
Modern speech recognition typically relies solely on acoustic recordings. However, speech is fundamentally a physical process; knowing how the articulators (tongue, lips, jaw) move provides a rich, complementary signal to the audio. Articulatory data acts as a powerful "supervisory signal," forcing models to learn features that are physically grounded and potentially more robust than those learned from audio alone.
The primary challenge is that articulatory data is difficult and expensive to collect, requiring specialized equipment like X-ray microbeams. Consequently, it is absent from the massive datasets used to train production speech models. While previous research utilized Canonical Correlation Analysis (CCA) to learn better features from paired data, a significant "missing link" remains: domain mismatch. This paper addresses how to transfer insights from a small, specialized articulatory "source" dataset to a completely different, audio-only "target" dataset with different recording conditions and vocabularies.
Methodology
The proposed framework builds upon Deep Variational Canonical Correlation Analysis with Private Variables (VCCAP) and introduces extensions for cross-domain adaptation.
VCCAP and VAEP
The VCCAP model assumes that paired acoustic and articulatory data are generated from three latent variables: a "shared" variable representing underlying phonetic intent, and two "private" variables representing noise or characteristics unique to either the audio or the sensors.
To handle new datasets, the authors introduce a Variational Autoencoder with Private variables (VAEP). This architecture allows the model to distinguish between universal acoustic features and domain-specific characteristics (e.g., recording environment noise).
Joint Training
The most effective approach involves joint training of the feature extractor and the speech recognizer—a bidirectional LSTM using Connectionist Temporal Classification (CTC) loss. This end-to-end optimization ensures that the "articulatory-aware" features are specifically tuned for the final phonetic recognition task.
Experimental Setup
The models were evaluated using one source domain and two target domains:
- Source Domain (Articulatory): The University of Wisconsin X-ray Microbeam Dataset (XRMB), containing acoustic recordings paired with 2D coordinates of 8 pellets attached to speakers' articulators.
- Target Domains (Acoustic only):
- TIMIT: A standard phonetic recognition dataset.
- Wall Street Journal (WSJ): A larger, more complex dataset to test scalability.
Results and Discussion
Performance was measured using the Phonetic Error Rate (PER). The results demonstrate that articulatory information provides complementary data that persists even in fully supervised settings.
Key Findings
- Internal Transfer (XRMB): When transferring between speakers within the same dataset, the PER dropped significantly from a 12.9% baseline to 4.9% with jointly trained VCCAP.
- Cross-Domain Transfer (TIMIT & WSJ): In the XRMB $\rightarrow$ TIMIT setting, the best model reduced PER from 20.9% to 20.2%. On the much larger WSJ dataset, the PER improved from 6.8% to 6.3%.
- The Necessity of Private Variables: Unsupervised transfer was found to fail without private variables to account for the specific noise and channel characteristics of the target domain.
Conclusion
This work demonstrates that "hallucinating" articulatory information during training can significantly improve speech recognition systems. While the benefits are most pronounced on smaller datasets, the gains on WSJ prove the method's relevance even when the target dataset is much larger than the source articulatory data. Future work will explore whether these phonetic-level improvements translate to full word-level Automatic Speech Recognition (ASR) systems and address the computational complexity of joint VCCAP-VAEP-LSTM training.
Summary updated