TL;DR
This paper tackles the problem of reconstructing missing data in X-ray microbeam recordings of speech production, a common issue due to tracking limitations. They propose a novel method using Gaussian mixture models to learn the underlying vocal tract shape dynamics, enabling accurate prediction of missing articulator positions even over extended periods. This work is crucial for maximizing the utility of valuable, yet imperfect, speech production datasets.
What Problem Does This Paper Address?
Recording the intricate movements of the vocal tract during speech is essential for research in speech production, therapy, and processing. Techniques like X-ray microbeam and electromagnetic articulography (EMA) track small pellets attached to articulators. However, these methods are prone to "mistracks" or complete loss of signal for certain pellets, resulting in missing data sequences. Discarding these imperfect recordings is wasteful given the high cost and difficulty of data acquisition. This paper addresses the challenge of reconstructing these missing data sequences, aiming to recover the full articulatory trajectory even when significant portions are absent.
Methods
The core of the proposed method lies in learning a probabilistic model of the vocal tract's shape dynamics. Specifically, the authors model the joint density of all articulator positions as a Gaussian Mixture Model (GMM). This GMM acts as a "master density" that encodes all possible relationships between articulator movements.
When data for certain articulators is missing at a given time frame, the method leverages the learned GMM to compute the conditional distribution of the missing articulators given the observed ones. This conditional distribution effectively provides a mapping from the present to the missing data. The key innovation is that this approach can construct these mappings "on demand" for any arbitrary set of present and missing variables, unlike previous methods that assumed fixed sets of observed articulators. The GMM is trained offline using a complete dataset, and then at each frame, the conditional mean of the missing articulators is used for reconstruction, without explicit temporal smoothing.
Datasets
The primary dataset used for training and evaluation is the Wisconsin X-ray microbeam database (XRMB). This database contains simultaneous acoustic and X-ray microbeam recordings of speech, with pellet positions sampled at 147 Hz. The authors specifically use data from two speakers, jw11 and jw45, with varying percentages of mistracks. They partition this data into training and testing sets, with training sets containing 50,000 frames and testing sets over 10,000 frames.
Key Insights and Ideas
- Parsimony is Powerful: A surprisingly simple and parsimonious Gaussian mixture model (specifically, 32 components) was sufficient to achieve highly accurate reconstructions, suggesting that the underlying dynamics of vocal tract movement are well-captured by such models.
- Conditional Distributions as Mappings: The insight that conditional distributions derived from a joint density model can serve as flexible, on-demand mappings for reconstructing missing data is central to the paper's contribution. This allows for handling varying patterns of missing data.
- Articulator Coupling: The results highlight that some articulators are more predictable from others. For instance, tongue dorsum pellets (T2, T3, T4) are generally easier to reconstruct than the tongue tip (T1) or lips, likely due to their stronger coupling with other articulators.
- Temporal Independence Works: Remarkably, the method achieves accurate reconstructions without explicitly incorporating temporal smoothing or dependencies between frames, demonstrating the strength of the learned static density model.
Key Results
- The proposed GMM-based method achieves reconstruction errors of 1 to 1.5 mm for most articulators on the XRMB database, with some exceptions (e.g., lower lip for jw11, T1 for jw45) reaching up to 2.0 mm. This is considered very accurate, especially given that the measurement error in the XRMB data itself is around 0.5 mm.
- The GMM approach consistently outperforms non-parametric kernel density estimation (KDE) methods, achieving an average improvement of 0.2 mm in reconstruction error.
- The best performing GMM used 32 full-covariance components, demonstrating that a relatively small number of components can effectively model the complex articulatory data.
- Reconstructions of missing trajectories over 5-second periods were visually very close to the true trajectories, even when the latter exhibited drastic oscillations.
Limitations and Future Work
- Training Data Dependency: A key limitation acknowledged is that the current approach relies on estimating a density model from a complete dataset. This is problematic for scenarios where little or no complete data is available for training, such as near real-time applications.
- Offline Training: The current method requires offline training of the density model. Future work should explore reconstruction in more challenging situations, including near real-time scenarios.
- Temporal Information: While the current method doesn't use temporal smoothing, the authors suggest that incorporating temporal information could further improve reconstructions, potentially by addressing small discontinuities that can occur.
- Handling More Complex Missing Data: The paper primarily focuses on reconstructing single missing articulators, though the method is generally applicable to multiple missing articulators. Future work could explore more challenging scenarios with more extensive data loss.
Summary updated