Introduction
The relationship between physical articulation—the complex movements of the tongue, jaw, and soft palate—and the resulting acoustic signal is a fundamental aspect of human speech. For individuals with speech disorders or those recovering from surgical interventions such as tongue cancer treatment, this articulatory-acoustic relationship is often disrupted. Understanding and quantifying these disruptions is vital for effective rehabilitation and clinical assessment.
However, detecting anomalies in this relationship presents a significant challenge. Traditional diagnostic methods often focus on a single modality: either analyzing the physical motion via imaging or evaluating the audio signal. A patient’s tongue motion might appear within normal ranges in isolation, and their speech might sound intelligible, yet the specific mapping between their unique physical movements and the sounds produced may be incongruent. This paper proposes a novel framework for multi-modal anomaly detection, identifying breakdowns in the translation from 4D physical motion to acoustic outcomes.
Methodology
The researchers developed an unsupervised deep learning framework centered on the concept of cross-modal translation. The core idea is to "translate" high-dimensional internal tissue motion into a visual representation of sound.
Data Acquisition and Representation
The study utilizes tagged MRI, a specialized imaging modality that allows for pixel-level tracking of internal tissue deformation. Unlike standard MRI, tagged MRI provides the resolution necessary to monitor individual tongue muscles during speech, resulting in 4D motion fields (3D spatial coordinates over time).
The Translation Model
The framework employs a deep learning "translator" designed to predict a 2D Mel-spectrogram from the 4D motion fields. The researchers evaluated two primary architectures to handle the high-dimensional input:
- 3D Convolutional Neural Network (CNN): Designed to capture spatial features within the volumetric motion data.
- Hybrid 3D CNN + Longformer: This architecture combines the spatial processing of a CNN with a transformer-based Longformer to better capture long-distance temporal correlations, which are essential for understanding the flow of speech over time.
Unsupervised Anomaly Detection
Crucially, the model is trained exclusively on data from healthy individuals. By learning the "standard" mapping between motion and sound, the model establishes a baseline for healthy speech production. When the model is presented with motion data from a patient with an articulatory disorder, it is expected to struggle with the reconstruction, as the patient's motion-to-sound mapping is "out-of-distribution."
To quantify this, a One-Class Support Vector Machine (SVM) analyzes the quality of the reconstructed spectrogram. If the reconstruction fidelity is significantly lower than the baseline established by the healthy training set, the system flags the instance as an anomaly.
Experimental Setup
The study utilized a specialized dataset involving 39 participants:
- 36 healthy individuals (used for training and baseline testing).
- 3 tongue cancer patients (used for anomaly detection testing).
Participants were recorded speaking specific phrases—"a geese" and "a souk"—while synchronized with a metronome during MRI scanning. The resulting audio was preprocessed into $64 \times 64$ Mel-spectrograms using the Librosa library, providing a target for the motion-to-speech translation.
Results and Discussion
The framework demonstrated a clear ability to distinguish between healthy articulatory patterns and those affected by pathology.
Reconstruction Fidelity
For healthy subjects, the 3D CNN + Transformer model achieved a 2D Pearson’s correlation (Corr2D) of 0.814. In contrast, the correlation for patients was lower at 0.786, indicating the model's relative inability to accurately map patient motion to the expected sound. Similarly, the Perceptual Evaluation of Speech Quality (PESQ) scores were higher for healthy individuals (1.638) than for patients (1.611).
Detection Accuracy
The effectiveness of the anomaly detection was measured using the Area Under the Curve (AUC). The Transformer-based model achieved an AUC of 0.82, while the standard 3D CNN model achieved an AUC of 0.79. These results suggest that the inclusion of temporal modeling via the Longformer significantly improves the system's ability to identify pathological speech patterns.
Conclusion and Future Work
This research introduces a powerful paradigm for clinical diagnosis: using a generative model’s failure as a diagnostic signal. By translating high-dimensional 4D motion fields into the more interpretable domain of audio spectrograms, the researchers have created a method to pinpoint where the articulatory-acoustic link breaks down.
While the results are promising, the study is limited by its small patient cohort (n=3) and the use of highly controlled, short phrases. Future work will focus on expanding the dataset to include a wider variety of speech disorders and more complex, fluid speech patterns. Additionally, the authors aim to perform localized analyses to determine if specific regions of the tongue motion field can be directly linked to specific acoustic errors, potentially providing a roadmap for targeted surgical or speech therapy interventions.
Summary updated