Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI

Xiaofeng Liu, Fangxu Xing, Jiachen Zhuo, Maureen Stone, Jerry L. Prince, Georges El Fakhri, Jonghye Woo

Introduction

The relationship between physical articulation—the complex movements of the tongue, jaw, and soft palate—and the resulting acoustic signal is a fundamental aspect of human speech. For individuals with speech disorders or those recovering from surgical interventions such as tongue cancer treatment, this articulatory-acoustic relationship is often disrupted. Understanding and quantifying these disruptions is vital for effective rehabilitation and clinical assessment.

However, detecting anomalies in this relationship presents a significant challenge. Traditional diagnostic methods often focus on a single modality: either analyzing the physical motion via imaging or evaluating the audio signal. A patient’s tongue motion might appear within normal ranges in isolation, and their speech might sound intelligible, yet the specific mapping between their unique physical movements and the sounds produced may be incongruent. This paper proposes a novel framework for multi-modal anomaly detection, identifying breakdowns in the translation from 4D physical motion to acoustic outcomes.

Methodology

The researchers developed an unsupervised deep learning framework centered on the concept of cross-modal translation. The core idea is to "translate" high-dimensional internal tissue motion into a visual representation of sound.

Data Acquisition and Representation

The study utilizes tagged MRI, a specialized imaging modality that allows for pixel-level tracking of internal tissue deformation. Unlike standard MRI, tagged MRI provides the resolution necessary to monitor individual tongue muscles during speech, resulting in 4D motion fields (3D spatial coordinates over time).

The Translation Model

The framework employs a deep learning "translator" designed to predict a 2D Mel-spectrogram from the 4D motion fields. The researchers evaluated two primary architectures to handle the high-dimensional input:

  1. 3D Convolutional Neural Network (CNN): Designed to capture spatial features within the volumetric motion data.
  2. Hybrid 3D CNN + Longformer: This architecture combines the spatial processing of a CNN with a transformer-based Longformer to better capture long-distance temporal correlations, which are essential for understanding the flow of speech over time.

Unsupervised Anomaly Detection

Crucially, the model is trained exclusively on data from healthy individuals. By learning the "standard" mapping between motion and sound, the model establishes a baseline for healthy speech production. When the model is presented with motion data from a patient with an articulatory disorder, it is expected to struggle with the reconstruction, as the patient's motion-to-sound mapping is "out-of-distribution."

To quantify this, a One-Class Support Vector Machine (SVM) analyzes the quality of the reconstructed spectrogram. If the reconstruction fidelity is significantly lower than the baseline established by the healthy training set, the system flags the instance as an anomaly.

Experimental Setup

The study utilized a specialized dataset involving 39 participants:

  • 36 healthy individuals (used for training and baseline testing).
  • 3 tongue cancer patients (used for anomaly detection testing).

Participants were recorded speaking specific phrases—"a geese" and "a souk"—while synchronized with a metronome during MRI scanning. The resulting audio was preprocessed into $64 \times 64$ Mel-spectrograms using the Librosa library, providing a target for the motion-to-speech translation.

Results and Discussion

The framework demonstrated a clear ability to distinguish between healthy articulatory patterns and those affected by pathology.

Reconstruction Fidelity

For healthy subjects, the 3D CNN + Transformer model achieved a 2D Pearson’s correlation (Corr2D) of 0.814. In contrast, the correlation for patients was lower at 0.786, indicating the model's relative inability to accurately map patient motion to the expected sound. Similarly, the Perceptual Evaluation of Speech Quality (PESQ) scores were higher for healthy individuals (1.638) than for patients (1.611).

Detection Accuracy

The effectiveness of the anomaly detection was measured using the Area Under the Curve (AUC). The Transformer-based model achieved an AUC of 0.82, while the standard 3D CNN model achieved an AUC of 0.79. These results suggest that the inclusion of temporal modeling via the Longformer significantly improves the system's ability to identify pathological speech patterns.

Conclusion and Future Work

This research introduces a powerful paradigm for clinical diagnosis: using a generative model’s failure as a diagnostic signal. By translating high-dimensional 4D motion fields into the more interpretable domain of audio spectrograms, the researchers have created a method to pinpoint where the articulatory-acoustic link breaks down.

While the results are promising, the study is limited by its small patient cohort (n=3) and the use of highly controlled, short phrases. Future work will focus on expanding the dataset to include a wider variety of speech disorders and more complex, fluid speech patterns. Additionally, the authors aim to perform localized analyses to determine if specific regions of the tongue motion field can be directly linked to specific acoustic errors, potentially providing a roadmap for targeted surgical or speech therapy interventions.

Summary updated

Abstract

Understanding the relationship between tongue motion patterns during speech and their resulting speech acoustic outcomes-i.e., articulatory-acoustic relation-is of great importance in assessing speech quality and developing innovative treatment and rehabilitative strategies. This is especially important when evaluating and detecting abnormal articulatory features in patients with speech-related disorders. In this work, we aim to develop a framework for detecting speech motion anomalies in conjunction with their corresponding speech acoustics. This is achieved through the use of a deep cross-modal translator trained on data from healthy individuals only, which bridges the gap between 4D motion fields obtained from tagged MRI and 2D spectrograms derived from speech acoustic data. The trained translator is used as an anomaly detector, by measuring the spectrogram reconstruction quality on healthy individuals or patients. In particular, the cross-modal translator is likely to yield limited generalization capabilities on patient data, which includes unseen out-of-distribution patterns and demonstrates subpar performance, when compared with healthy individuals. A one-class SVM is then used to distinguish the spectrograms of healthy individuals from those of patients. To validate our framework, we collected a total of 39 paired tagged MRI and speech waveforms, consisting of data from 36 healthy individuals and 3 tongue cancer patients. We used both 3D convolutional and transformer-based deep translation models, training them on the healthy training set and then applying them to both the healthy and patient testing sets. Our framework demonstrates a capability to detect abnormal patient data, thereby illustrating its potential in enhancing the understanding of the articulatory-acoustic relation for both healthy individuals and patients.
BibTeX
@misc{liu2024speech,
  author = {Xiaofeng Liu and Fangxu Xing and Jiachen Zhuo and Maureen Stone and Jerry L. Prince and Georges El Fakhri and Jonghye Woo},
  title = {{Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI}},
  year = {2024},
  eprint = {2402.06984},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.06984},
}
Figure