TL;DR
This paper introduces a novel deep learning approach for segmenting air-tissue boundaries (ATBs) in real-time MRI videos of the vocal tract. By leveraging Fully Convolutional Networks (FCNs) for semantic segmentation, the method achieves more accurate and detailed ATB predictions compared to existing techniques, crucial for speech production research.
What Problem Does This Paper Address?
Understanding how the vocal tract moves during speech is fundamental to speech production research. Real-time Magnetic Resonance Imaging (rtMRI) offers a non-invasive way to capture these dynamic movements. However, to analyze these videos effectively, it's essential to accurately identify the boundaries between air (in the vocal tract cavity) and tissue (the vocal tract walls). This air-tissue boundary (ATB) segmentation is a critical preprocessing step for various applications, including developing text-to-speech systems, studying articulatory control, and analyzing vocal tract morphology. Previous methods, while functional, often struggled with precision, detail, and handling the inherent noise and artifacts in rtMRI data.
Methods
The core of this work is the application of Fully Convolutional Networks (FCNs) for semantic segmentation. Unlike traditional CNNs that output a single class label, FCNs produce an output map of the same spatial dimensions as the input image, assigning a class label (air or tissue) to each pixel. This allows for pixel-level understanding of the image.
The authors employ a supervised learning approach, training separate FCN models for three distinct vocal tract contours (C1, C2, C3) identified in the midsagittal plane. The FCN architecture is inspired by the VGG-16 model, known for its strong performance in semantic segmentation.
Following the FCN's initial segmentation, a series of post-processing steps are applied to refine the predicted boundaries:
- Image Enhancement: This involves two stages:
- Morphological Filling: Operations like dilation are used to fill small holes within the segmented regions. A logical OR operation is then applied to preserve the original shape while filling.
- Image Smoothing: A moving average filter is used to remove sharp edges from the binary mask, resulting in smoother contours.
- Contour Prediction: The Canny edge detection algorithm is used to identify edge points, and a concave hull algorithm is employed to connect these points into closed contours.
- Contour Pruning: This crucial step refines the predicted contours to accurately represent the ATB within the vocal tract. Specific procedures are used for upper and lower ATBs, involving identifying key anatomical landmarks (like the velum tip) and applying smoothing techniques to handle challenging regions, such as those near the tongue base and lower teeth.
Datasets
The study utilizes the USC-TIMIT corpus, a well-established database of rtMRI videos of the upper airway. This corpus contains speech data from multiple subjects. For their experiments, the authors selected 16 rtMRI videos (one per sentence) from each of four subjects (two female, two male), resulting in a total of 64 videos. Manual annotations of three major contours (C1, C2, C3) were performed using a MATLAB GUI. The dataset was split into training, development, and testing sets using a four-fold cross-validation approach, with approximately 2900 training images per fold.
Key Insights and Ideas
- Semantic Segmentation for ATB: Applying FCNs for semantic segmentation proves highly effective for ATB prediction, outperforming traditional grid-based methods by capturing richer spatial information.
- Individual Contour Models: Training separate FCNs for each contour (C1, C2, C3) rather than a single model for the entire boundary significantly improves accuracy and avoids ambiguity.
- Post-processing is Key: The combination of morphological filling, smoothing, and specialized contour pruning is essential for transforming raw FCN outputs into realistic and accurate ATBs.
- Beyond the Vocal Tract: Unlike some prior unsupervised methods, the FCN approach can segment ATBs both inside and outside the vocal tract, offering a more comprehensive view.
Key Results
The proposed FCN-based approach demonstrated significant improvements over the baseline Maeda Grid (MG) scheme:
- Lower ATB: An average error reduction of 8.87% in Dynamic Time Warping (DTW) distance.
- Upper ATB: An average error reduction of 9.65% in DTW distance.
- Pixel Classification Accuracy: An average accuracy of 99.05% across all subjects.
Qualitatively, the FCN-based method produced smoother and more detailed contours, particularly in challenging regions. Figure 6 visually highlights instances where the FCN method outperforms the MG scheme.
Limitations and Future Work
- Low Resolution Impact: The authors acknowledge that the low resolution of the rtMRI images can make it difficult for the model to precisely differentiate contact points between upper and lower ATBs, leading to misclassifications. This directly impacts the accuracy of the predicted contours.
- Future Directions: The authors suggest future work could involve developing entirely automated boundary detection schemes by modifying the FCN architecture. This implies exploring more advanced network designs or training strategies to further enhance robustness and accuracy, especially in low-resolution scenarios.
Summary updated