TL;DR
This paper introduces and evaluates a low-cost, markerless 3D tracking system for studying speech articulation, specifically lip movements. By combining a consumer-grade depth sensor (Kinect-like) with a robust face-tracking algorithm, the system achieves promising accuracy comparable to established marker-based methods, paving the way for more accessible and patient-friendly speech analysis tools.
What Problem Does This Paper Address?
Studying the intricate movements of speech articulators (like lips, tongue, and jaw) is crucial for understanding speech disorders and developing effective therapies. However, existing methods, such as electromagnetic articulography (EMA) and optoelectronic systems, are often expensive, cumbersome for participants, and require lengthy preparation. While some marker-based systems have emerged to mitigate these issues, they still necessitate marker placement and calibration. This paper addresses the need for a truly markerless, low-cost, and user-friendly system that can accurately capture 3D articulatory movements, making advanced speech analysis more accessible for research and clinical applications, particularly in home-based therapy settings.
Methods
The core of this work lies in a markerless 3D tracking system built around a Primesense Carmine 1.09 sensor (a Kinect-like device) and the Intraface face-tracking algorithm.
- Depth and Color Data: The Primesense sensor provides simultaneous RGB color and depth video streams. The depth stream captures the distance of each pixel from the camera, enabling 3D reconstruction.
- Intraface Algorithm: This algorithm fits a deformable face model to the color image using SIFT (Scale-Invariant Feature Transform) texture descriptors. It's chosen for its robustness to illumination changes and its ability to generalize to unseen facial configurations. The model includes points for eyebrows, eyes, nose, and lips.
- 3D Point Extraction: For the points of interest (specifically eyebrows, nose, and outer lip contours), their 2D coordinates are obtained from the Intraface tracker. These 2D coordinates, along with the depth information from the Primesense sensor, are then used to calculate the 3D coordinates of these points.
- Marker-Based Ground Truth: A high-end Vicon Motion Systems MX3+ optoelectronic system with reflective markers served as the ground truth for comparison. 16 markers were precisely placed on the subjects' faces.
- Synchronization and Alignment: Both systems (Primesense/Intraface and Vicon) were used simultaneously and synchronized. A transformation (rotation and translation) was estimated to align the markerless 3D points with the Vicon coordinate system, using specific anatomical points as references.
- Articulatory Parameter Calculation: Beyond raw point trajectories, key articulatory parameters like lip width, lip opening, and lip protrusion were calculated from the tracked points.
- Error Metrics: The Root Mean Square Error (RMSE) was used to quantify the difference between the markerless and marker-based measurements for both point trajectories and articulatory parameters.
Datasets
The paper does not describe the use of pre-existing datasets for training. Instead, it details a custom data acquisition process:
- Participants: Two healthy subjects were recruited: one Italian native speaker and one French native speaker.
- Speech Corpus: For each language, a corpus consisting of 50 meaningful sentences and 100 meaningful words was used. The French sentences were from the Comberscure corpus, and the Italian sentences and words were from corpora defined by Bocca and Pellegrini.
- Acquisition Setup: Subjects were seated at a distance of 0.7-0.8 meters from the Primesense sensor, with the Vicon cameras positioned to capture the lower part of the face. Acquisitions were performed under constant and uniform illumination.
Key Insights and Ideas
- Feasibility of Markerless 3D Speech Articulation: The study demonstrates that a low-cost, markerless system can achieve accuracy comparable to expensive marker-based systems for tracking lip movements during speech. This is a significant step towards democratizing speech analysis.
- Intraface's Robustness: The choice of the Intraface algorithm proved effective, showing robustness to illumination changes and the ability to handle facial movements without requiring prior training on specific individuals.
- Trade-offs in Consumer-Grade Sensors: While promising, the results highlight that the resolution of consumer-grade depth sensors and the distance from the subject can impact accuracy. Lower video resolutions (320x240) and greater distances (0.7-0.8m) led to some indistinguishable depth variations, particularly in facial cavities.
- Importance of Distance and Resolution: The paper strongly recommends using higher video resolutions (at least 640x480) and closer acquisition distances (0.5-0.6m) for future experiments to improve accuracy.
- Potential for Speech Therapy: The markerless and low-cost nature of the system makes it highly suitable for integration into speech therapy applications, enabling remote monitoring, automated feedback, and more engaging home-based exercises.
Key Results
- RMSE for Lip Points: For most of the tracked lip points, the RMSE ranged between 1 and 3 mm.
- Articulatory Parameters (French Corpus): Mean RMSE values were around 2 mm for lip width, 2.72 mm for lip opening, and 1.36 mm for lip protrusion.
- Articulatory Parameters (Italian Corpus): Errors were generally higher, with values over 4 mm for protrusion. This was attributed to larger errors in the y-coordinate of specific lip points (L2, L3, L6) in the Italian corpus.
- Depth Accuracy: The Primesense sensor showed depth errors ranging from 0.7 mm to 2 mm depending on the distance from the object, with errors generally increasing with distance. These device-specific errors were found to be lower than the RMSEs obtained during speech acquisitions.
Limitations and Future Work
- Low Video Resolution: The authors acknowledge that the low video resolution (320x240) used in this experiment limited the ability to distinguish fine depth variations, potentially impacting accuracy.
- Acquisition Distance: The chosen acquisition distance (0.7-0.8m) was a compromise and could be optimized.
- Future Work Recommendations:
- Higher Resolution and Closer Distance: Future experiments should utilize higher video resolutions (e.g., 640x480) and acquire data at closer distances (0.5-0.6m) to improve accuracy.
- Phoneme-Level Analysis: Further analysis will focus on error patterns at the phoneme level to identify specific sounds for which the technique performs best or worst.
- Broader Applications: The promising results encourage further development for markerless, low-cost techniques for speech therapy, potentially leading to increased patient participation and better rehabilitation outcomes.
Summary updated