Advances in combined electro-optical palatography

Peter Birkholz, Philippe Dächert, Christiane Neuschaefer-RubeInterspeech 2012

TL;DR

Researchers have developed a novel device that combines electropalatography (EPG) with optical distance sensing to simultaneously measure tongue-palate contact and tongue shape in the oral cavity. This integrated approach shows promise for improving silent speech recognition by providing richer articulatory data than either method alone.

What Problem Does This Paper Address?

Understanding speech production often requires detailed information about how the tongue interacts with the palate. Electropalatography (EPG) is a well-established technique that maps where the tongue touches the roof of the mouth using an artificial palate embedded with electrodes. However, EPG can't tell us about the tongue's shape or position when it's not touching the palate, which is crucial for many speech sounds. Previous attempts to measure tongue shape using optical sensors faced limitations in spatial resolution and integration with EPG. This paper addresses the need for a more comprehensive system that captures both contact and distance information, aiming to improve our ability to analyze and potentially reconstruct speech movements.

Methods

The core innovation is the development of a combined electro-optical palatograph. This device utilizes a thin, custom-made acrylic pseudopalate that houses both traditional EPG electrodes and novel reflective optical sensors.

  • Pseudopalate Design: The pseudopalate is thermoformed to fit the speaker's hard palate. It's manufactured using flexible circuit boards, which significantly reduces production time and cost compared to older methods.
  • EPG Integration: Standard EPG electrodes are embedded within the flexible circuits on the palate.
  • Optical Distance Sensing: Five reflective optical sensors are strategically placed along the midline of the palate. Each sensor consists of an LED and a phototransistor. The LED illuminates the tongue surface, and the phototransistor measures the reflected light. The intensity of the reflected light is inversely proportional to the distance between the sensor and the tongue.
  • Sensor Selection: The researchers experimented with different combinations of LEDs and phototransistors to find sensors that were small, flat, had a sufficient measuring range (at least 25 mm), and were relatively insensitive to saliva. They settled on a specific combination (LED VSMY2850 and phototransistor VEMT2020) that offered a good balance of performance and physical characteristics.
  • Measurement System: EPG contact data and optical sensor data are captured simultaneously at 100 Hz. The optical sensors are switched sequentially to prevent cross-talk. Data from both systems are synchronized and processed on a laptop.

Datasets

The paper describes an evaluation using a corpus of recorded speech from a single speaker. This corpus included:

  • Logatoms of the form [bVbVbVbV] with various vowels (V ∈ {a:,e,i,o,u:,ε:,:,y:}).
  • Logatoms of the form [aCaCaCaCaCa] with various consonants (C ∈ {p,t,k,f,s,f,ç,x,1}).
  • A 148-second read passage from a German book.

This corpus was used to generate average EPG patterns and tongue contours for selected speech sounds and to evaluate vowel recognition rates.

Key Insights and Ideas

  • Synergistic Data: Combining EPG contact data with optical distance measurements provides a more complete picture of articulation than either modality alone. This is particularly useful for understanding sounds where the tongue doesn't make full contact with the palate.
  • Manufacturing Advancements: The use of flexible circuits for integrating both EPG electrodes and optical sensors represents a significant step towards more accessible and cost-effective palatography devices.
  • Vowel Recognition Improvement: While not dramatic, the combined sensor input did lead to a higher vowel recognition rate compared to using EPG or optical data independently. This suggests that the richer information captured by the combined system is indeed beneficial for speech analysis tasks.
  • Limitations of Current Optical Sensor Placement: The study identified potential issues with the placement of optical sensors for certain sounds (e.g., [f]), suggesting that more sensors or a different arrangement might be needed for precise contour reconstruction in all cases.

Key Results

  • The developed electro-optical palatograph successfully captured both tongue-palate contact and mid-sagittal tongue contours.
  • For vowel recognition using feed-forward neural networks, a system using only EPG contact data achieved a 64.9% recognition rate, and a system using only optical sensor data also achieved 64.9%.
  • When the combined EPG and optical sensor data were used as input, the vowel recognition rate increased to 69.1%.
  • The improvement in recognition rate was more pronounced for frequent vowels ([a:], [i:]) than for infrequent ones ([u:], [y:]).

Limitations and Future Work

  • Limited Sensor Coverage: The authors acknowledge that the current arrangement of four optical sensors along the midline might not be sufficient for accurately reconstructing the tongue contour for all consonants, suggesting a need for more sensors or a different spatial distribution.
  • Calibration Challenges: The tongue contours for some vowels ([a:], [u:]) appeared distorted, possibly due to inaccurate sensor calibration. Refining the calibration process is an area for future work.
  • Speaker Dependency: The evaluation was performed on a single speaker, and the results might not generalize directly to other speakers without recalibration or further training.
  • Classifier Sophistication: The authors suggest that using more sophisticated classifiers beyond simple feed-forward networks could further improve real-world speech recognition performance.
  • 3D Reconstruction Goal: The ultimate goal is to reconstruct the 3D shape of the oral cavity in real-time, which would require a more extensive sensor network and advanced modeling techniques.

Summary updated

BibTeX
@inproceedings{birkholz2012advances,
  author = {Peter Birkholz and Philippe Dächert and Christiane Neuschaefer-Rube},
  title = {{Advances in combined electro-optical palatography}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure