Acquisition and synchronization of multimodal articulatory data

Michael Aron, Nicolas Ferveur, Erwan Kerrien, Marie-Odile Berger, Yves LaprieInterspeech 2007

TL;DR

This paper presents a novel setup for precisely synchronizing multiple data streams (ultrasound, electromagnetic, and audio) used to track speech articulation. By developing an automatic synchronization method, they enable more accurate and reproducible studies of speech production, paving the way for better speech recognition and synthesis models.

What Problem Does This Paper Address?

Understanding how humans produce speech is crucial for advancing automatic speech processing, from recognition to creating more realistic talking avatars. This requires observing the speaker's vocal tract and face in detail. However, no single imaging technique can capture all aspects of articulation with sufficient spatial and temporal resolution without posing health risks. Existing methods often combine multiple modalities (like ultrasound for tongue shape and electromagnetic sensors for tongue position), but a major hurdle has been the difficulty in precisely synchronizing the data from these different sources. This temporal misalignment can lead to significant errors, especially when analyzing fast speech movements. The authors aim to solve this by creating an easily reproducible acquisition system with automatic, high-accuracy temporal synchronization.

Methods

The core of this paper lies in its automatic temporal synchronization strategy. The system couples three main modalities:

  • Ultrasound (US) Imaging: Captures the tongue's shape in a 2D midsagittal plane. The US machine stores the last 15 seconds of data in a cineloop, triggered by a footswitch.
  • Electromagnetic (EM) Sensors: Two sensors are attached to the tongue (tip and dorsum) to track their precise 3D positions. An additional EM sensor is mounted on the US probe for spatial calibration.
  • Audio Recording: Captures the speaker's voice.

The synchronization principle relies on an external event: an audio beep emitted by a control PC. This PC acts as the reference clock.

  1. EM-Audio Synchronization: The control PC emits a beep and records its clock time (Cbeep). The beep is also recorded in the audio signal (Tbeep). The delay between the PC and audio time scales is calculated. Simultaneously, the PC records its clock time (Cem) when the first EM sample is requested and uses the EM system's reported time for that sample (Tem) to calculate the delay between the PC and EM time scales. The difference between these two delays gives the EM-Audio synchronization delay.
  2. US Synchronization: The US data is synchronized using the US system's footswitch input. This input is controlled by a relay from the PC's parallel port. By registering the PC's clock value when the parallel port switch is activated, the last US frame can be temporally aligned with the PC's time scale.

Crucially, the authors emphasize measuring the sampling rate of each modality accurately, as manufacturer-provided values can have slight deviations that accumulate over time, leading to significant temporal shifts. They also highlight the importance of spatial calibration (previously developed by the authors) to express all data within a common coordinate system.

Datasets

Not discussed in the paper. The paper describes an acquisition setup and its validation using a French native speaker and a corpus of VCV utterances and two sentences.

Key Insights and Ideas

  • The critical need for precise temporal synchronization: Even small temporal misalignments between modalities can lead to significant errors in analyzing articulatory movements, especially for fast speech.
  • An external audio beep as a universal synchronization trigger: Using a simple audio beep from a control PC provides a robust and easily implementable method to align disparate data streams.
  • The importance of measuring actual sampling rates: Relying solely on theoretical sampling rates can introduce cumulative errors. Experimental measurement is essential for accuracy.
  • Reproducibility is key: The authors designed their setup to be easily reproducible, aiming to lower the barrier for other researchers to acquire similar multimodal data.

Key Results

  • The developed synchronization method was experimentally validated.
  • The residual delay between EM data and audio data was found to be consistently less than the EM acquisition period (25 ms).
  • The delay between the US recording process and the parallel port switch was experimentally determined to be 52 ms with a standard deviation of 14 ms. This translates to an accuracy of ±2 US images.
  • The system successfully recorded a corpus with synchronized US, EM, and audio data, demonstrating that the spatial calibration and temporal synchronization were satisfactory. The visual coherence between the movements and the sound was also confirmed.

Limitations and Future Work

  • US Synchronization Accuracy: The current accuracy of ±2 US images for US synchronization, while good, could be improved for a more seamless visual impression, especially for fast speech sequences.
  • Future Work: The authors plan to improve US synchronization by comparing optical flow computed on ultrasound images with the movement of electromagnetic sensors. They also suggest that the synchronization framework can be extended to other modalities, such as stereovision cameras for face tracking, by using synchronized visual and audio events.

Summary updated

BibTeX
@inproceedings{aron2007acquisition,
  author = {Michael Aron and Nicolas Ferveur and Erwan Kerrien and Marie-Odile Berger and Yves Laprie},
  title = {{Acquisition and synchronization of multimodal articulatory data}},
  booktitle = {Interspeech},
  year = {2007},
}
Figure