Synchronising Audio and Ultrasound by Learning Cross-Modal Embeddings

Aciel Eshky, Manuel Sam Ribeiro, Korin Richmond, Steve RenalsInterspeech 2019

TL;DR

This paper tackles the challenge of synchronizing speech audio with ultrasound tongue imaging (UTI) data, a crucial but error-prone step in speech therapy. They introduce UltraSync, a self-supervised neural network that learns to align these two modalities by creating cross-modal embeddings, achieving significant improvements over manual synchronization.

What Problem Does This Paper Address?

Accurate synchronization between speech audio and visual representations of the vocal tract, specifically ultrasound tongue imaging (UTI), is vital for speech therapy. While hardware synchronization is attempted during recording, it frequently fails, leading to unusable data. Manual correction is tedious and time-consuming. Existing methods for synchronizing audio with lip videos are not directly applicable to UTI due to its unique challenges: poor image quality, probe placement variations, inter-speaker differences, and uncorrelated audio/video segments (e.g., therapist speech vs. patient's tongue). This paper aims to bridge this gap by developing an automated post-hoc synchronization method for UTI data.

Methods

The authors adapt a successful approach for audio-lip synchronization, SyncNet, to the UTI domain. Their model, UltraSync, employs a two-stream neural network. One stream processes ultrasound frames, and the other processes Mel-Frequency Cepstral Coefficients (MFCCs) extracted from the audio. Both streams learn to map their respective inputs into low-dimensional embeddings. The core of the training is a self-supervised contrastive loss function. This function encourages the embeddings from corresponding audio and ultrasound segments (positive pairs) to be close in Euclidean distance, while embeddings from non-corresponding segments (negative pairs) are pushed apart. This allows the model to learn the correlation between the modalities without requiring manually labeled synchronized pairs. For predicting the synchronization offset of an entire utterance, the model calculates average distances across candidate offsets and selects the one with the minimum average distance.

Datasets

The primary dataset used is UltraSuite, a repository of ultrasound and acoustic data collected from child speech therapy sessions. This dataset comprises 13,815 spoken utterances from 86 speakers, totaling 35.9 hours. The utterances are categorized into types: Words (A), Non-words (B), Sentence (C), Articulatory (D), Non-speech (E), and Conversations (F). The authors exclude "Non-speech" utterances from their analysis. They also mention using VoxCeleb and VoxCeleb2 for training SyncNet in prior work, highlighting their much larger scale compared to UltraSuite.

Key Insights and Ideas

  • Self-supervision is powerful for UTI: The success of UltraSync demonstrates that learning cross-modal embeddings through self-supervision is a viable and effective strategy for synchronizing challenging modalities like ultrasound and audio, even without explicit ground-truth synchronization labels for training.
  • UTI presents unique challenges: The paper clearly articulates why UTI is harder to synchronize than lip videos, emphasizing issues like noise, partial views of the vocal tract, and probe variations, which impact model performance.
  • Natural speech variation aids synchronization: The model performs significantly better on utterances containing natural speech variations (words, sentences, conversations) compared to isolated phonetic articulations. This suggests that richer temporal and contextual information in natural speech provides stronger cues for synchronization.
  • Stop consonants are good cues: The model successfully synchronizes a high percentage of stop consonants, which are known to be salient audiovisual cues for human perception, further validating the model's ability to capture relevant articulatory-acoustic correlations.

Key Results

  • UltraSync achieves 82.9% accuracy in correctly synchronizing test utterances, with a mean discrepancy of 32 ms and a standard deviation of 223 ms. This is a substantial improvement over random prediction.
  • Performance is notably better on utterances with natural speech variation: 91.2% accuracy for Words, Sentences, and Conversations combined.
  • Synchronization accuracy drops to 55.9% for Articulatory utterances (isolated phones), highlighting a key limitation.
  • The model performs better when tested on new speakers compared to new sessions from known speakers, which was contrary to initial expectations.
  • The model achieves 65.3% binary classification accuracy on test samples when predicting whether a pair is synchronized or not, indicating a good understanding of the synchronization boundary.

Limitations and Future Work

  • Articulatory utterances are challenging: The model struggles with utterances containing isolated phones (Articulatory type D), likely due to subtle tongue movements and lack of rich contextual information.
  • Performance on UXTD dataset: The model shows lower performance on the UXTD dataset, which is attributed to its higher proportion of Articulatory utterances.
  • Future Work: The authors suggest integrating the model into speech therapy software for real-time feedback and exploring the learned embeddings for other tasks like active speaker detection.

Summary updated

BibTeX
@inproceedings{eshky2019synchronising,
  author = {Aciel Eshky and Manuel Sam Ribeiro and Korin Richmond and Steve Renals},
  title = {{Synchronising Audio and Ultrasound by Learning Cross-Modal Embeddings}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure