A French-Spanish Multimodal Speech Communication Corpus Incorporating Acoustic Data, Facial, Hands and Arms Gestures Information

Lucas D. Terissi, Gonzalo Sad, Mauricio Cerda, Slim Ouni, Rodrigo Galvez, Juan C. Gómez, Bernard Girau, Nancy Hitschfeld-KahlerInterspeech 2018

TL;DR

This paper introduces a novel multimodal speech corpus that captures not only audio and facial expressions but also hand and arm gestures, recorded in both French and Spanish. This comprehensive dataset aims to advance research in human-computer interaction and multimodal communication by enabling the study of correlations between speech, facial cues, and body language.

What Problem Does This Paper Address?

Human communication is inherently multimodal, with speech, facial expressions, and gestures working in concert to convey meaning. While many existing corpora focus on audio and facial data, there's a significant gap in datasets that capture the full spectrum of gestural information alongside speech. This limits our ability to build more natural and intuitive human-computer interfaces, understand the nuances of human interaction, and develop robust multimodal recognition systems. The authors aim to bridge this gap by creating a rich, bilingual corpus that includes detailed recordings of facial movements, hand gestures, and arm movements synchronized with speech.

Methods

The core contribution of this paper is the creation and description of a new multimodal corpus. The data acquisition involved a sophisticated setup using:

  • Acoustic Data: Captured using a professional microphone at a 44.1kHz sample rate.
  • Facial and Hand/Arm Gestures:
    • Vicon Motion System (MX-3+): An infrared marker-based system used to track reflective markers placed on the subject's face, head, hands, and arms. This system operates at a 100Hz sampling rate.
  • Intel RealSense™ F200: A marker-less, structured-light 3D sensor combined with a 2D camera. This system captures facial and hand movements at 60Hz using deformable 3D generic models.

The paper details specific acquisition protocols for two main scenarios:

  • Non-spontaneous Speech: Participants read scripted text and performed facial expressions.
  • Spontaneous Speech: Participants engaged in natural conversations, described objects/situations, or recalled memories. This modality also captured hand and arm movements.

Data processing involved:

  • Temporal Synchronization: A "beeper" device emitting audio tones and LEDs was used to synchronize audio and visual streams from multiple devices.
  • Spatial Alignment: Techniques were developed to align the 3D coordinate systems of the Vicon and RealSense systems, crucial for integrating data from different sensors. This involved using common marker points or approximating transformations based on anatomical landmarks.

Datasets

The paper introduces a newly created French-Spanish Multimodal Speech Communication Corpus.

  • Languages: French and Spanish.
  • Speakers: 4 Spanish speakers (2 male, 2 female) and 4 French speakers (1 female, 3 male).
  • Modalities: Acoustic speech, facial expressions, hand gestures, and arm gestures.
  • Speech Types: Scripted (non-spontaneous) and natural (spontaneous) speech.
  • Content:
    • Non-spontaneous: Digits, actions, facial expressions, consonant-vowel syllables, and phonetically balanced sentences.
  • Spontaneous: Natural conversations, descriptions of items/situations, and recall of memories.
  • Duration: Approximately 6 hours of non-spontaneous data and 4 hours of spontaneous data.
  • Availability: Freely available for research purposes at http://cimt.uchile.cl/mcc.

This corpus differs from existing ones by its inclusion of detailed hand and arm gesture data alongside speech and facial information, and its bilingual nature.

Key Insights and Ideas

  • Gestures are Integral to Spontaneous Speech: Preliminary analysis using Correlation Map Analysis (CMA) revealed a strong correlation between fundamental speech frequency (F0), head motion, and hand motion during spontaneous speech. This suggests that these modalities are tightly coupled when people communicate naturally.
  • Head-Speech Correlation is Consistent: The correlation between F0 and head movements was found to be similar across all participants, regardless of their expressiveness. This implies a fundamental, perhaps subconscious, link between vocal prosody and head orientation.
  • Hand-Speech Correlation Varies with Expressivity: The correlation between F0 and hand movements, however, was observed to vary with the speaker's expressiveness, indicating that the degree of hand gesticulation influences its synchronization with vocal cues.
  • Spontaneous Speech is Key for Gestural-Speech Links: The correlation between F0 and head motion was significantly lower in the non-spontaneous (scripted) material compared to spontaneous speech, highlighting the importance of natural interaction for observing these multimodal links.

Key Results

  • The paper successfully created and described a comprehensive multimodal corpus in French and Spanish, encompassing acoustic, facial, hand, and arm gesture data.
  • Preliminary CMA results demonstrated significant correlations between F0 and head movements, and F0 and hand movements during spontaneous speech.
  • The study found that head motion is consistently correlated with speech prosody across speakers, while hand motion's correlation is more dependent on individual expressiveness.
  • The corpus is made publicly available for research.

Limitations and Future Work

  • Limitations:
    • The spatial alignment for spontaneous speech in the sitting condition is described as an approximation, not a perfect transformation.
  • The number of speakers is relatively small (8 total), which might limit the generalizability of findings.
  • Future Work:
    • The authors plan to annotate the corpus further, including detailed annotations for speech, facial movements, and hands/arms gestures. This will enable more in-depth analysis and development of multimodal systems.

Summary updated

BibTeX
@inproceedings{terissi2018french,
  author = {Lucas D. Terissi and Gonzalo Sad and Mauricio Cerda and Slim Ouni and Rodrigo Galvez and Juan C. Gómez and Bernard Girau and Nancy Hitschfeld-Kahler},
  title = {{A French-Spanish Multimodal Speech Communication Corpus Incorporating Acoustic Data, Facial, Hands and Arms Gestures Information}},
  booktitle = {Interspeech},
  year = {2018},
}
Figure