Cross-Modal Coordination of Face-Directed Gaze and Emotional Speech Production in School-Aged Children and Adolescents with ASD

Tanner Sorensen, Emily Zane, Tiantian Feng, Shrikanth Narayanan, Ruth B. GrossmanScientific 2019 · DOI

Abstract

Autism spectrum disorder involves persistent difficulties in social communication. Although these difficulties affect both verbal and nonverbal communication, there are no quantitative behavioral studies to date investigating the cross-modal coordination of verbal and nonverbal communication in autism. The objective of the present study was to characterize the dynamic relation between speech production and facial expression in children with autism and to establish how face-directed gaze modulates this cross-modal coordination. In a dynamic mimicry task, experiment participants watched and repeated neutral and emotional spoken sentences with accompanying facial expressions. Analysis of audio and motion capture data quantified cross-modal coordination between simultaneous speech production and facial expression. Whereas neurotypical children produced emotional sentences with strong cross-modal coordination and produced neutral sentences with weak cross-modal coordination, autistic children produced similar levels of cross-modal coordination for both neutral and emotional sentences. An eyetracking analysis revealed that cross-modal coordination of speech production and facial expression was greater when the neurotypical child spent more time looking at the face, but weaker when the autistic child spent more time looking at the face. In sum, social communication difficulties in autism spectrum disorder may involve deficits in cross-modal coordination. This finding may inform how autistic individuals are perceived in their daily conversations.

Introduction: The Challenge of Multimodal Communication

Social communication is inherently multimodal; we do not simply speak in a vacuum. Effective communication relies on the seamless integration of verbal language with nonverbal cues, such as eye movements, eyebrow shifts, and mouth shapes, to provide emotional context and emphasis. While research has well-documented atypical prosody (the rhythm and tone of speech) and facial expressions in individuals with Autism Spectrum Disorder (ASD), the coordination between these two modes remains under-explored.

This "cross-modal coordination" is the glue that makes social interactions feel natural. When this synchrony is disrupted, a speaker may be perceived as "awkward" or "out of sync." This study aims to quantify this synchrony and investigate how visual social processing—specifically, looking at a speaker's face—influences the production of these integrated signals in children and adolescents with ASD.

Methodology

The researchers utilized a dynamic mimicry task involving 35 participants: 16 children with ASD and 19 neurotypical (NT) children, matched for age, IQ, and language ability. Participants viewed videos of actors delivering sentences that transitioned from neutral to emotional (e.g., pride or disgust) and were tasked with mimicking both the speech and the facial expression.

The study employed a rigorous technical framework:

  • Motion Capture: 28 reflective markers tracked facial movements at 100 frames per second.
  • Acoustic Analysis: Key speech features, including pitch and loudness, were extracted from audio recordings.
  • Eyetracking: Measured "net dwell time" on the actor's upper and lower face during the stimulus phase.
  • Granger Causality: This statistical method served as the primary metric for coordination, determining the extent to which facial movement signals could predict speech audio signals.

Key Findings: The Asynchrony of ASD Speech

The study revealed a significant divergence in how NT and ASD children manage multimodal signals. Neurotypical children demonstrated a sharp increase in cross-modal coordination when moving from neutral to emotional speech ($p < 1 \times 10^{-16}$). In contrast, while children with ASD also showed an increase, their level of coordination was significantly lower than that of their peers ($p = 3.8 \times 10^{-7}$), remaining relatively flat during emotional moments.

Figure 1: Comparison of cross-modal coordination levels between ASD and neurotypical groups across neutral and emotional conditions.

Crucially, the researchers found that the issue was not a lack of intensity in facial movement or speech volume. Instead, the deficit lies specifically in the temporal alignment—the millisecond-level timing—between the two modalities.

The Gaze Paradox

One of the most striking findings involves the role of visual attention. In neurotypical children, increased gaze toward the actor's face correlated with stronger cross-modal coordination in their own subsequent production. However, for children with ASD, this relationship was reversed: more time spent looking at the stimulus face was associated with weaker coordination (slopes of -0.16 for the upper face and -0.14 for the lower face).

Figure 3: Net dwell time on the upper and lower face for ASD and neurotypical participants during neutral and emotional stimuli.

This suggests that for some individuals with ASD, high social motivation (indicated by increased gaze) might lead to a cognitive "overload." The effort required to process complex social visual cues may interfere with the ability to integrate and reproduce those cues synchronously.

Limitations and Future Directions

While these findings provide a quantitative look at social asynchrony, several limitations exist. The study focused on a high-functioning cohort, and results may differ for individuals with intellectual disabilities or those who are non-verbal. Additionally, the mimicry task is a controlled exercise; real-world, spontaneous conversation is far more complex. Future research should explore whether these coordination deficits correlate directly with human perceptions of social "awkwardness" in naturalistic settings.

Ultimately, this research highlights that social challenges in ASD are not just about what is expressed, but the intricate timing that binds our words to our faces.

Summary updated

Abstract

Autism spectrum disorder involves persistent difficulties in social communication. Although these difficulties affect both verbal and nonverbal communication, there are no quantitative behavioral studies to date investigating the cross-modal coordination of verbal and nonverbal communication in autism. The objective of the present study was to characterize the dynamic relation between speech production and facial expression in children with autism and to establish how face-directed gaze modulates this cross-modal coordination. In a dynamic mimicry task, experiment participants watched and repeated neutral and emotional spoken sentences with accompanying facial expressions. Analysis of audio and motion capture data quantified cross-modal coordination between simultaneous speech production and facial expression. Whereas neurotypical children produced emotional sentences with strong cross-modal coordination and produced neutral sentences with weak cross-modal coordination, autistic children produced similar levels of cross-modal coordination for both neutral and emotional sentences. An eyetracking analysis revealed that cross-modal coordination of speech production and facial expression was greater when the neurotypical child spent more time looking at the face, but weaker when the autistic child spent more time looking at the face. In sum, social communication difficulties in autism spectrum disorder may involve deficits in cross-modal coordination. This finding may inform how autistic individuals are perceived in their daily conversations.
BibTeX
@inproceedings{sorensen2019cross,
  author = {Tanner Sorensen and Emily Zane and Tiantian Feng and Shrikanth Narayanan and Ruth B. Grossman},
  title = {{Cross-Modal Coordination of Face-Directed Gaze and Emotional Speech Production in School-Aged Children and Adolescents with ASD}},
  booktitle = {Scientific},
  year = {2019},
  doi = {10.1038/s41598-019-54587-z},
}
Figure