Computational elucidation of nonverbal behavior and body language in music therapy

Billie Sandak, Avi Gilboa, David HarelPNAS 2024 · DOI

TL;DR

Researchers have developed a multi-modal "Computational Paradigm" (CP) that uses AI pose estimation and audio-MIDI synchronization to quantify nonverbal behavior in music therapy. By transforming video of therapy sessions into objective metrics—like head tilt, torso openness, and "self-touch" frequency—the system reveals clinical insights and behavioral patterns that are often invisible to the human eye.

What Problem Does This Paper Address?

Music therapy is a powerful clinical tool, yet the "how" and "why" of its efficacy remain difficult to pin down. Traditionally, therapists rely on subjective observations and qualitative notes to track a client’s progress. However, the therapeutic environment is a complex "triangular" interaction between the therapist, the client, and the music itself.

Crucial nonverbal cues—micro-gestures, subtle shifts in posture, or the specific way a client interacts with an instrument—are often too fleeting or subtle for a human observer to record accurately over a 50-minute session. This paper addresses the need for a rigorous, quantitative framework to capture these "elusive" behaviors, providing an empirical baseline to complement clinical intuition.

Methods

The authors extended an existing Computational Paradigm (CP) to include a sophisticated "data mining" pipeline:

  • AI Pose Estimation: The system uses HRNet (High-Resolution Net), a deep neural network (DNN) pre-trained for human pose estimation. It generates a 17-joint skeleton for both the client and therapist from standard video frames (50 fps).
  • Multi-Modal Synchronization: To ensure accuracy, the researchers synchronized video data with MIDI recordings from the piano and audio analysis (using Praat) to distinguish between playing, speaking, and silence.
  • Feature Extraction: Using MATLAB and custom heuristics, the system calculates specific "psychometrics," such as:
  • Torso Pose: Measuring the distance between shoulders and elbows to determine "open" vs. "closed" postures.
  • Head Pose: Tracking the nose tip relative to the horizon to detect "up" (confident/free) vs. "down" (restricted/focused) positions.
  • Self-Touch: Computing the spatial relationship between wrists and the rest of the body to identify self-soothing or stress-related gestures.
  • Keyboard Use: Image processing detects which hands are occluding the keys, identifying if the client is playing with one hand or both.

Datasets

  • Training: The HRNet model was trained on the COCO (Common Objects in Context) dataset, a standard benchmark for object detection and segmentation.
  • Evaluation/Proof-of-Concept: The study analyzed data from four female subjects (ages 22–35) across six sessions each (~50 minutes per session). The paper specifically highlights comparative results for "Subject A" and "Subject B."

Key Insights and Ideas

  • The "Objective vs. Subjective" Gap: In one instance, a therapist noted that Subject A seemed "afraid" and hesitant. However, the CP data showed high levels of "torso openness" (16%) and "head up" poses, suggesting a level of physical engagement that contradicted the therapist's subjective feeling. This highlights how data can challenge or refine clinical hypotheses.
  • Emergent Properties: By looking at "macro-analysis" (across multiple sessions), the system identified "behavioral markers." For example, Subject B showed a significant increase in "down" head poses (from 49% to 88%) and self-touch over time, which could alert a therapist to a potential regression or hidden unease that wasn't immediately obvious in the moment.
  • Micro-Analysis Capabilities: The system can pinpoint the exact second a meaningful event occurs—such as a client standing up and walking away from the piano—and correlate it precisely with the musical passage being played at that moment.

Key Results

  • Quantifiable Metrics: The researchers successfully extracted seven distinct behavioral attributes (Playing segments, Silence, Body pose, Self-touch, Playing hand, Head pose, and Torso pose).
  • Subject Comparison: The data revealed clear idiosyncratic patterns. Subject B was objectively less talkative and more prone to self-touch and "one-handed" playing compared to Subject A, providing a "behavioral fingerprint" for each client.
  • Visualization: The CP produces concise graphical notations (dashboards) that allow therapists to "see" an entire hour-long session at a glance, making it easier to share data across scientific and clinical fields.

Limitations and Future Work

  • Computational Intensity: The authors acknowledge that AI pose estimation is currently time- and space-consuming, which may limit real-time application in clinical settings.
  • Privacy and Ethics: The method requires video recording, which necessitates high levels of client trust and strict data privacy protocols.
  • Scope: The current proof-of-concept is limited to piano-based therapy. Future work aims to expand the CP to include facial expressions, bodily synchronization between therapist and client, and more complex settings involving multiple instruments.

Summary updated

Abstract

Music therapy has shown efficacy in serious and chronic conditions, mental disorders, and disabilities. However, there is still much to explore regarding the mechanisms through which music interventions exert their effects. A typical session involves interactions between the therapist, the client, and the musical work itself, and to help address the challenges of capturing and comprehending its dynamics, we extend our general computational paradigm (CP) for analyzing the expressive and social behavioral processes in arts therapies. The extension includes bodily and nonverbal aspects of the behavior, offering additional insights into the client's emotional states and engagement. We have used this version of the CP, which employs AI pose estimation technology, image processing, and audio analysis, to capture therapy-related psychometrics and their intra- and inter-session analysis. The CP is applied in a real-world proof-of-concept study, and the results enable us to pinpoint meaningful events and emergent properties not captured by the human eye, complementing the therapist's interpretations. The resulting data may also be useful in other scientific and clinical areas.
BibTeX
@inproceedings{sandak2024computational,
  author = {Billie Sandak and Avi Gilboa and David Harel},
  title = {{Computational elucidation of nonverbal behavior and body language in music therapy}},
  booktitle = {PNAS},
  year = {2024},
  doi = {10.1093/pnasnexus/pgae475},
}
Figure