Hearing through Vibrations: Perception of Musical Emotions by Profoundly Deaf People

Anastasia Schmitz, Catherine Holloway, Youngjun ChoarXiv 2020 · arXiv

Abstract

This study investigates how profoundly deaf individuals perceive musical emotions through haptic feedback using a wearable vibrotactile backpack. While haptic technology effectively communicates musical "arousal" (energy and intensity), it faces challenges in conveying "valence" (specific emotional mood). The findings provide critical design guidelines for developing more inclusive audio-haptic technologies for the deaf community.

Introduction

For the profoundly deaf community, music is often physically inaccessible. Traditional assistive technologies like hearing aids and cochlear implants frequently offer poor spectral resolution, making it difficult to discern pitch and timbre—the fundamental building blocks of musical emotion. Recent advances in "tactile-audio" feedback allow users to feel sound waves as vibrations on the skin. However, a significant gap remains in understanding the extent to which profoundly deaf individuals can recognize intended musical emotions through vibrations alone, as previous research has often relied on hearing participants with simulated deafness.

Methodology

The researchers conducted a mixed-methods study involving 16 profoundly deaf participants. The primary hardware used was the SubPac M2x, a commercial vibrotactile backpack that transduces sound frequencies between 1–200Hz into physical vibrations felt on the back.

Experimental setup featuring the SubPac M2x vibrotactile backpack used to transmit musical vibrations.

Participants were presented with 20 piano excerpts, specifically composed by Vieillard et al. (2008) to evoke four distinct emotions: happy, angry, sad, and peaceful. These excerpts control for musical features such as mode, pitch, and tempo. During the quantitative phase, participants rated each excerpt on 9-point Likert scales for arousal (excited vs. bored), emotional valence (positive vs. negative), and confidence. They also completed a multiple-choice emotion identification task.

Key Results and Insights

Arousal vs. Valence

The study found that arousal is significantly easier to perceive through haptics than valence. Participants successfully distinguished "high energy" music (happy and angry) from "low energy" music (sad and peaceful). However, they struggled to differentiate between emotions with similar energy levels, such as sadness and peacefulness.

Scatter plot of valence and arousal ratings, showing clear clustering by arousal but significant overlap in valence.

The "Positive Bias" of Haptics

An interesting finding was the "positive bias" observed in deaf participants. Unlike hearing participants in previous studies who often found haptic vibrations annoying, deaf participants generally perceived the vibrations as positive and engaging. For this community, the sensation of "feeling" the music is inherently rewarding, leading to higher valence ratings across almost all stimuli.

Identification Accuracy

Statistical analysis showed that happy and angry excerpts were correctly identified significantly more often than sad or peaceful ones. Specifically, arousal ratings for Happy (M=6.40) and Angry (M=7.20) were markedly higher than those for Sad (M=3.79) and Peaceful (M=4.09). Qualitative feedback supported these findings, with 11 out of 14 respondents reporting that the backpack created an "engaging and enhanced" experience, even if bass-only vibrations lacked the nuance to distinguish complex musical narratives.

Discussion and Future Work

The research highlights that vibrations serve best as a complement rather than a replacement for auditory input. Participants expressed a strong desire for multimodal experiences—combining haptics with visual cues (like colors or lyrics) and residual auditory input via hearing aids to fill in the "missing pieces."

Future work will aim to address current limitations, including the small sample size and the controlled lab environment. Researchers plan to move into "ecological" settings, such as live concerts, and incorporate physiological metrics like heart rate variability (HRV) to gain more objective measures of emotional response. Additionally, investigating how haptics can convey text-based emotional content from lyrics remains a priority for inclusive design.

Summary updated

Abstract

Advances in tactile-audio feedback technology have created new possibilities for deaf people to feel music. However, little is known about deaf individuals' perception of musical emotions through vibrotactile feedback. In this paper, we present the findings from a mixed-methods study with 16 profoundly deaf participants. The study protocol was designed to explore how users of a backpack-style vibrotactile display perceive intended emotions in twenty music excerpts. Quantitative analysis demonstrated that participants correctly identified happy and angry excerpts and rated them as more arousing than sad and peaceful excerpts. More positive emotions were experienced during happy compared to angry excerpts while peaceful and sad excerpts were hard to be differentiated. Based on qualitative data, we highlight the benefits and limitations of using vibrations to convey musical emotions to profoundly deaf users. Finally, we provide guidelines for designing accessible music experiences for the deaf community.
BibTeX
@misc{schmitz2020hearing,
  author = {Anastasia Schmitz and Catherine Holloway and Youngjun Cho},
  title = {{Hearing through Vibrations: Perception of Musical Emotions by Profoundly Deaf People}},
  year = {2020},
  eprint = {2012.13265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.13265},
}
Figure