SensiTV: Smart EmotioNal System for Impaired People's TV

Diana Affi, Joël Dumoulin, Marco Bertini, Elena Mugellini, Omar Abou Khaled, Alberto Del Bimbo

Introduction

Modern television accessibility features, such as closed captions and audio descriptions, primarily focus on conveying cognitive content like dialogue and plot. However, they often fail to transmit the affective layer—the emotional nuances of tension, joy, or sorrow—that are central to the cinematic experience. SensiTV is an assistive technology designed to bridge this "emotional gap" for viewers with hearing or visual impairments by translating the emotional "vibe" of a film into a multi-sensory experience using ambient lighting, haptics, and adaptive visual cues.

Methodology

The SensiTV architecture consists of two primary modules: an Emotion Recognition engine and a Multi-Sensory Communication interface.

Emotion Recognition

The system classifies movie scenes into seven basic emotions: Anger, Disgust, Fear, Happiness, Sadness, Surprise, and Neutral. Feature extraction involves both audio and visual data. Audio features, such as prosody, are processed using the OpenSMILE toolkit. Visual features include brightness, shot cut density, edges, and color histograms. The researchers also explored "biconcept" features (adjective-noun pairs like "bloody scene"). Classification is performed using a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel, following dimensionality reduction via Linear Discriminant Analysis (LDA).

Multi-Sensory Communication

Once an emotion is detected, the system triggers several non-intrusive modalities to alert the viewer. Ambient lighting via Philips Hue LEDs adjusts room color and intensity (e.g., red for anger). Additionally, a mobile application provides haptic feedback through distinct vibration patterns for different emotions, functioning similarly to Morse code.

Vibration patterns mapped to specific emotions for haptic feedback.

The app also displays emoticons and plays "background mood music" for the visually impaired, while the TV displays "emotional subtitles" with dynamic fonts and colors.

Key Results

The system was trained and tested using the AFEW (Acted Facial Expression in the Wild) dataset. The SVM classifier achieved its best performance when combining audio and visual features with LDA. For instance, "Anger" was identified with 53% accuracy in a complex, real-time environment.

Confusion matrix showing the performance of the SVM classifier across seven emotion categories.

User validation through a survey of 213 participants confirmed that emoticons are highly effective for emotion communication, while haptic vibrations serve as an excellent notification for rising emotional intensity. However, the study also noted ambiguities in color mapping, such as users confusing purple and yellow between "Happiness" and "Surprise."

Discussion and Future Work

Preliminary results indicate that video streams are more critical than audio for cinematic emotion recognition. Future work will focus on testing the system directly with visually and hearing-impaired audiences to assess intrusiveness and effectiveness. Additionally, the researchers plan to implement deep learning models to improve accuracy and explore "smart objects," such as vibrating seats, to further enhance immersion.

Summary updated

BibTeX
@inproceedings{affi2015sensitv,
  author = {Diana Affi and Joël Dumoulin and Marco Bertini and Elena Mugellini and Omar Abou Khaled and Alberto Del Bimbo},
  title = {{SensiTV: Smart EmotioNal System for Impaired People's TV}},
  booktitle = {TVX},
  year = {2015},
  doi = {10.1145/2745197.2755512},
}
Figure