Inferring Association Between Alcohol Addiction and Defendant's Emotion Based on Sound at Court

Yun Song, Zhongyu WeiFrontiers in Psychology 2021 · DOI

Abstract

Alcohol addiction is a significant social and health issue that profoundly impacts emotional regulation. In legal settings, a defendant's emotional state can influence judicial outcomes, yet objective methods for analyzing these emotions in real-time are lacking. This paper proposes ResCNN-SER, a deep learning framework designed for Speech Emotion Recognition (SER) in courtroom environments. By applying this model to trial records, we investigate the correlation between alcohol addiction and emotional displays. Our findings indicate that defendants with a history of alcohol addiction exhibit higher frequencies of anger and fear during court proceedings compared to those without addiction.

Introduction

The courtroom is a high-stakes environment where emotions play a critical role in communication and decision-making. Manually analyzing hundreds of hours of trial footage for emotional cues is labor-intensive and subjective. While psychological research has linked alcohol addiction to emotional dysregulation, this relationship has rarely been quantified within the specific context of a legal trial. This study bridges that gap by introducing an automated, objective tool for SER to analyze how alcohol addiction manifests emotionally when a defendant stands before a judge.

The ResCNN-SER Framework

The proposed ResCNN-SER architecture is a hybrid deep learning model designed to capture both the spatial and temporal nuances of speech. The pipeline consists of five distinct stages:

  1. Feature Representation: Raw audio is converted into log-Mel spectrograms, incorporating static, delta, and delta-delta features to capture frequency and temporal changes.
  2. Residual CNN (ResCNN): A Convolutional Neural Network with residual blocks extracts high-level spatial features while preventing the vanishing gradient problem.
  3. Bi-LSTM: A Bidirectional Long Short-Term Memory network processes the sequence, considering both preceding and succeeding context to understand emotional prosody.
  4. Attention Mechanism: This layer assigns importance scores to different time steps, focusing the model on the most emotionally significant vocal cues.
  5. Classification: A fully connected layer with a Softmax function predicts the final emotion category.
Overview of the ResCNN-SER deep learning framework, illustrating the pipeline from raw speech data to emotion classification.

Experimental Setup and Datasets

The model was validated using two primary datasets:

  • Emo-DB (Berlin Emotional Database): A benchmark dataset of 535 utterances used to compare ResCNN-SER against existing state-of-the-art methods.
  • Court Sound Dataset: A custom dataset derived from 54 trial records from Datong People’s Court. This included 27 records of defendants with alcohol addiction and 27 control records, manually annotated by experts to establish ground truth.

Results and Analysis

ResCNN-SER demonstrated superior performance on the Emo-DB benchmark, achieving an Unweighted Average Recall (UAR) of 0.833, significantly outperforming the previous state-of-the-art model, ARCNN (0.773).

Performance comparison between ARCNN and ResCNN-SER (left) and the confusion matrix for the proposed model (right).

When applied to the courtroom dataset, the model revealed a stark contrast in emotional baselines:

  • Anger and Fear: Defendants with alcohol addiction were more likely to display anger (13 vs. 5 in control) and fear (8 vs. 4 in control).
  • Neutrality: Only 6 defendants with addiction remained neutral, compared to 18 in the control group.

These results suggest that alcohol addiction is associated with a shift away from neutral emotional states toward more volatile, negative reactions during legal proceedings.

Conclusion

The ResCNN-SER framework provides a robust method for objective emotion recognition in complex acoustic environments like courtrooms. Our analysis confirms a significant association between alcohol addiction and heightened displays of anger and fear in defendants. Future work will focus on improving speaker diarization to isolate individual voices in multi-speaker environments and expanding the dataset to include a broader range of legal contexts.

Summary updated

Abstract

Alcohol addiction can lead to health and social problems. It can also affect people's emotions. Emotion plays a key role in human communications. It is important to recognize the people's emotions at the court and infer the association between the people's emotions and the alcohol addiction. However, it is challenging to recognize people's emotions efficiently in the courtroom. Furthermore, to the best of our knowledge, no existing work is about the association between alcohol addiction and people's emotions at court. In this paper, we propose a deep learning framework for predicting people's emotions based on sound perception, named ResCNN-SER. The proposed model combines several neural network-based components to extract the features of the speech signals and predict the emotions. The evaluation shows that the proposed model performs better than existing methods. By applying ResCNN-SER for emotion recognition based on people's voices at court, we infer the association between alcohol addiction and the defendant's emotion at court. Based on the sound source data from 54 trial records, we found that the defendants with alcohol addiction tend to get angry or fearful more easily at court comparing with defendants without alcohol addiction.
BibTeX
@inproceedings{song2021inferring,
  author = {Yun Song and Zhongyu Wei},
  title = {{Inferring Association Between Alcohol Addiction and Defendant's Emotion Based on Sound at Court}},
  booktitle = {Frontiers in Psychology},
  year = {2021},
  doi = {10.3389/fpsyg.2021.669780},
}
Figure