Affect-Aware Machine Learning Models for Deception Detection

Leena MathurAAAI 2021 · OpenReview

Abstract

Automated deception detection systems can enhance societal well-being by helping humans detect deceivers and support people in high-stakes situations across health, social work, and legal domains. Existing computational approaches for detecting deception have not leveraged dimensional representations of affect, specifically valence and arousal, expressed during communication. My research presents a novel analysis of the potential for including affect in machine learning models for detecting deception. My work informs and motivates the development of affect-aware machine learning approaches for modeling deception and other social behaviors during human interactions in-the-wild. This research, independently defined and conducted by me, is from work-in-progress towards my undergraduate thesis in the Department of Computer Science at the University of Southern California.

Introduction

Detecting deception is a notoriously difficult task for humans, who typically perform at near-chance levels. While machine learning has advanced this field, most models focus on verbal content or physical movements without considering the underlying emotional state, known as affect. Psychological theory suggests that high-stakes lying creates "leakage"—involuntary nonverbal cues triggered by the stress of deceiving. Specifically, deceivers often exhibit lower valence (feeling more negative) and higher arousal (feeling more stressed) than truthful speakers. This research addresses the gap in computational models by testing whether these dimensional representations of affect can serve as a reliable "fingerprint" for deception.

Methodology

The study developed a pipeline to extract and fuse multiple modalities of communication using a benchmark dataset of 108 videos from real-world courtroom trials. This dataset includes 53 truthful and 55 deceptive testimonies from 47 diverse individuals, with ground truth verified by legal outcomes.

Feature Extraction

  1. Affect Extraction: A deep neural network trained on the Aff-Wild dataset extracted continuous values for facial valence and arousal.
  2. Visual Features: 31 features including facial action units, head pose, and eye gaze were extracted via OpenFace.
  3. Vocal Features: 65 features such as pitch, energy, and loudness were extracted via OpenSMILE.
  4. Verbal Features: 93 psycholinguistic features were extracted from transcripts using LIWC.

Modeling and Fusion

Support Vector Machines (SVMs) were used for unimodal classification. To combine these data streams, various fusion strategies were tested. AdaBoost, a generative ensemble method, emerged as the most effective approach for combining facial affect, visual, and vocal modalities.

Results

The affect-aware multimodal model using AdaBoost achieved a state-of-the-art AUC of 0.91 and an accuracy of 84%. Notably, facial affect features alone achieved an AUC of 0.80, confirming that emotional leakage is computationally detectable. The synergy between facial affect, visual movements, and vocal tone provided a more holistic and accurate view of the speaker's state than any single modality.

ROC curves for various unimodal and multimodal deception detection models.

As shown in the ROC curves, the multimodal AdaBoost model significantly outperforms unimodal baselines. The inclusion of affect allows the model to surpass previous state-of-the-art results (0.88 AUC) on this dataset, which relied on visual, vocal, and verbal cues but ignored affect.

Discussion and Future Work

This research demonstrates that "affect-aware" machine learning significantly improves automated deception detection. However, limitations remain, including the relatively small sample size of the courtroom dataset. Future work will explore the generalization of these models to other high-stakes domains like healthcare and social work, and investigate the integration of vocal affect to further enhance discriminative power. The findings emphasize that temporal patterns of valence and arousal are critical for modeling complex social behaviors in-the-wild.

Summary updated

Abstract

Automated deception detection systems can enhance societal well-being by helping humans detect deceivers and support people in high-stakes situations across health, social work, and legal domains. Existing computational approaches for detecting deception have not leveraged dimensional representations of affect, specifically valence and arousal, expressed during communication. My research presents a novel analysis of the potential for including affect in machine learning models for detecting deception. My work informs and motivates the development of affect-aware machine learning approaches for modeling deception and other social behaviors during human interactions in-the-wild. This research, independently defined and conducted by me, is from work-in-progress towards my undergraduate thesis in the Department of Computer Science at the University of Southern California.
BibTeX
@inproceedings{mathur2021affect,
  author = {Leena Mathur},
  title = {{Affect-Aware Machine Learning Models for Deception Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=Vcylf5zfwf},
}
Figure