Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection

Leena Mathur, Maja J. Mataric

TL;DR

Researchers have developed the first unsupervised transfer learning approach to detect real-world, high-stakes deception in videos without requiring labeled "high-stakes" training data. By using Subspace Alignment (SA) to bridge the gap between easy-to-get lab data and rare real-world data, their model outperformed human detection abilities and performed comparably to existing supervised systems.

What Problem Does This Paper Address?

Detecting deception is a high-stakes challenge in legal, medical, and social work domains. However, building machine learning models for this task faces a massive "data scarcity" wall.

There are two types of deception data:

  1. Low-Stakes: Collected in controlled lab settings (e.g., mock crimes). These are easy to label but lack the physiological and behavioral intensity of real-world lies.
  2. High-Stakes: Real-world scenarios (e.g., courtroom trials) where the liar faces actual consequences. This data is incredibly rare, difficult to verify (ground truth), and unethical to simulate in a lab.

Most existing models are supervised, meaning they need labeled high-stakes data to learn. This paper asks: Can we take what we know about low-stakes lies in the lab and "align" it to detect high-stakes lies in the real world without needing any labels for the high-stakes videos?

Methods

The authors propose an Unsupervised Domain Adaptation (UDA) approach using Subspace Alignment (SA).

The pipeline works as follows:

  • Feature Extraction: They used OpenSMILE for audio (prosody, pitch, etc.) and OpenFace for visual cues (facial action units, gaze, head pose). These were processed into 1,068 time-series features using the TsFresh toolkit.
  • Subspace Alignment: This is the "secret sauce." Since lab data (Source) and courtroom data (Target) have different distributions, you can't just train on one and test on the other. The researchers used Principal Component Analysis (PCA) to create $D$-dimensional subspaces for both. They then calculated a linear transformation matrix ($\phi$) to align the lab subspace with the courtroom subspace by minimizing the Frobenius norm between them.
  • Classification: Once aligned, they trained a K-Nearest Neighbors (KNN) classifier on the transformed lab data and tested it on the unlabeled courtroom data.

Datasets

The study utilized two primary datasets:

  • Source (Low-Stakes): The UR Lying Dataset, containing 107 videos of participants lying or telling the truth about images in a lab setting to win small cash prizes.
  • Target (High-Stakes): A benchmark dataset of 121 real-world courtroom trial videos. The authors used a subset of 108 videos (53 truthful, 55 deceptive) where the ground truth was verified by police testimony and trial outcomes.

Key Insights and Ideas

  • The Power of Eye Gaze: Interestingly, the visual-only models—specifically those focusing on eye gaze—showed the highest "transfer potential." This suggests that while vocal patterns might change significantly under pressure, certain ocular behaviors remain consistent across different levels of "stakes."
  • Universal Deception Cues: Through statistical testing (Welch’s t-test), the authors identified 10 behavioral signals that were remarkably similar across both lab and courtroom settings, including jitter, pitch, and yaw of head pose. These are the "transferable" cues that make unsupervised learning possible.
  • Beating the Human Baseline: Humans are notoriously bad at detecting lies, usually performing at near-chance levels (~50-54%). This unsupervised model significantly cleared that bar, proving that automated subspace alignment can "see" patterns that the human eye misses.

Key Results

The results demonstrate that you don't always need massive labeled datasets if you have smart alignment techniques:

  • Top Performance: The best unsupervised model (Visual SA) achieved 74% accuracy and a 75% AUC (Area Under the Curve).
  • SA vs. No SA: Models using Subspace Alignment significantly outperformed "naive" models that tried to transfer knowledge without alignment.
  • Competitive with Supervised Models: Perhaps most impressively, this unsupervised approach performed comparably to several state-of-the-art supervised models (which typically range from 75% to 79% accuracy on the same dataset) despite never seeing a single "high-stakes" label during training.

Limitations and Future Work

While a major step forward, the study has a few constraints:

  • Dataset Size: The courtroom dataset is still relatively small (108 videos), which is a perennial problem in high-stakes research.
  • Modality Fusion: While the visual models were strong, the multimodal "early fusion" (combining audio and visual) didn't significantly outperform the visual-only model, suggesting room for improvement in how different types of data are integrated.
  • Generalization: The authors suggest that future work should explore whether these subspace alignment techniques can be applied to other social behaviors where labeled data is scarce, such as detecting depression or anxiety in clinical settings.

Summary updated

Abstract

Automated systems that detect deception in high-stakes situations can enhance societal well-being across medical, social work, and legal domains. Existing models for detecting high-stakes deception in videos have been supervised, but labeled datasets to train models can rarely be collected for most real-world applications. To address this problem, we propose the first multimodal unsupervised transfer learning approach that detects real-world, high-stakes deception in videos with-out using high-stakes labels. Our subspace-alignment (SA) approach adapts audio-visual representations of deception in lab-controlled low-stakes scenarios to detect deception in real-world, high-stakes situations. Our best unsupervised SA models outperform models without SA, outperform human ability, and perform comparably to a number of existing supervised models. Our research demonstrates the potential for introducing subspace-based transfer learning to model high-stakes deception and other social behaviors in real-world contexts with a scarcity of labeled behavioral data.
BibTeX
@inproceedings{mathur2021unsupervised,
  author = {Leena Mathur and Maja J. Mataric},
  title = {{Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=GrlPak8QJw},
}

Loading related content...

Figure