Abstract
Automated systems that detect deception in high-stakes situations can enhance societal well-being across medical, social work, and legal domains. Existing models for detecting high-stakes deception in videos have been supervised, but labeled datasets to train models can rarely be collected for most real-world applications. To address this problem, we propose the first multimodal unsupervised transfer learning approach that detects real-world, high-stakes deception in videos without using high-stakes labels. Our subspace-alignment (SA) approach adapts audio-visual representations of deception in lab-controlled low-stakes scenarios to detect deception in real-world, high-stakes situations. Our best unsupervised SA models outperform models without SA, outperform human ability, and perform comparably to a number of existing supervised models. Our research demonstrates the potential for introducing subspace-based transfer learning to model high-stakes deception and other social behaviors in real-world contexts with a scarcity of labeled behavioral data.
The Challenge of High-Stakes Data Scarcity
Detecting deception is a critical challenge in legal, medical, and social work domains. However, building machine learning models for this task faces a significant "data scarcity" wall. Deception data typically falls into two categories:
- Low-Stakes: Collected in controlled lab settings (e.g., mock crimes). These are easy to label but lack the physiological and behavioral intensity of real-world lies.
- High-Stakes: Real-world scenarios (e.g., courtroom trials) where the liar faces actual consequences. This data is rare, difficult to verify (ground truth), and unethical to simulate in a lab.
Most existing models are supervised, requiring labeled high-stakes data to learn. This research asks: Can we take what we know about low-stakes lies in the lab and "align" it to detect high-stakes lies in the real world without needing any labels for the high-stakes videos?
Methodology: Subspace Alignment (SA)
The researchers propose an Unsupervised Domain Adaptation (UDA) approach using Subspace Alignment (SA). This technique bridges the distributional gap between lab data (Source) and courtroom data (Target).
The pipeline works as follows:
- Feature Extraction: Audio features (prosody, pitch, etc.) were extracted using OpenSMILE, and visual cues (facial action units, gaze, head pose) were captured via OpenFace. These were processed into 1,068 time-series features.
- Alignment: Since lab and courtroom data have different distributions, the researchers used Principal Component Analysis (PCA) to create $D$-dimensional subspaces for both. They then calculated a linear transformation matrix to align the lab subspace with the courtroom subspace.
- Classification: Once aligned, a K-Nearest Neighbors (KNN) classifier was trained on the transformed lab data and tested on the unlabeled courtroom data.
Key Results and Insights
The results demonstrate that smart alignment techniques can overcome the need for massive labeled datasets.
- Outperforming Humans: Humans typically detect lies at near-chance levels (~51%). The best unsupervised model (Visual SA) achieved 74% accuracy and a 75% AUC, significantly clearing the human bar.
- The Value of Alignment: As shown in the performance comparison, models using Subspace Alignment (SA) consistently outperformed "baseline" models that attempted transfer without alignment across every modality.
- Competitive with Supervised Models: Perhaps most impressively, this unsupervised approach performed comparably to several state-of-the-art supervised models (which typically range from 75% to 79% accuracy) despite never seeing a single "high-stakes" label during training.
- Universal Cues: Statistical testing identified 10 behavioral signals that were remarkably similar across both lab and courtroom settings, including jitter, pitch, and yaw of head pose. These "transferable" cues are what make unsupervised learning possible.
Conclusion and Future Work
This study represents a major step forward in detecting real-world deception without the need for expensive or impossible-to-get labels. While the visual models were particularly strong—suggesting that eye gaze and head pose are consistent indicators of deception even under pressure—future work will focus on improving multimodal fusion and testing these techniques on other social behaviors, such as detecting depression or anxiety in clinical settings.
Summary updated