Identity Unbiased Deception Detection by 2D-to-3D Face Reconstruction

Lê Minh Ngô, Wei Wang, Burak Mandira, Sezer Karaoglu, Henri Bouma, Hamdi Dibeklioglu, Theo GeversWACV 2021 · OpenReview

Abstract

Deception is a common phenomenon in society, both in our private and professional lives. However, humans are notoriously bad at accurate deception detection. Based on the literature, human accuracy of distinguishing between lies and truthful statements is 54% on average, in other words, it is slightly better than a random guess. While people do not much care about this issue, in high-stakes situations such as interrogations for series crimes and for evaluating the testimonies in court cases, accurate deception detection methods are highly desirable. To achieve a reliable, covert, and non-invasive deception detection, we propose a novel method that disentangles facial expression and head pose related features using 2D-to-3D face reconstruction technique from a video sequence and uses them to learn characteristics of deceptive behavior. We evaluate the proposed method on the Real-Life Trial (RLT) dataset that contains high-stakes deceits recorded in courtrooms. Our results show that the proposed method (with an accuracy of 68%) improves the state of the art. Besides, a new dataset has been collected, for the first time, for low-stake deceit detection. In addition, we compare high-stake deceit detection methods on the newly collected low-stake deceits.

Introduction

Detecting lies is notoriously difficult; the average human is only about 54% accurate—barely better than a coin flip. While high-stakes environments like courtrooms or interrogations desperately need reliable tools, traditional methods like polygraphs are often unreliable or easily "gamed." In the field of Computer Vision, previous automated attempts have struggled with two main issues: identity bias and environmental sensitivity. Identity bias occurs when models accidentally learn to associate specific individuals with lying rather than learning the behavior itself. Environmental sensitivity refers to changes in lighting, camera angle, or background that can confuse models relying on raw pixels or 2D landmarks.

Proposed Method: 2D-to-3D Disentanglement

The core innovation of this work is the use of 2D-to-3D face reconstruction to separate facial features into distinct components. By converting a 2D image into a 3D model, the system can mathematically separate the "shape" of a smile from the "identity" of the person smiling.

The proposed architecture for 2D-to-3D reconstruction and deceptive prediction.

As shown in Figure 1, the architecture consists of two parallel Convolutional Neural Network (CNN) backbones:

  • The Identity CNN: Extracts "static" features such as facial geometry (bone structure), skin albedo (reflectance), and ambient lighting conditions.
  • The Temporal CNN: Focuses on "dynamic" features—specifically facial expressions and head pose (rotation and translation).

For the final deception prediction, the model ignores the identity and environment data, feeding only the expression and pose sequences into a Long Short-Term Memory (LSTM) network. This ensures the model makes its decision based on movement patterns, such as micro-expressions or nervous tics, rather than the person's appearance. The system treats the problem as Multiple Instance Learning (MIL), assuming that if a video is labeled as a "lie," at least one segment contains deceptive behavior.

Datasets and Evaluation

The method was evaluated on two primary benchmarks:

  1. Real-Life Trial (RLT): A public dataset of 121 high-stakes videos from real courtrooms. This dataset is challenging due to environmental noise, such as multiple faces in a scene or the presence of prosecutors.
  2. Low-Stakes Deceit (LSD): A newly introduced dataset containing 624 high-resolution recordings of 312 subjects. These involve "low-stakes" lies (e.g., describing a card they didn't see), providing a benchmark for behavior without severe consequences.
Examples of challenging scenes in the RLT dataset, including multiple faces and non-target individuals.

Key Results

  • State-of-the-Art Performance: On the high-stakes RLT dataset, the proposed method achieved 68% accuracy. This is a significant improvement over previous automatic methods (56-59%) and is comparable to manual methods requiring human experts (71%).
  • Beating the Human Baseline: The model's 68% accuracy outperforms the reported human average for high-stakes deceit detection.
  • The Difficulty of Low-Stakes: On the new LSD dataset, accuracy dropped to roughly 54% for all automated methods. This suggests that without the pressure of a courtroom, deceptive cues are much subtler and remain a major challenge for current AI.

Conclusion

While the 3D reconstruction approach is a leap forward in creating identity-unbiased systems, limitations remain. The current model is unimodal, focusing only on the face and head, whereas deception often leaks through voice pitch and speech patterns. Furthermore, the "low-stakes wall" indicates that everyday lies are significantly harder to detect than high-stakes courtroom testimony. Future work will investigate demographic nuances, such as age and gender, to ensure fairness and accuracy across different populations.

Summary updated

Abstract

Deception is a common phenomenon in society, both in our private and professional lives. However, humans are notoriously bad at accurate deception detection. Based on the literature, human accuracy of distinguishing between lies and truthful statements is 54% on average, in other words, it is slightly better than a random guess. While people do not much care about this issue, in high-stakes situations such as interrogations for series crimes and for evaluating the testimonies in court cases, accurate deception detection methods are highly desirable. To achieve a reliable, covert, and non-invasive deception detection, we propose a novel method that disentangles facial expression and head pose related features using 2D-to-3D face reconstruction technique from a video sequence and uses them to learn characteristics of deceptive behavior. We evaluate the proposed method on the Real-Life Trial (RLT) dataset that contains high-stakes deceits recorded in courtrooms. Our results show that the proposed method (with an accuracy of 68%) improves the state of the art. Besides, a new dataset has been collected, for the first time, for low-stake deceit detection. In addition, we compare high-stake deceit detection methods on the newly collected low-stake deceits.
BibTeX
@inproceedings{ngo2021identity,
  author = {Lê Minh Ngô and Wei Wang and Burak Mandira and Sezer Karaoglu and Henri Bouma and Hamdi Dibeklioglu and Theo Gevers},
  title = {{Identity Unbiased Deception Detection by 2D-to-3D Face Reconstruction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=VebisssEgk1},
}
Figure