Identifying the Optimal Features in Multimodal Deception Detection

Amin Derakhshan, Mohammad Mikaeili, Tom Gedeon, Ali Motie NasrabadiMultimodal Technol. Interact 2020 · DOI

TL;DR

This paper introduces a multimodal approach to deception detection using facial thermal imaging, physiological signals (GSR, PPG), and advanced feature selection techniques. They demonstrate that focusing on specific facial regions, particularly the perinasal and chin areas, significantly improves deception detection accuracy, outperforming traditional physiological measures alone.

What Problem Does This Paper Address?

Detecting deception is a critical challenge in various fields, from law enforcement to security. Traditional methods like polygraphy, while widely used, have limitations including invasiveness, reliance on expert interpretation, and potential to induce anxiety. While facial thermal imaging offers a non-contact alternative for monitoring physiological stress responses associated with deception, previous studies have yielded inconsistent results regarding classification accuracy and the identification of relevant facial regions. This paper aims to address this by systematically investigating the effectiveness of facial thermal imaging, identifying optimal facial regions of interest (ROIs), and comparing its performance against established physiological measures within a multimodal framework.

Methods

The study employs a multimodal approach, integrating facial thermal imaging with Galvanic Skin Response (GSR) and Photoplethysmography (PPG) signals. The core methodology involves:

  • Data Acquisition: Collecting thermal, GSR, and PPG data from participants under two distinct scenarios: a "mock crime" and a "best friend" scenario.
  • Preprocessing: Standard signal processing techniques were applied to both thermal and physiological data, including bandpass filtering and normalization for physiological signals, and face tracking and smoothing for thermal videos.
  • Feature Extraction: Six temporal features (mean, min, max, standard deviation, and derivatives) were extracted from each signal.
  • Region of Interest (ROI) Selection: Five specific facial ROIs (periorbital, forehead, cheeks, perinasal, and chin) were identified based on anatomical knowledge and previous research.
  • Feature Selection: A crucial aspect of the methodology is the application of four common univariate filtering methods (t-test, relative entropy, ROC, and Mann-Whitney-Wilcoxon) to identify the most discriminative features and ROIs. The ROC (Receiver Operating Characteristic) method, in particular, was highlighted for its effectiveness.
  • Classification: Four classifiers were used: Decision Tree, K-Nearest Neighbors (KNN), Linear Discriminant Analysis (LDA), and Support Vector Machine (SVM) with Gaussian and polynomial kernels.
  • Cross-Validation: A "Leave One Out" cross-validation strategy was employed to ensure robust evaluation of the models.

Datasets

The study introduces a newly collected dataset named THEPHY, comprising multimodal data from 41 participants. This dataset includes:

  • Facial thermal imaging videos.
  • GSR (Galvanic Skin Response) signals.
  • PPG (Photoplethysmography) signals.

The data was collected under two scenarios:

  1. Mock Crime: Participants were assigned as either innocent or deceptive, with deceptive participants instructed to steal a necklace. This scenario aimed to induce a realistic stress response.
  2. Best Friend: Participants were asked to discuss their best friend (truthful) or someone they dislike (deceptive).

The THEPHY dataset is described as being under embargo for 12 months post-publication for commercialization, with requests for data considered thereafter.

Key Insights and Ideas

  • Perinasal and Chin ROIs Dominate: The study strongly suggests that the perinasal and chin regions of the face are the most sensitive indicators of deceptive anxiety, showing greater discriminatory power than other facial areas. This finding is crucial for optimizing future thermal imaging-based deception detection systems.
  • Feature Selection is Key: Employing feature selection techniques, especially the ROC method, significantly boosts classification accuracy. This highlights the importance of not just collecting data but intelligently selecting the most informative features and ROIs.
  • Thermal Imaging Outperforms Physiological Signals: In the "mock crime" scenario, facial thermal imaging achieved a notably higher classification accuracy (90.3%) compared to GSR (70.9%) and PPG (64.5%) after optimal feature selection. This suggests thermal imaging can be a powerful standalone or complementary tool.
  • Mock Crime Scenario is More Effective: The "mock crime" scenario yielded significantly better deception detection results than the "best friend" scenario, indicating that a more realistic and high-stakes situation is more effective in eliciting detectable physiological stress responses.
  • Multimodal Synergy: While thermal imaging showed strong performance, the paper implies that a multimodal approach, combining thermal with physiological signals, could offer further improvements, though the current study's thermal results were superior.

Key Results

  • Mock Crime Scenario: Achieved a peak accuracy of 90.3% using LDA with ROC feature selection on thermal data.
  • Best Friend Scenario: Achieved a peak accuracy of 74.7% using ROC feature selection.
  • ROI Importance: The perinasal and chin areas were identified as the most influential ROIs, appearing frequently in the top-ranked features.
  • Feature Selection Impact: The ROC feature selection method led to a substantial increase in accuracy, for example, from 41.9% to 90.3% for thermal data in the mock crime scenario using LDA.
  • Comparison to Other Modalities: Thermal imaging results outperformed GSR and PPG in the mock crime scenario, achieving higher accuracy rates.

Limitations and Future Work

  • Dataset Size: While 41 participants is a reasonable starting point, the authors acknowledge that a larger dataset would be beneficial for more robust generalization.
  • Scenario Realism: Although the "mock crime" scenario was designed to be realistic, future work could explore even more ecologically valid and high-stakes deception detection scenarios.
  • Tracker Improvements: The authors note that further improvements are needed for face and ROI trackers in thermal imaging to handle natural head movements more effectively. They mention using Kinect data for this purpose in future research.
  • FACS Integration: Investigating correlations between specific thermal Action Units (AUs) from the Facial Action Coding System (FACS) and deception could provide deeper insights.
  • Collaboration: The authors emphasize the need for interdisciplinary collaboration between psychology and computer science researchers to advance this field.
  • Unobtrusive Framework: The ultimate goal is to develop a reliable, unobtrusive, and ubiquitous framework for non-contact deception detection.

Summary updated

BibTeX
@article{derakhshan2020identifying,
  author = {Amin Derakhshan and Mohammad Mikaeili and Tom Gedeon and Ali Motie Nasrabadi},
  title = {{Identifying the Optimal Features in Multimodal Deception Detection}},
  journal = {Multimodal Technol. Interact.},
  year = {2020},
  doi = {10.3390/mti4020025},
}
Figure