TL;DR
This paper introduces a novel approach to automatic deception detection by incorporating the interrogator's judgment of truthfulness into the model. By analyzing speech and language features from both the deceiver and the interrogator, their proposed JEADDN model significantly improves deception detection accuracy, especially when the interrogator distrusts the deceiver.
What Problem Does This Paper Address?
Humans are notoriously bad at detecting deception, with average accuracy hovering around 54%. While automatic deception detection (ADD) systems have shown promise, most existing research focuses solely on the deceiver's behavior. This paper addresses the gap of understanding how the interrogator's perception and judgment of truthfulness can be leveraged to enhance ADD systems. The authors hypothesize that the interrogator's belief (trust or distrust) in the deceiver's statements provides a crucial contextual condition that can make deceptive behaviors more discernible. They also highlight the underrepresentation of Eastern cultures in deception research and aim to contribute to this area.
Methods
The core of this work is the Judgmental-Enhanced Automatic Deception Detection Network (JEADDN). This model integrates three types of speech-language features: acoustic-prosodic, linguistic, and conversational temporal dynamics (CTD). The novelty lies in using the interrogator's judgment as a condition to select specific classifiers, rather than as a feature itself.
The JEADDN architecture is based on a Bidirectional Long Short-Term Memory (BLSTM) layer with an attention mechanism, followed by fully-connected layers. For feature extraction, they utilize pre-trained models and established toolkits:
- Audio: Emobase (acoustic-prosodic features) and XLSR-53 (multilingual speech representation).
- Text: BERT and RoBERTa (contextual word embeddings), POS (Part-Of-Speech tags), NER (Named Entity Recognition), and LIWC (linguistic inquiry and word count).
- Conversational Temporal Dynamics (CTD): Features capturing turn-taking dynamics.
The model employs a late fusion strategy, combining the best feature sets from each modality. Crucially, the JEADDN is trained and evaluated under three conditions:
- Truthful and deceptive statements detection: No interrogator judgment considered (baseline).
- Trusted truthful and deceptive statements detection: Interrogator's judgment is "trusted."
- Distrusted truthful and deceptive statements detection: Interrogator's judgment is "distrusted."
Datasets
The primary dataset used is the Daily Deceptive Dialogues corpus of Mandarin (DDDM). This corpus is unique because it includes not only audio and transcripts but also the judgments of both interlocutors (interrogator and deceiver). The authors state that most deception databases lack these crucial interrogator judgments. They used 7126 utterances, transformed into 2764 Questioning-Answering (QA) pair turns for their analysis.
Key Insights and Ideas
- Interrogator's Judgment is Key: The interrogator's belief (trust or distrust) significantly impacts deception detection performance. This is a novel contribution, moving beyond solely analyzing the deceiver.
- Distrust Amplifies Deception Cues: When an interrogator distrusts the deceiver, the interrogator's own speech and language behaviors become more informative for detecting deception than the deceiver's.
- Human Deception Detection is Poor: The study provides further evidence that humans are inherently poor at detecting deception, with very few overlapping indicators across different scenarios.
- Contextual Information Matters: The interrogator's interpretation of the conversation's context, particularly their trust level, is a critical factor that can be leveraged by ADD systems.
- NER Features are Promising: Named Entity Recognition (NER) features, particularly in the context of interrogators asking for detailed information, show promise for deception detection.
Key Results
The JEADDN model achieved significant improvements over state-of-the-art approaches on the DDDM database.
- Overall Performance: With late fusion of audio, text, and TTD features, the model achieved 87.27% accuracy in the "trusted" condition and 94.18% accuracy in the "distrusted" condition.
- Improvements: These results represent a 7.27% and 13.57% improvement over the model without considering interrogator judgments in the respective conditions.
- Feature Effectiveness: In the "distrusted" scenario (condition iii), interrogator features were found to be more effective than deceiver features.
- Modality Importance: Ablation studies indicated that text modality (BERT) was the most effective among audio, text, and TTD.
Limitations and Future Work
- Limitations:
- The authors acknowledge that while their model shows promise, the DDDM database primarily consists of university students, which might limit generalizability to other demographics.
- The interpretation of features from models like XLSR, BERT, and RoBERTa can be challenging due to their black-box nature.
- The study primarily focuses on Mandarin Chinese, and cultural variations in deception expression are noted as a factor.
- Future Work:
- Extend the multimodal fusion framework to incorporate semantic information for enhanced robustness and predicting deception across multiple QA pairs.
- Investigate scenarios where interrogators initially distrust but later trust deceivers, or vice-versa, to better understand dynamic belief shifts.
- Further research into how practitioners can corroborate the accuracy of new details provided by deceivers.
Summary updated