TL;DR
This paper investigates acoustic-prosodic features in interview dialogues to identify deception and trust. They developed a machine learning classifier that can distinguish truthful from deceptive speech with an F1-score of 72.77%, outperforming both random and human baselines. This work advances our understanding of deception detection and has implications for creating more trustworthy AI systems.
What Problem Does This Paper Address?
Identifying deception is a long-standing challenge with significant implications across various fields, from law enforcement to human-computer interaction. While previous research has explored linguistic cues to deception, the acoustic-prosodic characteristics of both deceptive and trustworthy speech remain less understood. This paper aims to fill this gap by analyzing how vocal features like pitch, intensity, and speaking rate differ in truthful versus deceptive responses during interviews. Furthermore, it investigates how these vocal cues are perceived by interviewers, contributing to our understanding of what makes speech sound trustworthy. The study also explores whether these patterns vary across different genders and native language speakers, and whether this information can be leveraged for automated deception detection.
Methods
The researchers extracted eight key acoustic-prosodic features: intensity mean, intensity max, pitch mean, pitch max, jitter, shimmer, noise-to-harmonics ratio (NHR), and speaking rate. These features were extracted from interview responses, considering two segmentation units: the immediate response to a question (single turn) and the entire segment of responses including follow-ups (multiple turns). Paired t-tests were used to compare these features between truthful and deceptive speech, as well as between perceived truthful and deceptive speech. To automate deception detection, they employed machine learning classifiers, specifically Random Forest, Logistic Regression, and SVM, using the extracted acoustic-prosodic features. They also experimented with incorporating demographic features (gender, native language) into the classification models.
Datasets
The study utilized the Columbia X-Cultural Deception (CXD) Corpus. This corpus consists of dialogues from 340 subjects speaking in English, including native speakers of Standard American English (SAE) and Mandarin Chinese (MC). The data was collected using a variation of a fake resume paradigm where participants were instructed to provide false answers for half of the biographical questions. Interviewers recorded their judgments on the veracity of each answer, providing data for analyzing human perception of deception.
Key Insights and Ideas
- Deceptive speech is louder and higher-pitched (at its peak): Speakers tend to increase their maximum pitch and volume when lying, suggesting a conscious or unconscious effort to emphasize their statements.
- Trustworthy speech is faster: A faster speaking rate is associated with perceived truthfulness, likely because it conveys fluency and confidence. This effect is particularly pronounced in non-native speakers, suggesting that cognitive load from lying is more apparent when speaking in a second language.
- Gender and language matter for deception cues: While some acoustic-prosodic features are universal indicators of deception (e.g., increased pitch max), others are specific to certain groups. For instance, increased pitch max in deceptive speech was observed in males and native Chinese speakers, but not in females or native English speakers.
- Perception of trust is nuanced: While some features align with deception (e.g., higher intensity max in deceptive speech), others show interesting divergences. For example, speaking rate was faster in trusted speech overall, but this difference was only significant for native Chinese speakers in the deception analysis.
- Acoustic features are powerful for classification: The study demonstrates that acoustic-prosodic features alone are highly effective for automatically detecting deception, even outperforming human judgment in some cases.
Key Results
The paper reports that acoustic-prosodic features are effective indicators of deception. Specifically, pitch max and intensity max were found to be significantly increased in deceptive speech across all speakers. For perceived deception, speaking rate was significantly faster in truthful speech.
In classification experiments, the Random Forest classifier achieved an F1-score of 72.77% using acoustic-prosodic features for deception detection. This performance significantly surpassed the random baseline (50%) and the human performance baseline (56.75%) reported for this corpus. The study found that classifying based on the "single turn" (immediate response) was slightly more effective than using "multiple turns" (full conversational context). Incorporating gender and native language as additional features did not significantly improve classification performance, suggesting these aspects might already be implicitly captured by the acoustic-prosodic features.
Limitations and Future Work
The authors acknowledge that while their classifier achieved high performance, there is still room for improvement. They suggest that future work should focus on developing models for trust classification (identifying trustworthy vs. untrustworthy speech). Additionally, they plan to investigate deception corpora in other domains to identify domain-independent deception indicators and explore the integration of linguistic features alongside acoustic-prosodic ones for more robust deception detection. The study also highlights the need to further explore individual differences in deception production and perception.
Summary updated