Acoustic Indicators of Deception in Mandarin Daily Conversations Recorded from an Interactive Game

Chih-Hsiang Huang, Huang-Cheng Chou, Yi-Tong Wu, Chi-Chun Lee, Yi-Wen LiuInterspeech 2019

TL;DR

This paper introduces the first Mandarin "daily deceptive dialogue" (DDD) corpus, collected from a novel interactive game designed to elicit naturalistic deception. They analyze acoustic features to identify deception cues, finding that individuals exhibit distinct patterns and that these patterns can be clustered, though not all individuals show clear acoustic indicators.

What Problem Does This Paper Address?

Distinguishing truth from deception is crucial in many real-world scenarios, from financial fraud to everyday interactions. While research on acoustic indicators of deception in English is relatively extensive, there's a significant gap in understanding these cues in Mandarin. Cultural differences and linguistic nuances can profoundly impact how deception is expressed and perceived, making it essential to develop specialized resources and analyses for Mandarin. This paper aims to bridge this gap by creating a new dataset and exploring acoustic features of deception in Mandarin daily conversations.

Methods

The researchers designed an interactive game where participants were encouraged to lie about personal experiences to their opponents. This game-based approach aimed to elicit more naturalistic deceptive speech compared to staged scenarios. After data collection, they extracted acoustic-prosodic features (pitch, intensity, formants, duration, silence ratio) and performed L2 normalization to account for individual differences.

Key analytical techniques included:

  • Intra-subject analysis: Using independent two-sample t-tests to identify significant acoustic differences between truthful and deceptive utterances for each individual. This led to the creation of "significance-indicator vectors" for each subject.
  • Clustering: A dendrogram was generated based on Euclidean distance and the Ward method to group subjects with similar deception indicator patterns.
  • Inter-subject analysis: T-tests were used to identify significant acoustic features across all subjects, with p-values corrected for the False Discovery Rate (FDR).
  • Automatic Deception Detection: Random Forest and Support Vector Machine (SVM) classifiers were trained using both all extracted features and only the significant indicators identified in the inter-subject analysis.

Datasets

The paper introduces a newly constructed Daily Deceptive Dialogue (DDD) corpus of Mandarin.

  • Size: Approximately 27.2 hours of clean recording data.
  • Participants: 96 native Mandarin speakers with Taiwanese accents, aged 20-25, balanced in gender.
  • Content: Daily conversations recorded during an interactive game where participants were instructed to lie about specific personal experiences.
  • Labeling: Global deception labeling was used, meaning all responses related to a particular question were labeled as either truthful or deceptive. Interviewers' perceived truth/deception labels were also collected.

This corpus differs from existing English deception datasets by focusing on naturalistic daily conversations and the Mandarin language.

Key Insights and Ideas

  • Individualized Deception Patterns: The intra-subject analysis revealed that individuals exhibit distinct acoustic patterns when lying. This suggests that deception is not a monolithic behavior acoustically, and a one-size-fits-all approach to detection might be insufficient.
  • Clustering of Deceptive Behaviors: The dendrogram analysis showed that subjects could be clustered into groups based on their common significant acoustic indicators for deception. This implies that certain groups of people might employ similar acoustic strategies when lying.
  • Absence of Indicators for Some: Notably, a significant portion of participants (31 out of 96) showed no statistically significant acoustic indicators for deception. This highlights the challenge of universal deception detection and the importance of considering individual differences.
  • Gender-Specific Acoustic Differences: Inter-subject analysis indicated that acoustic indicators for deception can differ between male and female speakers, particularly concerning formants. The authors hypothesize this might be due to differences in speech rate and tone.
  • Silence Ratio Nuance: Interestingly, the silence ratio was not found to be a significant indicator of deception production. The authors suggest this might be because participants paused not only when lying but also when preparing truthful statements to ensure credibility.

Key Results

  • Clustering: Subjects were successfully clustered into 10 groups (A-J) based on their acoustic deception indicators, with Group F (31 subjects) showing no significant indicators.
  • Significant Acoustic Indicators: Inter-subject analysis identified several acoustic features that were significantly different between truthful and deceptive speech across the corpus (e.g., Intensity mean, Intensity_pstd, Intensity_max, Formant_3_mean, Pitch_mean, Pitch_max).
  • Automatic Deception Detection Performance: The best performing classifier (Random Forest) achieved an F1 score of 53.37% for distinguishing deceptive and truthful segmentation units when using all features. When using only significant indicators, the F1 score was 53.35%. These results are comparable to human deception detection accuracy reported in other studies.

Limitations and Future Work

  • Limited Labeling: The study primarily used global deception labeling due to the experimental design. Local deception labeling (labeling individual utterances within a response) would provide finer-grained analysis.
  • Classifier Performance: The achieved F1 score of ~53% indicates that automatic deception detection based solely on acoustic features in this context is challenging. The authors acknowledge that the variance between folds in their cross-validation was large, impacting performance.
  • Future Work:
    • Incorporate Linguistic and Lexical Cues: Combine acoustic features with linguistic and lexical information for more robust deception analysis.
  • Explore More Features: Investigate additional acoustic features or more complex feature extraction methods.
  • Address Individual Differences: Further research into why some individuals lack acoustic deception indicators and how to account for this.
  • Expand Corpus: Increase the size and diversity of the DDD corpus.
  • Investigate Perceived Deception: Analyze the relationship between acoustic cues and how listeners perceive deception.

Summary updated

BibTeX
@inproceedings{huang2019acoustic,
  author = {Chih-Hsiang Huang and Huang-Cheng Chou and Yi-Tong Wu and Chi-Chun Lee and Yi-Wen Liu},
  title = {{Acoustic Indicators of Deception in Mandarin Daily Conversations Recorded from an Interactive Game}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure