UNIDECOR: A Unified Deception Corpus for Cross-Corpus Deception Detection

Aswathy Velutharambath, Roman KlingerACL 2023

TL;DR

This paper introduces UNIDECOR, a unified corpus of English deception datasets, to address the challenge of comparing deception detection models across different domains. They found that deception detection models struggle to generalize across diverse datasets, highlighting the need for more robust, domain-independent approaches.

What Problem Does This Paper Address?

Automatic deception detection is a crucial task with applications in various fields, from forensics to online communication. However, research in this area is fragmented due to a lack of standardized datasets and varying methodologies. Deception itself is a complex human behavior, and its linguistic cues can be subtle and context-dependent. This makes it difficult to build models that can reliably detect deception across different types of text (e.g., online reviews, court testimonies, game dialogues) and to compare findings across studies. The authors aim to bridge this gap by creating a unified resource and analyzing deception cues across diverse datasets.

Methods

The core of this work is the creation of the UNIDECOR corpus, which consolidates 14 publicly available English deception datasets into a single, unified format. This involves standardizing labels and aggregating relevant annotation dimensions like age, gender, country, and sentiment.

To evaluate the generalizability of deception detection models, the authors employ RoBERTa-base, a pre-trained transformer model. They conduct two main experimental setups:

  1. Within-corpus evaluation: Fine-tuning and evaluating RoBERTa on each dataset individually using 10-fold cross-validation.
  2. Cross-corpus evaluation: Training a model on one dataset and testing it on all other datasets.

Additionally, they perform a linguistic correlation analysis using features from the Linguistic Inquiry and Word Count (LIWC) lexicon and readability scores (Flesch-Kincaid, Gunning Fog) to understand how linguistic cues of deception vary across datasets. They use point-biserial correlation to measure the relationship between deception labels and these linguistic features.

Datasets

The paper introduces the UNIDECOR corpus, which is a consolidation of the following 14 datasets:

  • Deceptive Opinion Spam (OPSPAM): Online reviews.
  • Cross-cultural Deception (CROSSCULTDE): Short essays on various topics.
  • Deception in Reviews (DEREV2014/2018): Book reviews from Amazon.
  • Open Domain Deception (OPENDOMAIN): Single-sentence statements.
  • Real-life Trial Data (TRIAL): Transcripts from trial hearings.
  • Boulder Lies and Truth Corpus (BLTC): Reviews of electronic appliances and hotels.
  • Online Deceptive Reviews (ONLINEDE): Manipulated online reviews across many domains.
  • Mafiascum Dataset (MAFIASCUM): Dialogue from an online strategy game.
  • Miami University Deception Detection Database (MU3D): Interview transcripts.
  • Multimodal Deception in Dialogues (BOXOFLIES): Transcripts from a game show segment.
  • Diplomacy Dataset (DIPLOMACY): Chat messages from an online strategy game.
  • Deceptive Opinion (DECOP): Opinion statements on various topics.
  • Bluff the Listener (BLUFF): Humorous stories from a radio show.

The UNIDECOR corpus is made available for research purposes.

Key Insights and Ideas

  • Deception is highly domain-specific: The linguistic cues associated with deception are not consistent across different types of text and communication contexts. What signals deception in a game might not in a formal trial.
  • Cross-corpus generalization is a major challenge: Even with powerful pre-trained models like RoBERTa, models trained on one dataset perform significantly worse when tested on another, especially if the domains are dissimilar.
  • Existing datasets have limitations: The paper highlights issues like imbalanced label distributions (e.g., in DIPLOMACY) and long document lengths (e.g., in MAFIASCUM) that can impact model performance.
  • No universal deception cues: The linguistic correlation analysis revealed a lack of consistent, universally applicable linguistic markers for deception across all datasets.

Key Results

  • Within-corpus performance is strong: RoBERTa models achieve high F1 scores when trained and tested on the same dataset, indicating their ability to learn domain-specific deception patterns.
  • Cross-corpus performance is significantly lower: The F1 scores drop considerably when models are tested on unseen datasets, particularly those from different domains. For example, models trained on game data often perform poorly on review data.
  • Domain similarity matters: Datasets from similar domains (e.g., reviews) or with similar collection methods tend to show better cross-corpus performance than those from vastly different domains.
  • Linguistic cues are inconsistent: The correlation analysis showed weak or inconsistent relationships between LIWC categories and deception across datasets, reinforcing the idea that deception cues are not universal. For instance, the expected correlation of fewer self-references ("i") in deceptive statements was not consistently observed.

Limitations and Future Work

  • Limited model optimization: The study reports baseline performance using RoBERTa without extensive hyperparameter tuning specific to each dataset or cross-corpus scenario. Better results might be achievable with further optimization.
  • Biased datasets: Some datasets (e.g., MAFIASCUM, ONLINEDE, DIPLOMACY) have imbalanced label distributions, favoring truthful labels, which can lead to artificially low F1 scores for deception.
  • Future Work: The authors plan to explore techniques like oversampling to address class imbalance. Their ultimate goal is to develop robust deception detection models that can generalize across corpora and domains, requiring a deeper understanding of how deception is conceptualized and linguistically realized across different contexts. They also emphasize the ethical considerations of using deception detection on non-anonymous data.

Summary updated

BibTeX
@inproceedings{velutharambath2023unidecor,
  author = {Aswathy Velutharambath and Roman Klinger},
  title = {{UNIDECOR: A Unified Deception Corpus for Cross-Corpus Deception Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure