TL;DR
This paper explores identifying deception in dialogues by analyzing linguistic features within the game of Werewolf. They demonstrate that psycholinguistic and syntactic features, particularly production rules, are highly effective in distinguishing deceptive (werewolf) players from non-deceptive ones, paving the way for more robust dialogue-based deception detection systems.
What Problem Does This Paper Address?
Deception detection is a challenging NLP task, with most prior work focusing on static text like reviews or arguments. However, detecting deception in dynamic, high-conflict dialogues, where participants have conflicting goals and are highly motivated to mislead, remains largely unexplored. This is particularly relevant in scenarios like negotiations or trials. The authors identify the game of Werewolf as an ideal testbed for this problem, as it inherently involves deception and manipulation for players to achieve their objectives.
Methods
The researchers propose a binary classification task to distinguish between werewolf (deceptive) and non-werewolf (non-deceptive) roles. They extract a rich set of features from player utterances, categorizing them into:
- Psycholinguistic Features: These include standard measures like word count, negative emotion words (using MPQA lexicon), motion, sense, and exclusion words. They also incorporate features related to cognitive complexity, such as long words, conjunctions, and cognitive words, drawing inspiration from previous deception detection studies.
- Syntactic Features: Moving beyond lexical analysis, they leverage Part-of-Speech (POS) tags and dependency parses. Crucially, they introduce production rules extracted from constituency and dependency parses, hypothesizing that complex syntax reflects cognitive complexity and persuasive intent.
They experiment with various classifiers, including Logistic Regression, SVM, Naive Bayes, Decision Trees, Voted Perceptron, and Random Forest, using 10-fold cross-validation. Feature selection is also performed to identify the most impactful features.
Datasets
The study utilizes a dataset of 86 game transcripts from the game of Werewolf, collected by Barnwell. Each transcript contains an average of 205 messages. The authors manually annotated role assignments for 72 transcripts and used player self-disclosure for the remaining 14. The final dataset comprises 701 instances, with 116 instances of the werewolf role. The dataset is resampled to address its skewed distribution.
Key Insights and Ideas
- Werewolf as a Deception Sandbox: The game of Werewolf provides a unique, high-stakes environment where deception is not only permitted but essential for success, making it a valuable domain for studying deceptive communication.
- Syntax Matters (A Lot): The most significant finding is the power of syntactic features, particularly production rules, in detecting deception. This suggests that the underlying structure of language, not just the words themselves, carries crucial information about a player's deceptive intent.
- Cognitive Load and Deception: The authors' findings align with the idea that maintaining deception, even in a game context, can impose a cognitive load, reflected in language complexity and potentially verbosity.
- Reversed Associations: Interestingly, some features that previous studies linked to deception (e.g., sense words) or honesty (e.g., exclusion words) showed reversed associations in this dialogue-based, identity-concealment context. This highlights the context-dependent nature of deception cues.
- Game vs. Real-World Deception: The study suggests that deception in game or dialogue contexts might differ from deception in storytelling or argument contexts, with identity concealment potentially favoring logical over emotional presentation.
Key Results
- Ensemble Methods Shine: Ensemble methods like Voted Perceptron and Random Forest significantly outperformed basic classifiers, achieving an F1 score of 90.8% and 90.8% respectively.
- Production Rules Boost Performance: When combined with psycholinguistic features, production rules (POS+con) led to a substantial improvement in performance, reaching an F1 score of 90.6%.
- Best Performance with Syntactic Features: The best overall performance was achieved using a combination of POS tags and syntactic features (POS+syn), reaching an F1 score of 91.6%. Further refinement with tf-idf weighting (POS+syn (tf-idf)) pushed the F1 score to an impressive 92.3%.
- Werewolf Characteristics: The analysis of the decision tree revealed that non-werewolves tend to use fewer words, no connectors, and no negations. Werewolves, conversely, tend to use more words, more adjectives, fewer negative emotion words, and not many words longer than 6 letters.
Limitations and Future Work
- Dataset Size: The authors acknowledge that the dataset is relatively small, limiting the complexity of models that can be trained.
- Contextual Nuances: The study did not consider word sequence, player interaction, individual player characteristics, or non-literal meaning, which could provide further insights.
- Future Directions: The researchers plan to collect more data to enable richer models that incorporate utterance sequence and dialogue features. They also aim to further investigate the deeper nonlinear dependencies between features and the internal structure of ensemble methods.
Summary updated