TL;DR
This paper introduces a novel Self-Attention-Based Autoencoder (SABA) model to detect faked responses in questionnaires. SABA uniquely relies solely on honest responses for training, outperforming existing methods by identifying specific faked answers rather than just a general tendency to fake.
What Problem Does This Paper Address?
Questionnaires are widely used for data collection, but respondents may intentionally alter their answers to gain an advantage (e.g., in job applications). Existing methods for detecting this "faking" often identify a general propensity to lie but struggle to pinpoint which specific answers are faked. Furthermore, these methods typically require extensive datasets of both honest and faked responses tailored to the specific context of the questionnaire, making them costly and difficult to apply in real-world scenarios. This paper aims to address this by developing a method that can reliably identify individual faked responses using only honest data.
Methods
The core of the proposed solution is a Self-Attention-Based Autoencoder (SABA). Autoencoders are unsupervised deep learning models that learn to reconstruct their input from a compressed representation. SABA enhances this by incorporating a self-attention mechanism, inspired by transformer models.
Here's a breakdown:
- Autoencoder Architecture: The model takes a sequence of questionnaire responses as input. It first embeds each response into a vector. Then, a self-attention layer processes these embeddings to capture dependencies and correlations between different responses. Finally, a feedforward layer reconstructs the input.
- Self-Attention Mechanism: This layer allows the model to weigh the importance of different responses when reconstructing a masked response. It helps in identifying subtle patterns and relationships within the honest responses.
- Training Strategy (Masked Language Model - MLM): Similar to BERT, SABA is trained by masking a random subset of responses in a questionnaire and then tasking the model to predict these masked values based on the remaining ones. This is done using only honest responses. The model learns to reconstruct honest response patterns.
- Faking Detection: During inference, the trained SABA model is used to reconstruct masked responses in a potentially faked questionnaire. If the reconstructed value significantly differs from the actual faked response (beyond a certain threshold), that response is flagged as faked. This leverages the idea that faked responses will deviate from the learned honest response patterns.
Datasets
The study utilized data collected from 694 participants who completed a personality questionnaire twice: once honestly and once while faking their responses. The questionnaire was a short version of the Big-Five personality test. Data was collected in three distinct scenarios, each representing a different faking objective:
- Child Custody Litigation (CC): Participants faked responses to appear more suitable for obtaining child custody.
- Job Interview, Salesperson (JIS): Participants faked responses to present themselves as ideal candidates for a sales position.
- Job Interview, Humanitarian Organization (JIHO): Participants faked responses to be hired by a humanitarian organization.
These three datasets (CC, JIS, JIHO) were used for training and evaluating the model.
Key Insights and Ideas
- Leveraging Honest Patterns: The most significant insight is that a robust faking detection system can be built by learning the intricate patterns and correlations present only in honest responses. This bypasses the need for extensive, context-specific faking data.
- Spotting Individual Fakes: Unlike previous methods that identify a general tendency to fake, SABA can pinpoint specific questions that have been altered, offering more granular insights.
- Self-Attention for Context: The self-attention mechanism is crucial. It allows the model to understand how different responses relate to each other, enabling it to detect deviations from established honest patterns that might be missed by simpler models.
- Faking Objectives Matter: The study observed that faking patterns differ based on the objective (e.g., child custody vs. job interview), highlighting the complexity of deception.
- Correlation as a Signal: The research found that while honest responses show complex intercorrelations, faked responses often exhibit less nuanced or even nonexistent correlations with honest patterns, providing a key signal for detection.
Key Results
- Superior Performance: SABA achieved the highest F1 scores across all three datasets compared to three baseline models: a feedforward autoencoder, a distribution model, and a k-nearest-neighbor model.
- For example, on the CC dataset, SABA achieved an F1 score of 0.6335, compared to 0.5742 for the feedforward autoencoder, 0.5482 for Bi-LSTM, 0.4057 for the distribution model, and 0.5024 for the k-NN model.
- High Precision and Recall: SABA generally demonstrated a good balance between precision (correctly identifying faked responses) and recall (finding most of the faked responses).
- Robustness to Faking Amount: The model's performance remained relatively stable when the proportion of faked responses was high (over 60%), though it decreased linearly with fewer faked responses.
- Low False Negatives: The model exhibits a high sensitivity to faking and a low incidence of false negatives, meaning honest responses are unlikely to be flagged as faked. The average reconstruction error on honest data was very low.
Limitations and Future Work
- Performance Degradation with Few Fakes: SABA's performance decreases as the number of faked responses becomes very small, as it relies on identifying deviations from learned patterns.
- Interpretability: While SABA flags faked responses, the paper acknowledges the need for models that can provide human-understandable explanations for their predictions.
- Future Directions:
- Pre-training on Public Data: Fine-tuning the model on domain-specific faking behaviors after pre-training on large public datasets could improve performance.
- Ethical Considerations: The authors note that identifying faked responses is a sensitive task, and the application of automated approaches has ethical implications that warrant further discussion.
- Explainable AI: Developing models that offer interpretable predictions is a key area for future research.
Summary updated