TL;DR
This paper investigates how well large language models (LLMs) memorize and generate conspiracy theories without access to their training data. The authors found that LLMs, particularly larger ones and those with lower temperature settings, are prone to generating conspiracy theories, highlighting a significant risk of spreading misinformation.
What Problem Does This Paper Address?
As natural language generation (NLG) models become more sophisticated and widely adopted, there's a growing concern about their potential to generate and spread harmful misinformation. While previous research has focused on biases in social media, this paper tackles the less-explored issue of whether LLMs themselves have memorized and can readily produce conspiracy theories. This is particularly problematic because these models are often treated as black boxes, making it difficult to audit their training data for problematic content. The authors aim to understand how to detect and potentially mitigate the generation of such harmful content.
Methods
The researchers employed a multi-pronged approach:
- Dataset Creation: They curated two datasets:
- Wikipedia Dataset: Consisting of 257 Wikipedia pages related to conspiracy topics, refined to 17 well-known topics based on human consensus.
- General Dataset: Generated by prompting GPT-2 Large with "The conspiracy theory is that" at various temperature settings, yielding 5000 theories per setting.
- Elicitation of Conspiracy Theories: They used GPT-2 Large and its variants (Small, Medium, Large) with varying temperature settings (0.25 to 1.0) to generate text.
- Human Evaluation: Amazon Mechanical Turk workers were used to evaluate generated text, determining if it affirmed a conspiracy theory. This involved providing workers with reference texts and asking them to identify if generated statements aligned with known conspiracy narratives.
- Automated Evaluation: To address the scalability limitations of human evaluation, they explored an automated approach by correlating model perplexity with Google search results (using BLEU scores) for generated conspiracy theories.
- Linguistic Analysis: They analyzed the sentiment and diversity of generated conspiracy theories across different model settings.
Datasets
- Wikipedia Dataset: A curated list of 17 well-known conspiracy theory topics derived from Wikipedia.
- General Dataset: Machine-generated conspiracy theories created by prompting GPT-2 Large.
Key Insights and Ideas
- Memorization is Deep-Rooted: Even without explicit prompts, LLMs demonstrate a significant propensity to generate conspiracy theories, suggesting these narratives are deeply embedded in their training data.
- Model Size Matters: Larger models (like GPT-2 Large) are more likely to memorize and generate conspiracy theories compared to smaller models (GPT-2 Small). Reducing model size appears to be a viable strategy for mitigating this.
- Temperature is a Key Lever: Lowering the temperature setting in LLMs increases the likelihood of generating conspiracy theories. This suggests that controlling generation randomness is crucial.
- Human Evaluation is Costly but Necessary: While automated methods are explored, human evaluation remains critical for accurately identifying conspiracy theories, especially given their nuanced and often non-keyword-driven nature.
- Conspiracy Theories are Emotionally Charged: The linguistic analysis revealed that generated conspiracy theories tend to be negative in sentiment, aligning with existing research on the emotional underpinnings of belief in such theories.
Key Results
- GPT-2 Large: Generated conspiracy theories in nearly 10% of cases at a temperature of 1.0. This percentage increased significantly at lower temperatures.
- Model Size Impact: GPT-2 Medium generated almost 50% fewer conspiracy theories than GPT-2 Large, and GPT-2 Small generated over 90% fewer.
- Temperature Impact: As temperature decreased, the percentage of conspiracy theories generated by GPT-2 Large generally increased, with notable spikes for topics like UFOs, 9/11, Holocaust, Flat Earth, Illuminati, and Moon landing.
- Automated Evaluation Correlation: A strong relationship was found between a generated conspiracy theory's perplexity and its appearance in Google search results, suggesting potential for automated detection. This correlation weakened at higher temperatures.
- Sentiment Analysis: Decreasing model temperature led to more negative sentiment in generated conspiracy theories.
Limitations and Future Work
- Human Evaluation Scalability: The authors acknowledge that human evaluation, while effective, is not scalable for large-scale detection of conspiracy theories.
- Automated Evaluation Challenges: While progress was made in correlating perplexity with search results, further research is needed to refine automated evaluation techniques.
- Adversarial Attacks: The authors note that adversaries could potentially adapt their methods to hide harmful content, making detection more challenging.
- Future Directions:
- Data Curation: Preventing harmful data from entering training sets by using reliable sources and creating databases of debunked conspiracy theories.
- Model Size Reduction: Continuing to explore the trade-offs between model size, memorization, and fluency.
- Factual Oversampling: Supplementing training data with factually correct information to counterbalance conspiracy narratives.
- Early Intervention: Emphasizing the need for proactive measures rather than relying solely on post-generation detection.
Summary updated