Abstract
The adoption of natural language generation (NLG) models can leave individuals vulnerable to the generation of harmful information memorized by the models, such as conspiracy theories. While previous studies examine conspiracy theories in the context of social media, they have not evaluated their presence in the new space of generative language models. In this work, we investigate the capability of language models to generate conspiracy theory text. Specifically, we aim to answer: can we test pretrained generative language models for the memorization and elicitation of conspiracy theories without access to the model's training data? We highlight the difficulties of this task and discuss it in the context of memorization, generalization, and hallucination. Utilizing a new dataset consisting of conspiracy theory topics and machine-generated conspiracy theories helps us discover that many conspiracy theories are deeply rooted in the pretrained language models. Our experiments demonstrate a relationship between model parameters such as size and temperature and their propensity to generate conspiracy theory text. These results indicate the need for a more thorough review of NLG applications before release and an in-depth discussion of the drawbacks of memorization in generative language models.
Introduction
As Natural Language Generation (NLG) models become integrated into everyday tools like chatbots and creative writing assistants, there is a growing risk that they will propagate harmful misinformation. While the role of social media in spreading conspiracy theories is well-documented, the "black box" nature of generative models presents a novel challenge. Conspiracy theories are notoriously difficult to detect; they lack a unified vocabulary and are instead defined by narrative structures that oppose "official" accounts. This study seeks to determine if a model can be audited for such harmful content without direct access to its massive training datasets.
Methodology
To investigate how GPT-2 handles conspiratorial content, researchers employed several strategies to elicit and analyze text:
- Neutral Prompting: Wikipedia conspiracy titles were modified into neutral prompts (e.g., "Flat Earth" became "The Earth is") to see if the model would spontaneously generate conspiratorial claims.
- Parameter Variation: The team tested the effects of model size (Small, Medium, Large) and temperature (a hyperparameter controlling prediction randomness).
- Validation: Human workers via Amazon Mechanical Turk verified if generated sentences actually affirmed a conspiracy theory.
- Memorization Proxy: To check for memorization, generated theories were queried on Google Search. High similarity (measured via BLEU score) between model output and search snippets was used as an indicator of regurgitated training data.
- Linguistic Analysis: Sentiment analysis (VADER, TextBlob, DistilBERT) and diversity metrics (BERTScore) were applied to identify unique linguistic traits in conspiratorial text.
Key Results
The Impact of Model Scaling
The study found a clear correlation between model parameters and the propensity for misinformation. As models grow in size, their capacity to store and recall fringe narratives increases significantly. GPT-2 Large was found to generate conspiracy theories in nearly 10% of its outputs when prompted with specific topics, a stark contrast to the ~1% generated by GPT-2 Small.
The "Temperature" Trap
While "hallucinations" are often associated with high-randomness (high temperature) settings, the researchers discovered the opposite for conspiracy theories. Lower temperatures—which favor the most probable next words—actually led to a higher frequency of conspiratorial generations. For GPT-2 Large, decreasing the temperature from 1.0 to 0.25 increased the percentage of conspiratorial generations from approximately 14% to 24%.
Discussion and Insights
These findings suggest that conspiracy theories are deeply rooted in the models' training data. The fact that lower temperatures increase conspiratorial output indicates that, for certain prompts, the model views these theories as the most "probable" or "correct" continuation of the text. Furthermore, generated theories consistently exhibited more negative sentiment than neutral text, aligning with psychological research that links conspiracy beliefs to negative emotions and a sense of powerlessness.
Conclusion
Simply filtering offensive keywords is insufficient for mitigating the risk of conspiracy theory generation, as these narratives often use mundane language to weave harmful claims. The authors argue that future work must focus on dataset curation, ensuring that factually accurate "debunking" information is sufficiently represented during training to counter the prevalence of conspiracy theories on the open web. These results underscore the urgent need for thorough safety reviews of NLG applications before their public release.
Summary updated