TL;DR
This paper introduces OpinionGPT, a novel demonstration that makes biases in instruction-tuned LLMs explicit rather than trying to suppress them. By fine-tuning a Llama 2 model on data curated from Reddit subreddits representing specific demographic and political biases, OpinionGPT allows users to compare model responses across these different "opinions," fostering a deeper understanding of how bias manifests in AI.
What Problem Does This Paper Address?
Large Language Models (LLMs), particularly those fine-tuned for instruction following, have become incredibly adept at generating human-like text. However, a significant challenge remains: these models often inherit and perpetuate biases present in their training data. While much research focuses on mitigating or removing these biases, this paper argues for an alternative approach: making biases explicit and transparent. The authors aim to address the question of how different demographic and ideological perspectives, as reflected in online text, influence LLM outputs, and to provide a tool for researchers and users to explore these influences.
Methods
The core of OpinionGPT lies in its bias-aware instruction-tuning data and a Mixture-of-Experts (MoE) model architecture.
- Bias-Aware Corpus Creation: The authors leveraged Reddit's "AskX" subreddits (e.g., r/AskAGerman, r/AskAnAmerican) where specific demographic groups are expected to answer questions. They manually selected 13 subreddits representing 11 distinct biases: political (liberal, conservative), geographic (USA, Germany, Middle East, Latin America), age (teenager, over 30, over 45), and gender (male, female). Instruction-response pairs were created using post titles as instructions and the most upvoted direct responses as answers, with several filtering steps to ensure data quality and relevance.
- Model Training: A Llama 2 13B parameter model was used. Instead of full fine-tuning, they employed a LoRA (Low-Rank Adaptation) approach, training a separate LoRA adapter for each of the 11 identified biases. This resulted in an MoE model where a specific bias adapter can be selected during inference. This approach was chosen for its ability to capture biases more precisely with less overlap.
- Web Demonstration: A user-friendly web interface was developed, built upon the open-source Chat UI project. Users can input a question and select one or more bias groups to see side-by-side comparisons of the model's responses, each highlighted in a different color.
Datasets
- Training Data: A custom corpus derived from 13 specific Reddit "AskX" subreddits, curated to represent 11 distinct biases (political, geographic, age, gender).
- Evaluation Data: The BOLD dataset (Dhamala et al., 2021) was used for quantitative evaluation. This dataset consists of Wikipedia prompts designed to measure attitudes towards various demographics and ideologies.
Key Insights and Ideas
- Transparency over Suppression: The paper champions making biases explicit as a valuable research tool, rather than solely focusing on their removal. This allows for a deeper understanding of how different perspectives shape AI outputs.
- Reddit as a Proxy for Bias: The use of Reddit subreddits provides a direct, albeit imperfect, window into how specific demographic and ideological groups express themselves on various topics.
- MoE for Granular Bias Control: The Mixture-of-Experts approach, with individual LoRA adapters for each bias, allows for more precise control and differentiation between the learned biases.
- Bias Conflation is a Challenge: The authors acknowledge that biases can become conflated during training, leading to potential information leakage and less clear bias boundaries. For example, a "woman" from Australia might respond differently than a "woman" from the US.
Key Results
- Qualitative Evaluation: The model demonstrated the ability to generate nuanced responses reflecting the underlying biases of the selected demographics. For instance, responses to "stricter gun laws?" varied significantly across different bias groups, showcasing distinct viewpoints.
- Quantitative Evaluation (BOLD Dataset): The "Liberal" bias group showed a higher proportion of negative regard towards several demographics and a negative sentiment towards liberalism itself. Conversely, "women" and "people over 30" tended to exhibit more positive sentiment and regard, potentially reflecting a more polite language style on Reddit.
Limitations and Future Work
- Reddit's Limitations: The authors acknowledge that their model represents "Americans who post on Reddit" rather than all Americans, highlighting the inherent limitations of using online forums as a data source.
- Bias Overlap: The multifaceted nature of biases means that individual responses can be influenced by multiple overlapping biases, making it challenging to isolate a single bias's effect.
- Future Work: The authors plan to investigate sources of bias leakage and enable more granular and compositional representations of biases (e.g., "conservative Germans") in future versions of OpinionGPT.
Summary updated