Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

Irene Solaiman, Christy DennisonNeurIPS 2021

Introduction

Large language models (LLMs) are trained on the vast, unfiltered expanse of the internet, often leading them to mirror human biases, generate toxic content, and adopt harmful stereotypes. While there is a broad consensus that models should be "safe," defining "good" behavior is inherently subjective; what is appropriate in one culture or legal framework might not be in another.

The challenge is twofold: First, how do we adjust model behavior without retraining the entire system from scratch, which is prohibitively expensive? Second, how can we make this adjustment flexible enough for different organizations to instill their own specific cultural or ethical values?

The PALMS (Process for Adapting Language Models to Society) framework demonstrates that aligning large language models with specific social values does not require massive datasets. By fine-tuning GPT-3 on just 80 hand-curated, high-quality examples, researchers significantly reduced toxicity and improved adherence to target ethical norms. Surprisingly, the larger the model, the more effective this "steering" became.

The PALMS Methodology

To address the alignment challenge, the authors introduce PALMS, an iterative seven-step process designed to modify model behavior efficiently:

The 7-step PALMS iterative process
  1. Topic Selection: Identifying sensitive areas (e.g., race, religion, human characteristics).
  2. Desired Behavior Description: Writing explicit "position statements" on how the model should respond (e.g., "The model should support the idea that beauty is subjective").
  3. Prompt Creation: Formulating 80 questions (70 broad, 10 targeted at known weaknesses).
  4. Completion Creation: Hiring a professional writer to craft high-quality, long-form answers (40–340 words) that embody the desired values.
  5. Fine-Tuning: Training the base GPT-3 model on this tiny 80-sample dataset.
  6. Evaluation: Using a mix of automated toxicity scores (Perspective API), human ratings (1–5 scale), and qualitative "co-occurrence" checks (analyzing word associations).
  7. Iteration: Adding more samples to the training set based on where the model failed during evaluation.

Datasets

The core contribution of this work is the Values-Targeted Dataset—a tiny but "gold-standard" set of 80 Q&A pairs designed to reflect a specific ethical lens (in this case, a U.S./International human rights perspective). To ensure the improvements weren't just a result of training on high-quality text, the researchers also used a Control Dataset of 100 snippets from books and Wikipedia. They tested these datasets across the full range of GPT-3 sizes, from 125 million to 175 billion parameters.

Key Insights and Results

The "Bigger is Better" Alignment Law

One of the most striking findings is that PALMS is more effective as the model size increases. While smaller models struggled to generalize the values, the 175B parameter model showed dramatic improvements.

Human evaluation scores showing the values-targeted model outperforming base models, with the gap widening at larger parameter counts

On a 1–5 scale measuring "adherence to values," the 175B PALMS model jumped from a baseline of ~3.8 to nearly 5.0 in several categories. The model didn't just parrot the 80 answers; it learned the sentiment behind them, successfully extrapolating the desired behavior to new, unseen topics.

Toxicity Reduction and Qualitative Shifts

The PALMS-tuned models consistently scored lower on toxicity metrics across all model sizes compared to the base and control models. Furthermore, qualitative shifts were evident in co-occurrence tests. For instance, while the base model often associated "women" with "motherhood" or slurs, the PALMS model shifted these associations toward more neutral or empowering terms like "independent."

Capability Integrity

A common fear in AI safety is that "neutering" a model makes it less capable. This paper shows that fine-tuning on 80 samples has a negligible impact (less than 1% degradation) on the model's performance in standard tasks like trivia or translation.

Limitations and Future Work

While highly effective, the PALMS framework has limitations:

  • Cultural Lens: The authors acknowledge their "values" were chosen through a specific U.S. English and international human rights lens. PALMS is a process, not a universal set of rules; different cultures would need to create their own datasets.
  • Format Constraints: The evaluation focused heavily on a question-answer format. It remains unclear if these behavioral changes hold up as strongly in long-form creative writing or open-ended chat.
  • The "Black Box" of Generalization: While the model generalized well, the exact mechanism making 80 samples so effective is unknown. The authors speculate the model is "unlocking" similar patterns it already saw during its massive pre-training phase.

Future work will explore how this method scales as models continue to grow and how to include more marginalized perspectives in the position-writing phase.

Summary updated

BibTeX
@inproceedings{solaiman2021process,
  author = {Irene Solaiman and Christy Dennison},
  title = {{Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure