Towards Low-Resource Alignment to Diverse Perspectives with Sparse Feedback

Chu Fei Luo, Samuel Dahan, Xiaodan Zhu

TL;DR

This paper tackles the challenge of aligning large language models (LLMs) to diverse human values, especially when only a small amount of feedback is available. They introduce "pluralistic decoding" and "model steering" using Sparse Autoencoders (SAEs) to enable LLMs to better reflect nuanced perspectives without requiring extensive training data.

What Problem Does This Paper Address?

Modern LLMs are increasingly used in high-stakes applications, making it crucial that they align with human values. However, current alignment methods often assume a single "optimal" answer, leading to generic responses that fail to capture diverse perspectives. This is particularly problematic in fields like law or medicine, where nuanced, context-dependent values are paramount. The authors highlight that traditional machine learning paradigms struggle with "extreme diverging preferences," resulting in reward models that favor universally bland outputs. Furthermore, obtaining extensive, high-quality feedback for alignment is resource-intensive (both in terms of compute and human annotation). This paper aims to address this gap by developing methods for low-resource alignment, enabling LLMs to adapt to diverse viewpoints with minimal expert input.

Methods

The paper proposes two main techniques:

  • Pluralistic Decoding (PD): This method adapts contrastive decoding to combine multiple perspectives dynamically during the text generation process. It aims to strengthen predictions that deviate from the baseline distribution, thereby amplifying minority preferences. PD works by weighting contrastive logits based on the entropy of their probabilistic distribution, encouraging higher weighting for distributions with more uncertainty or plausible token predictions.
  • Model Steering with Sparse Autoencoders (SAEs): This approach leverages SAEs to intervene in the LLM's internal representations. SAEs learn to encode intermediate layer representations into a higher-dimensional space, optimizing for reconstruction while enforcing sparsity. By extracting "steering vectors" (the difference in SAE representations with and without specific feedback), the authors can modify the LLM's behavior at inference time without retraining. This allows for flexible alteration of model behavior, reducing the need for large training datasets and enabling quicker adaptation to novel tasks.

Datasets

The authors evaluated their methods on the following datasets:

  • GlobalOpinionQA (GQA): Used to assess distributional alignment.
  • Legal Hate Speech (LHS) and Misinformation with Legal Consequences (MisLC): These datasets are from the legal domain and are used to evaluate performance on specialized tasks like hate speech and misinformation detection, featuring fine-grained annotations based on legal definitions.

Key Insights and Ideas

  • Sparse feedback is surprisingly effective: The paper demonstrates that model steering with SAEs can achieve significant alignment improvements with as few as 50 annotated samples, challenging the notion that extensive data is always necessary.
  • Model steering offers a flexible alternative: Unlike methods that require retraining or modifying prompts, SAE-based steering allows for direct manipulation of model behavior at inference time, making it adaptable to new tasks and perspectives without costly fine-tuning.
  • Pluralistic decoding enhances diverse perspectives: PD effectively combines multiple viewpoints, leading to better distributional alignment and a stronger reflection of minority preferences, especially in datasets with dense feedback.
  • SAE steering can reduce false positives in high-stakes domains: For legal tasks like hate speech and misinformation detection, SAE steering reduced false positives without significantly sacrificing accuracy on the positive class, indicating improved precision in critical applications.
  • Intervening too far from learned space can be detrimental: The authors note that combining model steering and pluralistic decoding in an end-to-end pipeline did not yield improvements, theorizing that interventions in intermediate layers might move the model too far from its learned activation space, similar to how PPO uses KL divergence penalties to prevent drastic deviations.

Key Results

  • Model steering with SAEs provided consistent improvements over zero-shot and few-shot baselines with only 50 calibration samples.
  • The proposed methods decreased false positives in hate speech detection and misinformation detection tasks.
  • Pluralistic decoding improved distributional alignment to human values in the GlobalOpinionQA dataset.
  • For Gemma2-9b on MisLC, full feedback with PD enhanced performance on the positive class by 6.3 F1 score.
  • SAE model steering decreased the number of false positives on legal datasets for Llama3.1-8b without sacrificing accuracy on the positive class.

Limitations and Future Work

  • Noise in feedback: The current work does not evaluate performance in the presence of noisy feedback, which could affect the clarity of steering vectors.
  • SAE scalability and availability: SAEs are experimental, and their scalability to larger models is a concern. The method assumes pre-trained SAEs are available; otherwise, training them from scratch adds significant cost.
  • Potential for jailbreaks: Intervening in intermediate layers with SAE steering might make LLMs more susceptible to jailbreaks or other vulnerabilities if the intervention strays too far from the learned activation space.
  • Interaction with other modifications: SAE steering does not interact well with other model modifications like alignment tuning or pluralistic decoding when combined.
  • Future work: The authors suggest further research into the interaction of SAE steering with other alignment techniques and exploring its application in more diverse settings. They also highlight the potential for using steering vectors in vector databases for task-specific adaptation, similar to system prompts or LoRA adapters.

Summary updated

BibTeX
@inproceedings{luo2025towards,
  author = {Chu Fei Luo and Samuel Dahan and Xiaodan Zhu},
  title = {{Towards Low-Resource Alignment to Diverse Perspectives with Sparse Feedback}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure