Towards Low-Resource Alignment to Diverse Perspectives with Sparse Feedback

Chu Fei Luo, Samuel Dahan, Xiaodan Zhu

Introduction

Standard AI alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), often converge toward a single "average" response. While this ensures safety in general contexts, it frequently erases the nuance of diverse human values and specialized domain knowledge. This "regression to the mean" can be problematic in fields like law, medicine, or finance, where perspectives vary significantly. Furthermore, traditional alignment is computationally expensive and data-intensive.

This paper introduces a low-resource framework designed for pluralistic alignment. By utilizing Sparse Auto-Encoders (SAEs) for model steering and a novel Pluralistic Decoding (PD) technique, the authors demonstrate that Large Language Models (LLMs) can be aligned to multiple niche perspectives using as few as 50 annotated examples.

Methodology

The proposed approach intervenes at the internal activation and generation stages of the model without requiring additional fine-tuning or heavy compute resources.

SAE Model Steering

Sparse Auto-Encoders are employed to decompose the model's internal activations into interpretable features. By comparing activations from a specific perspective (e.g., legal feedback) against a baseline, a "steering vector" is extracted. During inference, this vector is added to the model's layers to nudge its behavior toward the target perspective.

Pluralistic Decoding (PD)

PD is a variation of contrastive decoding that incorporates multiple perspectives simultaneously. It employs entropy-based weighting, which amplifies predictions that diverge from the baseline distribution. This ensures that minority or specialized viewpoints are preserved rather than being overwhelmed by the model's default probabilistic tendencies.

The framework for obtaining steering vectors and applying them during inference via Pluralistic Decoding.

Experimental Evaluation

The framework was evaluated across three specialized datasets designed to test distributional and categorical alignment:

  • GlobalOpinionQA (GQA): Captures cross-national perspectives on social and political topics.
  • Legal Hate Speech (LHS): Focuses on varying definitions of hate speech across legal and social frameworks.
  • Misinformation with Legal Consequences (MisLC): Utilizes granular feedback from legal experts to identify impactful misinformation.

Key Insights and Analysis

Data Efficiency

A significant finding is the "sparse" sweet spot: performance for model steering plateaued at approximately 50 samples. This makes the method highly viable for domain experts who cannot provide the thousands of labels typically required for traditional alignment.

Layer Sensitivity

The effectiveness of steering is highly dependent on the transformer layer where the vector is applied. As shown in the experimental results, the optimal layer varies by task and model architecture, indicating that steering is not a universal "plug-and-play" intervention.

Performance metrics (JS Distance and Binary f1) across different SAE layers for GQA and LHS datasets, illustrating layer sensitivity.

Precision in High-Stakes Tasks

In legal contexts, the authors observed that SAE steering specifically decreased false positives in hate speech and misinformation detection. This suggests the model becomes more aligned with strict, precise definitions rather than broad, generic patterns.

Results Summary

  • Distributional Alignment: On the GQA dataset, the methods reduced the Jensen-Shannon (JS) distance by 0.10 points for Llama 3.1-8b, showing significantly better alignment with target human distributions.
  • Performance Gains: For the MisLC dataset, combining full feedback with PD improved the F1 score on the positive class by 6.3 points for Gemma 2-9b compared to the base model.
  • Baseline Comparison: Model steering consistently outperformed zero-shot and naive few-shot prompting across multiple model sizes.

Limitations and Future Work

While promising, the method has limitations. "Steering too hard" by increasing vector magnitude can lead to nonsensical outputs or increased susceptibility to jailbreaks. Additionally, while these methods improve the reflection of a perspective, they do not necessarily enhance the model's underlying logical reasoning. Future research may explore dynamic retrieval of steering vectors from vector databases, similar to Retrieval-Augmented Generation (RAG) workflows.

Summary updated

BibTeX
@inproceedings{luo2025towards,
  author = {Chu Fei Luo and Samuel Dahan and Xiaodan Zhu},
  title = {{Towards Low-Resource Alignment to Diverse Perspectives with Sparse Feedback}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure