Abstract
When different groups' values differ, one approach to model alignment is to steer models at inference time towards each group's preferences. However, techniques like in-context learning only consider similarity when drawing few-shot examples and not cross-group differences in values. We propose SPICA, a framework that accounts for group-level differences during in-context example retrieval. SPICA introduces three designs: scenario banks, group-informed retrieval metrics, and in-context alignment prompts. From an evaluation of SPICA on an alignment task collecting inputs from four demographic groups ($n = 544$), our metrics retrieve in-context examples that more closely match observed preferences, with the best prompt configuration using multiple contrastive responses to demonstrate examples. In an end-to-end evaluation ($n = 120$), we observe that SPICA is higher rated than similarity-based retrieval, with groups seeing up to a +0.16 point improvement on a 5 point scale. Additionally, gains from SPICA were more uniform, with all groups benefiting from alignment rather than only some. Finally, we find that while a group-agnostic approach can align to aggregated values, it is not most suited for divergent groups.
Introduction
Most AI alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), aim for a "one-size-fits-all" set of universal values. However, society is inherently pluralistic; different demographic groups often hold irreconcilable disagreements on how an AI should handle sensitive topics like religion, politics, or personal advice.
In-Context Alignment (ICA) has emerged as a popular alternative, steering models using few-shot examples at inference time. However, current ICA systems typically retrieve these examples based solely on semantic similarity—finding examples that "look like" the user's prompt. This approach ignores critical "second-order" group norms: how much a specific community agrees on a value (crystallization) and how strongly they feel about it (intensity). Without accounting for these norms, ICA often defaults to majority preferences, leaving minority or divergent groups with poorly aligned model outputs.
The SPICA Framework
To address these limitations, we introduce SPICA, a framework designed to account for group-level differences during in-context example retrieval. SPICA consists of three core components:
- Scenario Banks: A structured collection of "scenarios" containing a prompt, a set of potential response strategies (e.g., "Refusal," "Opinionated," "Multi-perspective"), and disaggregated human ratings for those strategies.
- Group-Informed Retrieval Metrics: Instead of relying solely on cosine similarity, SPICA incorporates two new metrics to find the most "salient" examples for a specific group:
- Stability ($g_{stability}$): Measures "crystallization"—how consistently members of a group agree on a specific behavior (visualized as "Agreement" in the pipeline).
- Contrast ($g_{contrast}$): Measures "intensity"—whether a group is highly opinionated or indifferent about a specific scenario.
- Pluralistic Prompting Setups: We explored various ways to present these retrieved scenarios to the LLM, including Positive Instructions (P-I) and Contrastive Responses (C-R), which show the model both preferred and non-preferred behaviors to define acceptable boundaries.
Experimental Setup
We evaluated SPICA using challenging alignment prompts from the PRISM dataset. We constructed a scenario bank by generating responses across five behavior classes: Refusal, High-level factual, Specific factual, Multi-perspective, and Opinionated.
To provide ground-truth ratings, we recruited 544 participants from Prolific, categorized into four demographic groups: Republicans, Democrats, Religious, and Non-religious. These participants provided the ratings used to calculate group norms and evaluate the system's ability to match observed preferences.
Key Results and Discussion
Our evaluation revealed that semantic similarity is insufficient for effective pluralistic alignment. Traditional retrieval often fails to illustrate the specific values a user's group cares about most. In contrast, SPICA's group-informed metrics retrieve examples that more closely match observed preferences.
One of the most effective prompting strategies was Contrastive Responses (C-R). By showing the model what not to do alongside what to do, the model gains a clearer "decision boundary" for group-specific alignment. As shown in our results, this strategy was particularly effective for larger models like GPT-4o.
Equitable Gains and Model Sensitivity
Crucially, SPICA makes alignment more equitable. Under baseline "universal" alignment, groups like "Republican/Non-religious" actually saw negative alignment. SPICA reversed this trend, providing the largest gains (+0.16 points) to the groups previously most disadvantaged. Furthermore, SPICA produced outputs that were uniformly preferred by all groups relative to their own norms, whereas baseline ICA resulted in disparate outcomes.
We also observed significant model sensitivity. The framework performed better on GPT-4o than on Llama-3-8B, suggesting that the ability to follow complex, contrastive in-context alignment instructions scales with model size and capability.
Conclusion and Future Work
SPICA demonstrates that moving beyond simple semantic similarity to incorporate group-informed metrics can significantly improve AI alignment for diverse populations. While our study focused on four US-centric demographic slices, future work should explore scaling this to thousands of niche communities globally. Additionally, while building scenario banks is currently resource-intensive, future iterations could leverage LLM-backed agents to dynamically elicit group values. Finally, pluralistic alignment must be carefully balanced with external safeguards and a search for common ground to avoid the risk of creating self-reinforcing echo chambers.
Summary updated