Abstract
As Large Language Models (LLMs) transition from conversational interfaces to decision-making agents within Intelligent User Interfaces (IUIs), ensuring their alignment with complex human preferences is critical. A key challenge is the "indifference" problem: humans often find multiple options equally desirable, yet standard ranking methods often force arbitrary choices. This paper introduces design principles for implementing "rational choice functions" using LLMs and proposes new metrics—Strict Preference Overlap (SPO) and Kendall Distance with Penalty ($K^{(p)}$)—to measure alignment accuracy.
Introduction: Beyond Safety in LLM Alignment
Current research in LLM alignment primarily targets safety, focusing on mitigating bias, toxicity, and hallucinations. However, for practical applications like virtual automotive assistants, alignment must also encompass a model's ability to reflect specific user preferences.
Existing ranking methods, such as Pointwise or Listwise ranking, often exhibit "irrational" behavior. They can be sensitive to presentation order or fail to account for indifference. For instance, if a user views a raincoat and an umbrella as equally useful, an aligned agent should recognize both as valid "best" choices. Forcing a strict ranking in such scenarios leads to arbitrary and potentially untrustworthy behavior.
Methodology: Designing Rational Choice Functions
The authors propose three design principles to extract rational preferences from LLMs using pairwise prompting (comparing two items at a time):
- Pairwise-Score: Based on the "PRP-Allpair" method, the model compares every possible pair in a set. Items receive 1 point for a win and 0.5 points for a tie, allowing for a ranking that explicitly accommodates ties.
- Pairwise-SCC (Strongly Connected Components): This graph-based approach constructs a directed graph where nodes represent items and edges represent preferences. By identifying Strongly Connected Components, the system groups items that are mutually preferred or where the model is inconsistent, enforcing a mathematically rational structure.
- Pairwise-Test: A baseline method that queries the LLM for preferences without enforcing transitivity or structural rationality.
To evaluate these methods, the researchers utilized a custom automotive dataset featuring 23 scenarios (e.g., driving to a park in the rain) and 40 personal items. Performance was measured using Strict Preference Overlap (SPO) for partial alignment and Kendall Distance with Penalty ($K^{(p)}$) for full alignment, which penalizes failures to recognize indifference.
Key Results and Insights
The study evaluated GPT-4, Gemini Pro, and Llama 2 (70B). The results highlighted significant differences in model capability and method effectiveness.
The Superiority of Pairwise Methods
GPT-4 and Gemini Pro emerged as leaders. As shown in Table 1, GPT-4 achieved a peak SPO of 0.85 using the Pairwise-Score method. In contrast, "Listwise" prompting—asking the LLM to rank an entire list at once—proved highly unreliable. Models frequently hallucinated items or failed to follow output schemas, with Llama 2 failing to produce a single valid response in the Listwise test.
Rationality via Graph Theory
While Pairwise-Score is effective for identifying top items, Pairwise-SCC proved superior for full alignment. By resolving inconsistencies in LLM outputs through graph theory, SCC transforms noisy pairwise comparisons into a consistent logic.
As illustrated in Table 2, Pairwise-SCC consistently achieved lower (better) Kendall Distance scores compared to Pairwise-Score, indicating a more accurate reflection of the user's overall preference structure, including ties.
Discussion and Future Work
Despite the success of pairwise methods, several challenges remain:
- Computational Cost: Pairwise comparisons require $O(n^2)$ queries. For 40 items, this translates to 800-1,600 queries, which is currently too slow for real-time IUIs.
- Prompt Sensitivity: Model performance varied significantly based on the specific prompt template used, highlighting the fragility of current alignment techniques.
- Scalability: Future research should focus on developing principles that maintain rationality while scaling more efficiently than $O(n^2)$.
- Weighted Preferences: Incorporating the "stakes" of a decision (e.g., prioritizing alignment on essential items like car keys over optional items like a raincoat) remains an open area for exploration.
By moving toward rational choice functions, developers can build LLM agents that more accurately reflect the nuanced reality of human decision-making.
Summary updated