TL;DR
Large Language Models (LLMs) struggle to consistently rank preferences, exhibiting biases and illogical reasoning even when presented with clear criteria. This paper formalizes consistency in ordinal preference formation and reveals significant inconsistencies across state-of-the-art LLMs, highlighting a critical gap in their reasoning capabilities.
What Problem Does This Paper Address?
Expressing preferences is fundamental to human decision-making. In the realm of AI, aligning LLMs with human preferences is crucial for tasks like recommendation and content moderation. While LLMs are increasingly used to generate these preferences, their ability to do so consistently and logically remains underexplored. This paper tackles the problem of measuring and understanding the inconsistencies in how LLMs form ordinal preferences, moving beyond simple single-choice selection to more complex ranking scenarios. The authors highlight that existing benchmarks often focus on tasks with a single "correct" answer, whereas preferential ranking deals with subjective choices where logical consistency is paramount.
Methods
The authors introduce a formal framework for measuring consistency in LLM preferential ranking, drawing from order theory. They define five key criteria that a consistent preferential ranking must satisfy:
- Irreflexivity: An option cannot be preferred over itself.
- Asymmetry: If A is preferred to B, then B cannot be preferred to A.
- Transitivity: If A is preferred to B, and B is preferred to C, then A must be preferred to C.
- Independence from Irrelevant Alternatives (IIA): Introducing or removing an option should not change the relative preference order of the original options.
- Reversibility: The ranking should be logically consistent when reversed (e.g., if A > B > C, then C < B < A).
To test these criteria, they devise specific measurement metrics, including constructing binary comparison matrices to assess asymmetry and transitivity, and using similarity scores to evaluate IIA and reversibility.
Datasets
The primary dataset used is a subset of the MMLU (Massive Multitask Language Understanding) benchmark. The authors selected the first 20 cases from each of the 57 subject areas, resulting in a total of 1,140 cases. While MMLU is originally designed for Multiple-Choice Question Answering (MCQA), the authors adapted it for preferential ranking by leveraging its multiple-choice options as the alternatives to be ranked. They also explored variations in label tokens (Alphabetic, Arabic, Roman) to check for label bias.
Key Insights and Ideas
- LLMs are surprisingly bad at consistent ranking: Despite their advanced capabilities, even the most sophisticated LLMs fail to meet basic logical consistency criteria when asked to rank preferences.
- Positional bias is rampant: LLMs show a strong tendency to favor options based on their position in the prompt or list, violating the asymmetry criterion. This means the order in which options are presented significantly influences the LLM's output.
- Transitivity is fragile: Concatenating pairwise preferences doesn't reliably lead to a coherent overall ranking. LLMs often produce rankings that are contested or even cyclic, meaning A > B, B > C, but C > A.
- Irrelevant alternatives matter: The presence or absence of seemingly unrelated options can drastically alter an LLM's preference ordering for the original set of choices, violating the IIA principle.
- Reversibility is a challenge: LLMs struggle to produce logically equivalent rankings when asked to rank in ascending versus descending order.
Key Results
- Asymmetry: LLMs exhibit significant positional bias, with low scores for asymmetry, indicating that the order of presentation heavily influences their preferences. For instance, the order of options in binary comparisons markedly affects LLM preferences.
- Transitivity: Models show moderate transitivity scores, but subtle differences between upper and lower triangles of comparison matrices suggest positional biases also impact transitivity.
- Independence from Irrelevant Alternatives (IIA): The removal of "gold" (correct) answers significantly alters LLM preferences, while removing non-gold options has a less pronounced impact. This suggests LLMs are sensitive to the perceived "correctness" or quality of options when making rankings.
- Reversibility: LLMs demonstrate suboptimal performance on full match length for reversibility tests, with match rates below 45% for some models. GPT-4o performed significantly better than others in this regard.
Limitations and Future Work
The authors acknowledge that using MMLU, an MCQA benchmark, is not perfectly aligned with preferential ranking, as MCQA typically has predetermined "correct" answers, whereas preferential ranking often involves subjective choices. They suggest future work should focus on developing benchmarks specifically designed for consistency measurement in preferential ranking, moving beyond true/false judgments. Additionally, they call for research into mitigation methods to enhance the consistency of LLMs in ranking tasks.
Summary updated