Knowing Which, Not How Much: Language Models Order Party Positions They Cannot Place

Yousef Attia, Jarosław A. ChudziakarXiv 2026 · arXiv

This research paper, titled "Knowing Which, Not How Much: Language Models Order Party Positions They Cannot Place," explores a fascinating quirk in how Large Language Models (LLMs) process social and political information. The authors demonstrate that while LLMs are surprisingly good at ranking entities (e.g., "Which party cares more about the environment?"), they struggle significantly when asked to place those same entities on an absolute scale.

TL;DR

LLMs can reliably rank political parties by their policy priorities but fail to accurately place them on an absolute scale, often performing worse than simple constant-answer baselines. This "ordering without magnitude" suggests that current benchmarks, which typically use single-item questions, may be measuring the wrong type of competence.

What Problem Does This Paper Address?

As LLMs become a primary source for voters and journalists seeking to understand political platforms, we need to know how they represent "gradable" social facts—facts that have both an order (Party A cares more than Party B) and a magnitude (Party A cares "a lot").

Most existing benchmarks test one entity at a time (absolute judgment). However, human psychology has long shown that we are much better at comparing two things than assigning an absolute score to one. The authors argue that because training text (news, social media) contains far more comparisons ("The Greens are more radical than Labor") than quantified measurements, LLMs might learn the relative hierarchy of social actors without ever anchoring them to a specific scale.

Methods

The researchers designed a controlled experiment to test seven production-grade LLMs (including DeepSeek-V3, Gemini-Flash, and Claude-Sonnet) on their knowledge of party issue salience.

The experimental setup, illustrated in Figure 1, split the task into two "lanes":

  1. Absolute Condition: The model is asked about one party's emphasis on an issue using a four-level scale (e.g., "core priority" to "not really addressed").
  2. Comparative Condition: The model is given four parties and asked to identify which has the most and least emphasis on a specific issue.
Figure 1: The life cycle of the data. Coded CMP percentages are quantised into ordinal buckets, giving each benchmark cell a gold level and a tie-aware gold ranking; the same cell is then put to a model two ways. The two lanes are separate API calls with no shared history, and both ask a four-alternative question: four emphasis levels in A, four named parties in B. Each lane starts from a structured prompt; the model answers in free prose that ends with one machine-readable tagged line (Section 4.2), and both lanes are parsed identically. Dashed paths are the only ones that read the gold column.

To distinguish between a lack of knowledge and a failure to understand the scale, they also ran a Grounding Intervention, where the correct facts were prepended to the prompt to see if the model could then use the scale correctly.

Datasets

The study uses data from the Manifesto Project (CMP/MARPOR), a gold-standard dataset in political science where trained human coders quantify the percentage of election manifestos dedicated to specific policy categories.

  • Scope: 8 major parties across Japan (2021) and South Korea (2020).
  • Issues: 10 policy categories (e.g., Military, Environmental Protection, Welfare State Expansion).
  • Novelty: This benchmark is unique because it tests parametric knowledge (what the model "knows" from training) against an external, professionally coded key without providing the source text in the prompt.

Key Insights and Ideas

  • The Dissociation: There is a massive gap between a model's ability to rank and its ability to place. As shown in Table 2, models beat the baseline for ordering by a factor of 2.62x, but fall below the baseline for exact placement.
  • The "Hidden" Order: Interestingly, even when a model assigns the wrong absolute levels to parties, the relative order implied by those wrong levels is often correct. The hierarchy exists in the model's weights, but it isn't "anchored" to the scale.
  • Systematic Upward Bias: When models fail at placement, they don't just guess randomly; they systematically overestimate how much parties care about issues, frequently labeling minor mentions as "moderate" or "core" priorities.
  • Knowledge vs. Mapping: The grounding intervention proved that models can use the four-level scale perfectly if the facts are provided. This suggests the failure isn't a reasoning or "comprehension" issue, but a specific "knowledge deficit" regarding absolute magnitudes.

Key Results

The quantitative results highlight a stark contrast in model performance:

  • Ordering Accuracy: Models correctly identified the highest-emphasis party 58.1% of the time (against a 22.2% baseline).
  • Placement Accuracy: Models achieved only 22.4% accuracy on exact placement, which is worse than the 37.9% accuracy one would get by simply answering "not a stated priority" to every question.
Table 2: The dissociation between ordering and placement tasks.

As seen in Table 1, while models like DeepSeek-V3 and Gemini-Flash lead in accuracy, they still struggle with absolute placement. Claude-Sonnet was notably "cautious," declining to answer (No Knowledge) more than half the time.

Table 1: Absolute-placement performance across different models.

Limitations and Future Work

  • Scale Granularity: The study used a 4-bucket scale. Future work could test if finer scales (e.g., 1-10) reveal more "magnitude" knowledge or just more noise.
  • Geographic Scope: The findings are based on Japan and South Korea. While chosen for their high-quality data, the results might differ for Western democracies where training data is more abundant.
  • Language: All prompts were in English, despite the subject matter being Japanese and Korean politics. While LLMs are multilingual, cross-language retrieval gaps might influence the results.

The authors conclude that for any "gradable" property—be it political stance, risk level, or severity—benchmarks must report both comparative and absolute performance. Otherwise, we risk understating what models know while overstating their ability to communicate it accurately.

Summary updated

Abstract

Language models are increasingly asked where real actors stand: what a party's platform prioritises, what an institution has committed to. Such judgments are gradable, carrying both an order and a magnitude. Nearly every benchmark that tests them asks about one entity at a time, then reads that score as the model's knowledge of the domain. Order and magnitude, however, need not be acquired together. Text about social entities carries far more comparison than measurement, so a model may hold a dependable ordering with no anchored scale beneath it. We ask whether a model can place a documented social position on a scale or only order the actors it compares, and what that means for how such knowledge is measured. We test this on party positions in Japan and South Korea, where the Comparative Manifesto Project's coded data supplies a rare external answer key. Asked to order four parties, models beat the strongest fixed-answer baseline by more than a factor of two; asked to place one party on the same scale, they fall below it. The ordering survives inside those failed placements, and supplying the coded values lifts placement to near ceiling. Comparative and absolute questions about the same fact therefore measure different things here, and for party issue salience a benchmark built from one cannot speak for the other.
BibTeX
@misc{attia2026knowing,
  author = {Yousef Attia and Jarosław A. Chudziak},
  title = {{Knowing Which, Not How Much: Language Models Order Party Positions They Cannot Place}},
  year = {2026},
  eprint = {2611.21878},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21878},
}
Figure