Learning Compact Representations of LLM Abilities via Item Response Theory

Jianhao Chen, Chenxu Wang, Gengrui Zhang, Peng Ye, Lei Bai, Wei Hu, Yuzhong Qu, Shuyue HuSubmission to ICLR 2026 · Rejected

TL;DR

With tens of thousands of open-source language models available, determining which model can answer a specific query is both critical and computationally challenging. This paper introduces IrtNet, a framework that adapts psychometric Item Response Theory (IRT) to learn compact, interpretable representations of both LLM capabilities and query properties. By combining model skill embeddings with query difficulty and discrimination parameters via a dense Mixture-of-Experts network, IrtNet sets new state-of-the-art results in LLM routing and predicts benchmark performance using a fraction of the usual training data.

What Problem Does This Paper Address?

As open-source LLMs proliferate (over 97,000 text-generation models with $\ge$6B parameters on Hugging Face), system designers increasingly need to:

  1. Route queries dynamically: Dispatch a user prompt to the smallest or most suitable model in a pool to cut inference costs while matching or beating proprietary systems.
  2. Predict benchmark scores efficiently: Evaluate models without running thousands of expensive forward passes over massive, redundant benchmarks.

Existing routing and profiling methods often treat models as black boxes or rely on complex heuristics that lack interpretability and require large amounts of evaluation data to generalize.

Methods

The authors cast the problem of predicting whether a model $m$ correctly answers a query $q$ ($y \in \{0, 1\}$) through the lens of classical two-parameter Item Response Theory (2PL-IRT):

$$P(y=1 \mid m, q) = \sigma\left(\boldsymbol{\alpha}_q^\top \boldsymbol{\theta}_m - \beta_q\right)$$

  • Model Latent Ability ($\boldsymbol{\theta}_m \in \mathbb{R}^d$): A compact $d$-dimensional embedding ($d=232$) representing the model’s multi-dimensional skill profile.
  • Query Discrimination ($\boldsymbol{\alpha}_q \in \mathbb{R}^d$): A vector reflecting which latent skills the query tests and how strongly proficiency in those skills influences success.
  • Query Difficulty ($\beta_q \in \mathbb{R}$): A scalar reflecting the baseline hardness of the item.
Query Text ──► Sentence Transformer ──► Dense MoE (40 experts) ──┬──► Linear ──► Discrimination (α_q)
                                                                 └──► Linear ──► Difficulty (β_q)
                                                                                       │
Model ID ────────────────────────────────► Latent Ability (θ_m) ───────────────────────┴──► Sigmoid ──► P(y=1)

To predict $\boldsymbol{\alpha}_q$ and $\beta_q$ directly from query text, the authors introduce IrtNet:

  1. The query text is encoded into a dense vector (via all-mpnet-base-v2).
  2. A Dense Mixture-of-Experts (MoE) layer with 40 always-active experts refines general semantic embeddings into task-specific discriminative representations, using auxiliary-loss-free load balancing to stabilize training.
  3. Two linear heads predict $\boldsymbol{\alpha}_q$ and $\beta_q$.
  4. The network is trained end-to-end with binary cross-entropy loss against historical response matrices of models answering queries.

Datasets

The authors evaluated IrtNet on two distinct benchmark setups:

  • EmbedLLM Dataset (Primary): 35,673 queries across 10 benchmarks (ASDiv, GPQA, GSM8K, MathQA, LogiQA, MedMCQA, MMLU, SocialIQA, PIQA, TruthfulQA) evaluated on 112 open-source LLMs (e.g., LLaMA, Qwen, Mistral, CodeLlama families). Split: 29,673 train, 3,000 validation, and 3,000 test queries.
  • Avengers Dataset (Generalization): 8,550 queries across 10 benchmarks (BBH, EmoryNLP, FinQA, HumanEval, K&K, LiveCodeBench, Math500, MBPP, MELD, MMLU-Pro) evaluated on 20 modern 7–9B LLMs (including DeepSeek-R1-Distill, Qwen2.5/3, Gemma-2, Llama-3.1).
  • Easy2Hard-Bench: Used as an external reference to validate the learned query difficulty parameters on GSM8K.

Key Insights and Ideas

  • Psychometrics translates cleanly to AI evaluation: Formulating query-model interactions as latent abilities aligned with item difficulty and discrimination significantly outperforms purely data-driven contrastive or encoder-decoder representations.
  • Learned parameters are remarkably interpretable without supervision:
  • Difficulty ($\beta_q$): Exhibited a $-0.9721$ Pearson correlation with average ground-truth model accuracy across benchmarks (and $-0.8993$ on GSM8K). Higher $\beta_q$ strictly corresponds to harder questions.
  • Discrimination ($\boldsymbol{\alpha}_q$): Visualized via t-SNE, the learned $\boldsymbol{\alpha}_q$ vectors formed clear, distinct clusters by task domain (math, logic, medical, commonsense) despite never receiving domain or benchmark labels during training.
  • Model Ability ($\boldsymbol{\theta}_m$): Embeddings clustered naturally by model family (LLaMA vs. Qwen) and by domain specialization (math, code, medical), confirming that $\boldsymbol{\theta}_m$ captures meaningful functional characteristics.
  • Dense MoE acts as a semantic filter: Ablations showed that replacing the Dense MoE with a standard MLP or sparse MoE degraded performance, as all-active experts provide a rich, multi-perspective breakdown of prompt characteristics without routing instability.

Key Results

  • Model Routing:
  • On the primary 112-model benchmark, IrtNet achieved 67.4% micro-accuracy (and 62.0% macro-accuracy), significantly outperforming EmbedLLM (60.2%), Avengers-Pro (62.1%), MODEL-SAT (56.7%), and RouterDC (54.9%).
  • On the 20 newer LLMs across 10 additional benchmarks, IrtNet led with 72.1% micro-accuracy.
  • Robustness: In a noisy-label setting, IrtNet maintained its performance (67.5% micro).
  • Data-Efficient Benchmark Prediction:
  • With only 1,000 training queries (<4% of the dataset), IrtNet reached 69.9% prediction accuracy, nearly matching the state of the art (EmbedLLM's 70.6%) trained on the entire 29K dataset. Full training pushed accuracy to 72.2%.
  • On leave-one-benchmark-out (OOD) accuracy estimation, IrtNet achieved an overall RMSE of 0.19, outperforming EmbedLLM (0.21), Observational Scaling Laws (0.32), and Random Sampling (0.29).
  • Efficiency: Training took 15 minutes on a single NVIDIA A100 GPU (2GB VRAM), with an inference latency of 6.7 ms per query across 112 candidate models.

Limitations and Future Work

  • Binary response assumption: The formulation relies on binary correctness ($y \in \{0, 1\}$). Adapting to open-ended generative evaluations where answers vary continuously on quality or partial credit would require polytomous or graded IRT models.
  • Transductive model embeddings: Learning $\boldsymbol{\theta}_m$ currently requires observed response data for that model ID; evaluating a brand-new model zero-shot without running any prior queries is not supported without an additional meta-model.
  • Sensitivity to query encoders: Performance declined when switching from all-mpnet-base-v2 to a weaker BERT encoder, showing that quality text embeddings remain a prerequisite for accurate discrimination modeling.

Summary updated

Abstract

Recent years have witnessed a surge in the number of large language models (LLMs), yet efficiently managing and utilizing these vast resources remains a significant challenge. In this work, we explore how to learn compact representations of LLM abilities that can facilitate downstream tasks, such as model routing and performance prediction on new benchmarks. We frame this problem as estimating the probability that a given model will correctly answer a specific query. Inspired by the item response theory (IRT) in psychometrics, we model this probability as a function of three key factors: (i) the model’s multi-skill ability vector $\theta$, (ii) the query’s discrimination vector $\alpha$ that separates models of differing skills, and (iii) the query’s difficulty scalar $\beta$. To learn these parameters jointly, we introduce a Mixture-of-Experts (MoE) network that couples model- and query-level embeddings. Extensive experiments demonstrate that our approach leads to state-of-the-art performance in both model routing and benchmark accuracy prediction. Moreover, analysis validates that the learned parameters encode meaningful, interpretable information about model capabilities and query characteristics.
BibTeX
@misc{chen2025learning,
  author = {Jianhao Chen and Chenxu Wang and Gengrui Zhang and Peng Ye and LEI BAI and Wei Hu and Yuzhong Qu and Shuyue Hu},
  title = {{Learning Compact Representations of LLM Abilities via Item Response Theory}},
  year = {2025},
  eprint = {2510.00844},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.00844},
}
Figure