User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs

Sougata Saha, Monojit ChoudhuryACL 2025

Introduction

As Large Language Models (LLMs) are trained on increasingly vast datasets, the AI community faces a critical challenge: Data Contamination. Traditional benchmarks like MMLU or GSM8K are static; once published, they often leak into the training sets of future models. Consequently, high scores may reflect memorization rather than true intelligence or generalization.

This paper, accepted for ACL 2025, proposes a shift in evaluation strategy. Instead of harder logic puzzles, the authors suggest evaluating LLMs on their ability to predict User Behavior. By repurposing personalization tasks—such as movie or music recommendations—into a statistical framework, they create a dynamic, scalable, and contamination-resistant measure of generalization.

Theoretical Framework: The Inversion Point

The authors introduce a novel framework based on Information Theory to quantify generalization. The core idea is to measure the gap between target entropy (the inherent uncertainty of a user's choice) and cross-entropy (the model's prediction error).

In an ideal scenario, a model's error should track perfectly with task difficulty, following the $X=Y$ line on an entropy plot. The authors identify the Inversion Point: the threshold where a model can no longer handle the specificity of a user group, causing its error to spike or flatten.

Conceptual framework illustrating the generalization zone and the inversion point where model performance deviates from ideal behavior.

Methodology

The study utilizes two well-established recommendation datasets:

  • MovieLens (1M): 1 million ratings from ~6,000 users, including demographics like age, gender, and occupation.
  • Last.fm: Music listening habits of ~1,000 users, including country and age data.

Models were tested across three levels of information availability:

  1. Proxies Only: Predicting behavior based on broad demographics.
  2. History Only: Predicting behavior based solely on an individual's unique history.
  3. Proxies and History: Using both demographics and past actions.

Key Results and Insights

The authors evaluated GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct against a random baseline.

Empirical results comparing LLM performance across three scenarios: Proxies and History, Only History, and Only Proxies.
  • GPT-4o Superiority: GPT-4o consistently stayed closest to the ideal $X=Y$ line and exhibited the lowest inversion point, demonstrating a superior ability to handle specific user profiles.
  • The Generalization Gap: As target entropy decreases (requiring more specific, individual-level prediction), LLMs struggle significantly. They excel at "average" behavior but often fail at the "long tail" of individual uniqueness.
  • Contextual Synergy: Models performed best when given both history and demographics. This suggests LLMs leverage "cultural proxies" to narrow the search space for behavior prediction.
  • Domain Difficulty: Music prediction proved harder than movie prediction across all models, likely due to the higher subjectivity and weaker correlation with broad demographic proxies.

Conclusion and Future Work

User behavior prediction offers a robust, low-cost alternative to traditional benchmarks. The experiments conducted cost approximately $192, a fraction of the resources required for high-quality reasoning benchmarks.

Future research will expand this framework to "Agentic" models and explore more complex psychological features, such as personality traits, to provide even more nuanced proxies for human behavior. While currently limited by domain and language (English-only), this approach marks a significant step toward measuring true model generalization in an era of massive data leakage.

Summary updated

BibTeX
@inproceedings{saha2025user,
  author = {Sougata Saha and Monojit Choudhury},
  title = {{User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure