Introduction
As Large Language Models (LLMs) are trained on increasingly vast datasets, the AI community faces a critical challenge: Data Contamination. Traditional benchmarks like MMLU or GSM8K are static; once published, they often leak into the training sets of future models. Consequently, high scores may reflect memorization rather than true intelligence or generalization.
This paper, accepted for ACL 2025, proposes a shift in evaluation strategy. Instead of harder logic puzzles, the authors suggest evaluating LLMs on their ability to predict User Behavior. By repurposing personalization tasks—such as movie or music recommendations—into a statistical framework, they create a dynamic, scalable, and contamination-resistant measure of generalization.
Theoretical Framework: The Inversion Point
The authors introduce a novel framework based on Information Theory to quantify generalization. The core idea is to measure the gap between target entropy (the inherent uncertainty of a user's choice) and cross-entropy (the model's prediction error).
In an ideal scenario, a model's error should track perfectly with task difficulty, following the $X=Y$ line on an entropy plot. The authors identify the Inversion Point: the threshold where a model can no longer handle the specificity of a user group, causing its error to spike or flatten.
Methodology
The study utilizes two well-established recommendation datasets:
- MovieLens (1M): 1 million ratings from ~6,000 users, including demographics like age, gender, and occupation.
- Last.fm: Music listening habits of ~1,000 users, including country and age data.
Models were tested across three levels of information availability:
- Proxies Only: Predicting behavior based on broad demographics.
- History Only: Predicting behavior based solely on an individual's unique history.
- Proxies and History: Using both demographics and past actions.
Key Results and Insights
The authors evaluated GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct against a random baseline.
- GPT-4o Superiority: GPT-4o consistently stayed closest to the ideal $X=Y$ line and exhibited the lowest inversion point, demonstrating a superior ability to handle specific user profiles.
- The Generalization Gap: As target entropy decreases (requiring more specific, individual-level prediction), LLMs struggle significantly. They excel at "average" behavior but often fail at the "long tail" of individual uniqueness.
- Contextual Synergy: Models performed best when given both history and demographics. This suggests LLMs leverage "cultural proxies" to narrow the search space for behavior prediction.
- Domain Difficulty: Music prediction proved harder than movie prediction across all models, likely due to the higher subjectivity and weaker correlation with broad demographic proxies.
Conclusion and Future Work
User behavior prediction offers a robust, low-cost alternative to traditional benchmarks. The experiments conducted cost approximately $192, a fraction of the resources required for high-quality reasoning benchmarks.
Future research will expand this framework to "Agentic" models and explore more complex psychological features, such as personality traits, to provide even more nuanced proxies for human behavior. While currently limited by domain and language (English-only), this approach marks a significant step toward measuring true model generalization in an era of massive data leakage.
Summary updated