TL;DR
P-ContextBench is a new, highly realistic benchmark designed to evaluate how well Large Language Models (LLMs) understand and act upon fragmented personal context. It moves beyond simple fact retrieval to test complex capabilities like temporal state tracking and constraint-aware agent behavior, revealing a significant "reliability gap" in even the most advanced current models.
What Problem Does This Paper Address?
As LLMs transition into personalized agents—handling our schedules, shopping, and daily decisions—they must navigate "personal context." This context is notoriously difficult to benchmark because it is:
- Private and Fragmented: It lives in scattered app logs (messages, calendars) rather than neat profiles.
- Implicit: Preferences and constraints are often revealed through behavior, not direct statements.
- Dynamic: Our routines, goals, and health statuses change over time.
Existing benchmarks often rely on static, explicit profiles or focus purely on fact retrieval. They fail to capture the nuance of an agent needing to infer a constraint (e.g., a user's diabetes) from fragmented logs and then decide whether to comply with a risky request (e.g., "Should I skip my pill?"). P-ContextBench fills this gap by providing a verifiable, scalable framework for testing these "agentic" reasoning capabilities.
Methods
The authors developed a sophisticated six-stage pipeline to generate synthetic but realistic digital humans and their digital traces:
- Population-Structured Seeds: Personas start from demographic seeds reflecting U.S. Census distributions.
- Life-Story Grounding: Each persona is enriched via a simulated 100+ question interview covering history, values, and health.
- Social Circle & App-Log Synthesis: The pipeline generates messenger and calendar logs. Crucially, facts are implicitly encoded across multiple fragments. For example, a medical condition might be spread across a pharmacy pickup notification and a doctor's appointment reminder.
- Multi-Stage Verification: Every stage uses an LLM auditor to ensure the personas are plausible, the facts are recoverable from the logs (but not "leaked" via explicit labels), and the test cases are grounded.
The benchmark evaluates five task types (T1–T5), ranging from simple fact recovery (T1) to Constraint-Aware Agent Behavior (T5), where the model must decide how to respond to a user request that might conflict with their personal context.
Datasets
The authors instantiated P-ContextBench with:
- 100 Personas based on NVIDIA Nemotron-Personas-USA.
- 10,088 Hidden Facts distilled from life-story interviews.
- 18,261 App-Trace Fragments (Messenger and Calendar).
- 4,810 Verified Test Cases.
The framework is designed to be scalable, allowing researchers to generate new personas for different demographic groups or app formats.
Key Insights and Ideas
- The Reliability Gap: The authors introduce the $Pass^3$ metric, which counts a test case as solved only if the model gets it right in three independent runs. While models might show decent "overall" accuracy, their $Pass^3$ scores are abysmally low (often <10% for complex tasks), suggesting that LLM performance on personal context is highly inconsistent.
- Synthesis is Harder than Retrieval: Models are relatively good at finding a single fact (T1/T2) but struggle significantly when they must synthesize multiple facts to infer a higher-level pattern (T4), such as a user's professional identity or underlying motives.
- Temporal Blindness: Models frequently fail at Temporal State Tracking (T3). They often detect that a change has occurred but fail to reconstruct the current state accurately, often mixing it up with "stale" information from earlier in the logs.
- Constraint Recognition Failures: In agent-oriented tasks (T5), the primary failure mode (37.5% of errors) is failing to recognize a constraint at all. This leads to "blind compliance" with potentially unsafe user requests.
Key Results
The evaluation compared cost-effective SOTA models (like GPT-4o-mini and Gemini-3-Flash) against a high-performance reference (Claude Sonnet 4.6).
- Performance Drop-off: Accuracy drops sharply as tasks move from fact retrieval (T1/T2) to reasoning and behavior (T3–T5).
- Consistency Crisis: For GPT-4o-mini, while overall accuracy on T3–T5 was around 50%, the $Pass^3$ score was below 4%. Even the high-performing Claude Sonnet 4.6 achieved only ~34.6% $Pass^3$ on average.
- Failure Analysis: As shown in the failure category table, T3 errors are dominated by omitting current-state facts (81%), while T5 errors are led by constraint recognition failures.
Limitations and Future Work
- App Diversity: The current instantiation is limited to messenger and calendar logs. The authors plan to expand to richer multimodal traces (e.g., photos, location history).
- System-Level Evaluation: The current study focuses on standalone LLMs. Future work will evaluate complete agent systems, including their long-term memory architectures, RAG pipelines, and specific safety safeguards.
- Demographic Specificity: While the current seeds reflect the U.S. population, the authors aim to explore domain-specific subpopulations where personal context might follow different patterns.
Summary updated