P-ContextBench: From Personal Fact Recall to Personalized Agent Behavior

Grant Nasir Brown, Abdul Basir Bari, Anmol Tadikonda, Isabel SongarXiv 2026 · arXiv

TL;DR

P-ContextBench is a new, highly realistic benchmark designed to evaluate how well Large Language Models (LLMs) understand and act upon fragmented personal context. It moves beyond simple fact retrieval to test complex capabilities like temporal state tracking and constraint-aware agent behavior, revealing a significant "reliability gap" in even the most advanced current models.

What Problem Does This Paper Address?

As LLMs transition into personalized agents—handling our schedules, shopping, and daily decisions—they must navigate "personal context." This context is notoriously difficult to benchmark because it is:

  1. Private and Fragmented: It lives in scattered app logs (messages, calendars) rather than neat profiles.
  2. Implicit: Preferences and constraints are often revealed through behavior, not direct statements.
  3. Dynamic: Our routines, goals, and health statuses change over time.

Existing benchmarks often rely on static, explicit profiles or focus purely on fact retrieval. They fail to capture the nuance of an agent needing to infer a constraint (e.g., a user's diabetes) from fragmented logs and then decide whether to comply with a risky request (e.g., "Should I skip my pill?"). P-ContextBench fills this gap by providing a verifiable, scalable framework for testing these "agentic" reasoning capabilities.

Methods

The authors developed a sophisticated six-stage pipeline to generate synthetic but realistic digital humans and their digital traces:

  1. Population-Structured Seeds: Personas start from demographic seeds reflecting U.S. Census distributions.
  2. Life-Story Grounding: Each persona is enriched via a simulated 100+ question interview covering history, values, and health.
  3. Social Circle & App-Log Synthesis: The pipeline generates messenger and calendar logs. Crucially, facts are implicitly encoded across multiple fragments. For example, a medical condition might be spread across a pharmacy pickup notification and a doctor's appointment reminder.
  4. Multi-Stage Verification: Every stage uses an LLM auditor to ensure the personas are plausible, the facts are recoverable from the logs (but not "leaked" via explicit labels), and the test cases are grounded.

The benchmark evaluates five task types (T1–T5), ranging from simple fact recovery (T1) to Constraint-Aware Agent Behavior (T5), where the model must decide how to respond to a user request that might conflict with their personal context.

Figure 1: Overall pipeline of P-ContextBench. Ex, Mx, and Fx denote calendar, messenger, and filler fragments.

Datasets

The authors instantiated P-ContextBench with:

  • 100 Personas based on NVIDIA Nemotron-Personas-USA.
  • 10,088 Hidden Facts distilled from life-story interviews.
  • 18,261 App-Trace Fragments (Messenger and Calendar).
  • 4,810 Verified Test Cases.

The framework is designed to be scalable, allowing researchers to generate new personas for different demographic groups or app formats.

Key Insights and Ideas

  • The Reliability Gap: The authors introduce the $Pass^3$ metric, which counts a test case as solved only if the model gets it right in three independent runs. While models might show decent "overall" accuracy, their $Pass^3$ scores are abysmally low (often <10% for complex tasks), suggesting that LLM performance on personal context is highly inconsistent.
  • Synthesis is Harder than Retrieval: Models are relatively good at finding a single fact (T1/T2) but struggle significantly when they must synthesize multiple facts to infer a higher-level pattern (T4), such as a user's professional identity or underlying motives.
  • Temporal Blindness: Models frequently fail at Temporal State Tracking (T3). They often detect that a change has occurred but fail to reconstruct the current state accurately, often mixing it up with "stale" information from earlier in the logs.
  • Constraint Recognition Failures: In agent-oriented tasks (T5), the primary failure mode (37.5% of errors) is failing to recognize a constraint at all. This leads to "blind compliance" with potentially unsafe user requests.

Key Results

The evaluation compared cost-effective SOTA models (like GPT-4o-mini and Gemini-3-Flash) against a high-performance reference (Claude Sonnet 4.6).

  • Performance Drop-off: Accuracy drops sharply as tasks move from fact retrieval (T1/T2) to reasoning and behavior (T3–T5).
  • Consistency Crisis: For GPT-4o-mini, while overall accuracy on T3–T5 was around 50%, the $Pass^3$ score was below 4%. Even the high-performing Claude Sonnet 4.6 achieved only ~34.6% $Pass^3$ on average.
  • Failure Analysis: As shown in the failure category table, T3 errors are dominated by omitting current-state facts (81%), while T5 errors are led by constraint recognition failures.
Table 2: Overall accuracy, worst-run accuracy, and Pass3 accuracy on P-ContextBench. Avg. denotes the test-count-weighted average across T1–T5.

Limitations and Future Work

  • App Diversity: The current instantiation is limited to messenger and calendar logs. The authors plan to expand to richer multimodal traces (e.g., photos, location history).
  • System-Level Evaluation: The current study focuses on standalone LLMs. Future work will evaluate complete agent systems, including their long-term memory architectures, RAG pipelines, and specific safety safeguards.
  • Demographic Specificity: While the current seeds reflect the U.S. population, the authors aim to explore domain-specific subpopulations where personal context might follow different patterns.

Summary updated

Abstract

Large language models (LLMs) are increasingly used for personal tasks, yet evaluating their safe use of personal context remains difficult because such contexts are private, fragmented, dynamic, and socially situated. We introduce P-ContextBench, a realistic, verifiable, and scalable benchmark for evaluating personal-context understanding. P-ContextBench constructs population-structured, life-story-grounded digital humans from demographic seeds reflecting the U.S. population. Rather than using explicit, static profiles, P-ContextBench embeds traits, routines, goals, and constraints as implicit evidence across fragmented, evolving synthetic mobile-app records. It evaluates five capabilities: single-source fact recovery, multi-source fact recovery, temporal state tracking, inference of higher-level personal patterns from routine behavior, and safe handling of requests conflicting with persona-specific constraints through compliance, warning, clarification, or refusal. A multi-stage validation pipeline checks persona plausibility, context recoverability, information leakage, evidence preservation, test-case grounding, and human-judge agreement. We instantiate P-ContextBench with 100 personas, 10,088 hidden facts, 18,261 app-trace fragments, and 4,810 verified test cases. The pipeline supports configurable generation across scales and demographic groups. Cost-effective state-of-the-art (SOTA) LLMs relevant to personal agents achieve below 60% accuracy on temporal tracking, higher-level pattern inference, and constraint-aware action-boundary decisions, with fewer than 10% of test cases answered correctly across all three repeated runs. Even a high-performing SOTA model remains inconsistent on these capabilities. P-ContextBench provides measurable targets for improving the accuracy, reliability, and safety of LLM-based personal agents.
BibTeX
@misc{brown2026p,
  author = {Grant Nasir Brown and Abdul Basir Bari and Anmol Tadikonda and Isabel Song},
  title = {{P-ContextBench: From Personal Fact Recall to Personalized Agent Behavior}},
  year = {2026},
  eprint = {2610.40959},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40959},
}
Figure