PAL-Bench: Evidence-Grounded Profile Reconstruction from Longitudinal Personal Albums

Qiwei Yan, Zhiqiang Yuan, Zexi Jia, Nanxing Hu, Kailin Lyu, Jie Zhou, Jinchao ZhangarXiv 2026 · arXiv

TL;DR

PAL-Bench is a novel, privacy-safe benchmark designed to evaluate how well AI agents can reconstruct structured owner profiles and social networks from long-term personal photo albums while accurately citing public evidence. To tackle this challenge, the authors introduce PAL-TRACE, a database-inspired framework that stabilizes identity bindings (who is who) before mining open-ended facts, significantly outperforming standard large language model (LLM) and retrieval-augmented generation (RAG) baselines.

What Problem Does This Paper Address?

Our personal photo albums are essentially messy, "weak-schema" multimodal databases. They contain a chaotic mix of faces, receipts, screenshots, and location data spanning months or years. Reconstructing a coherent story from this data—such as identifying the owner's hobbies, occupation, or recurring social circle—requires complex "perceptual joins" across multiple modalities and timeframes.

Evaluating an AI's ability to perform this task is incredibly difficult due to a fundamental privacy paradox: the complete ground truth required for evaluation (true profiles, social graphs, and face-name maps) is exactly the kind of highly sensitive private data that real users cannot safely share. Existing benchmarks either focus on single images (VQA) or lack the rigorous, auditable ground truth needed to test long-term, album-scale reasoning.

Methods

The authors address this challenge with a two-part approach:

  1. The Evidence Compiler: To create a clean, privacy-safe benchmark, the authors built a pipeline that generates "latent private worlds" (synthetic profiles, timelines, and social graphs). It programs specific cross-modal evidence paths, renders realistic photos using a generative model (Gemini 3.1 Flash Image), and then processes those images through separate perception pipelines (OCR, face clustering, captioning). Crucially, the AI agents being tested only see these perception-derived public records, ensuring no hidden metadata or private states are leaked.
  2. PAL-TRACE Framework: As a baseline solution, the authors propose PAL-TRACE (Personal Album Traceable Reconstruction via Anchored Cross-modal Evidence). Borrowing principles from database master-data management, PAL-TRACE separates identity state from fact mining. It first locks down and stabilizes person identities across the album (Anchored Identity Tracing) before allowing an open-world LLM to mine facts about the owner under those strict identity constraints.
[Public Album] ➔ [Evidence Inventory] ➔ [Stage 1: Identity Tracing (Frozen)] ➔ [Stage 2: Type-Open Fact Mining] ➔ [Structured Profile]

Datasets

The paper introduces PAL-Bench, which includes:

  • 50 synthetic users with distinct occupations, cultures, and archetypes.
  • 36,659 public photo records containing captions, OCR text, and face clusters.
  • 2,799 evaluation targets covering 840 owner-fact atoms and 653 distinct social identities.

To ensure the synthetic data reflects reality, the authors conducted a 10-participant real-album audit. Volunteers scanned their own private albums to confirm that the complex multi-photo evidence chains and recurring identity structures controlled in PAL-Bench genuinely exist in real-world personal data.

Key Insights and Ideas

  • Profile Summarization $\neq$ Social Reconstruction: Standard LLMs are decent at skimming text or captions to guess an owner's general lifestyle, but they fail miserably at tracking and naming recurring faces over time.
  • More Compute Without Discipline Hurts: Naively giving an LLM more context or allowing it to loop through the album repeatedly actually degrades identity tracking. Without a frozen identity table, the model constantly re-imagines who people are in fresh prompt contexts, leading to identity drift.
  • Faces are Structurally Necessary: Field-level ablations revealed that while text carries the bulk of owner-fact signals, faces are structurally mandatory to anchor social graphs. Removing face data collapses identity metrics to zero, proving that album reconstruction cannot be solved by text processing alone.

Key Results

The evaluation utilized a seven-metric protocol, highlighting Owner Fact Recovery (OFR) and Person Identity Resolution (PIR):

  • PAL-TRACE achieved the highest overall performance with an OFR of 0.6057 and a PIR of 0.4792 using a GPT-5.4 backbone.
  • When paired with a frontier model like Gemini 3.1 Pro Preview, PAL-TRACE pushed performance even further to 0.6142 OFR and 0.5648 PIR.
  • In contrast, standard Long-context Multimodal LLMs lagged significantly behind (0.4009 OFR / 0.2336 PIR), proving that raw context length cannot substitute for structured state management.
  • The Hard Axis: Despite PAL-TRACE's success, "hard" identity targets (those requiring cross-month or cross-event reasoning) remain an open challenge, with the framework resolving fewer than 1 in 5 hard identities (0.1657 PIR).

Limitations and Future Work

While PAL-Bench provides a rigorous testbed, it features a higher density of clean OCR data (82.4%) than typical real-world personal albums (found to be around 17% in the user audit). Additionally, the dataset size of 50 users is sufficient for benchmarking but does not capture rare demographics or long-tail cultural variations.

Future research directions suggested by the work include developing more robust perceptual entity-resolution layers, improving alias consolidation (e.g., linking nicknames to full names), and refining temporal alignment under highly degraded or noisy real-world conditions.

Summary updated

Abstract

Longitudinal personal albums are weak-schema multimodal databases: noisy perceptual records whose key facts require joins across faces, text, timestamps, locations, and repeated events. Existing visual, video, document, and lifelog benchmarks test sub-problems, but not album-scale profile reconstruction with social identity binding and evidence citation. Benchmarking this task is difficult because the ground truth needed for evaluation--owner profiles, social graphs, face-name maps, and evidence provenance--is private state that real albums cannot safely release. We introduce PAL-Bench, a controlled benchmark for evidence-grounded reconstruction under a public-record contract. Its Evidence Compiler builds latent private worlds, programs target-level evidence paths, renders album pixels, re-measures them through perception pipelines, and exports audited public/private views. Agents receive only perception-derived public records; targets, identifier maps, and evidence paths remain hidden. PAL-Bench contains 50 synthetic users, 36,659 public photo records, and 2,799 targets over owner facts, identities, and relations. A privacy-preserving audit with 10 participants confirms that PAL-Bench evidence structures match real private albums, though equivalent releases remain privacy-prohibitive. Across seven systems and two compute-matched diagnostics, a seven-metric protocol reveals a gap between plausible profile summarization and faithful social reconstruction: systems recover some owner facts but struggle with recurring identities and evidence citation. PAL-TRACE, a reference framework that freezes identity bindings before owner-fact mining, performs best but leaves hard identity resolution far from solved. PAL-Bench provides a testbed for perceptual entity resolution, multimodal data integration, temporal evidence aggregation, and provenance-aware structured prediction.
BibTeX
@misc{yan2026pal,
  author = {Qiwei Yan and Zhiqiang Yuan and Zexi Jia and Nanxing Hu and Kailin Lyu and Jie Zhou and Jinchao Zhang},
  title = {{PAL-Bench: Evidence-Grounded Profile Reconstruction from Longitudinal Personal Albums}},
  year = {2026},
  eprint = {2606.16175},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.16175},
}

Loading related content...

Figure