CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors

Hang Su, Zequn Liu, Chen Hu, Xuesong Lu, Yingce Xia, Zhen Liu

TL;DR

CoPA introduces a novel framework for evaluating personalized Question Answering (QA) by moving beyond simple word-matching to deeper cognitive alignment. By mining "Community-Individual Preference Divergence" (CIPD) from Stack Overflow, the authors distilled six key cognitive factors—such as Cognitive Trust and Schema Consistency—to build a benchmark of 1,985 user profiles that significantly outperforms traditional metrics in identifying truly personalized responses.

What Problem Does This Paper Address?

While Large Language Models (LLMs) are excellent at answering general questions, tailoring those answers to a specific user (personalization) remains difficult to measure. Current evaluation methods typically rely on lexical overlap (like BLEU or ROUGE scores), which only check if the model uses the same words as a reference text.

However, personalization is about intent and context: explaining "quantum physics" to a PhD student requires different depth and terminology than explaining it to a middle-schooler. The challenge is that two answers can both be high-quality and factually correct, yet one is "better" for a specific individual. The authors seek to bridge this gap by identifying the latent cognitive drivers that cause a user to prefer one answer over a community-voted favorite.

Methods

The authors leverage a phenomenon they call Community-Individual Preference Divergence (CIPD). They analyzed Stack Overflow data where a user "accepted" an answer that was not the top-voted answer by the community. This divergence provides a clear signal of individual preference over consensus.

Figure 1: This Stack Overflow example illustrates Community-Individual Preference Divergence (CIPD): the user accepted an empirical, code-based solution while the community overwhelmingly favored a con- ceptual, schematic one.

To turn this signal into an evaluation framework, they followed a multi-step pipeline:

  1. Rationale Extraction: Using LLMs to analyze user histories and explain why a user chose a specific answer over the community favorite.
  2. Factor Distillation: Aggregating these rationales into six core cognitive dimensions:
  3. Cognitive Trust (CT): Credibility and verifiability.
  4. Situational Anchoring (SA): Calibration to the user's immediate, specific problem.
  5. Schema Consistency (SC): Alignment with the user's prior knowledge.
  6. Cognitive Load Management (CLM): Complexity tailored to the user's capacity.
  7. Metacognitive Scaffolding (MS): Structural support for critical thinking.
  8. Affective and Motivational Resonance (AMR): Emotional and motivational alignment.
  9. Benchmark Construction: Creating the CoPA (Cognitive-based Personalized Assessment) benchmark, which includes 1,985 unique user profiles across diverse domains.
Figure 3: The pipeline of the Factor Distillation process. We first collect and filter raw CIPD data (a), then employ LLMs to extract decision rationales behind user choices (b), and finally distill these rationales into six cognitive factors through multi-model consensus (c).

Datasets

The primary data source is Stack Exchange, covering four major categories: Engineering & Tools, Science & Theory, Lifestyle & Society, and Leisure & Fandom.

  • Initial Pool: 6.8 million questions.
  • Final CIPD Dataset: 626,786 questions from 15,963 users.
  • CoPA Benchmark: A curated subset of 1,985 users with detailed historical interaction profiles.
  • External Validation: The authors also tested their factors on existing benchmarks like UPGC-QA and LaMP-QA to prove generalizability.

Key Insights and Ideas

  • Personalization is Pervasive: CIPD isn't just for subjective topics; it occurs in 12-18% of questions even in objective technical fields like Engineering & Tools.
  • Lexical Metrics are Blind: Traditional metrics like BLEU/ROUGE fail to distinguish between "Accepted" and "Top-voted" answers because both are high-quality. CoPA’s factor-based scoring shows a much higher "Accuracy" (the ability to rank the user's choice higher than the community's).
  • The "Scaffolding" Gap: Detailed factor analysis revealed that while models are good at managing cognitive load, they struggle significantly with Metacognitive Scaffolding—helping the user reflect on their own learning process.
  • Profile-Based Personalization Wins: Constructing structured, semantically rich user profiles (Profile-Pers) consistently outperformed simpler methods like RAG or time-based history.

Key Results

The authors compared their factor-based evaluation against "LLM-as-a-Judge" and heuristic baselines.

  • Discriminative Power: CoPA achieved an Accuracy of 51-55% in identifying the user-accepted answer, compared to only ~12-18% for direct LLM scoring.
  • Tie Reduction: The "Tie Rate" (where a judge can't decide which answer is better) dropped from over 80% in direct scoring to roughly 18-26% using CoPA factors.
  • Generalizability: Integrating these six factors into the system prompts of existing models led to significant performance gains on external benchmarks like LaMP-QA, proving these cognitive dimensions are universal.
Figure 4: The proposed evaluation pipeline.

Limitations and Future Work

  • Factor Orthogonality: The authors admit the six factors are not perfectly independent; for example, Cognitive Trust relies heavily on Schema Consistency.
  • Data Source Bias: The factors were distilled from Stack Overflow, which is learning-centric. They may not fully capture preferences in other contexts like creative writing or social chat.
  • LLM Dependency: The framework relies on LLMs for both distilling factors and acting as a judge, which could introduce "epistemic circularity" or model-specific biases.
  • Future Directions: The team plans to integrate more heterogeneous data sources and explore "human-in-the-loop" validation to further refine the objectivity of the benchmark.

Summary updated

Abstract

While LLMs have demonstrated remarkable potential in Question Answering (QA), evaluating personalization remains a critical bottleneck. Existing paradigms predominantly rely on lexical-level similarity or manual heuristics, often lacking sufficient data-driven validation. We address this by mining Community-Individual Preference Divergence (CIPD), where individual choices override consensus, to distill six key personalization factors as evaluative dimensions. Accordingly, we introduce CoPA, a benchmark with 1,985 user profiles for fine-grained, factor-level assessment. By quantifying the alignment between model outputs and user-specific cognitive preferences inferred from interaction patterns, CoPA provides a more comprehensive and discriminative standard for evaluating personalized QA than generic metrics. The code is available at https://github.com/bjzgcai/CoPA.
BibTeX
@misc{su2026copa,
  author = {Hang Su and Zequn Liu and Chen Hu and Xuesong Lu and Yingce Xia and Zhen Liu},
  title = {{CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors}},
  year = {2026},
  eprint = {2604.14773},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.14773},
}
Figure