PJB: A Reasoning-Aware Benchmark for Person-Job Retrieval

Guangzhi Wang, Xiaohui Yang, Kai Li, Jiawen He, Kai Yang, Ruixuan Zhang, Zhi Liu

TL;DR

PJB (Person-Job Benchmark) is a new diagnostic evaluation framework designed to move recruitment AI beyond simple leaderboards. By categorizing real-world job-resume matching tasks by industry domain and reasoning complexity, it reveals that specialized domain-adapted models far outperform general-purpose ones, and that common "upgrades" like query rewriting can actually degrade performance in high-stakes retrieval scenarios.

What Problem Does This Paper Address?

In the world of Information Retrieval (IR), we are reaching a point of diminishing returns on general benchmarks like MTEB or BEIR. While a model might score high on average, these scores don't tell engineers where a system fails or why.

This is particularly problematic in Person-Job (PJ) retrieval. Matching a resume to a job description isn't just about semantic similarity; it requires two distinct types of logic:

  1. Parallel Reasoning: Verifying explicit, independent constraints (e.g., Does the candidate have 5 years of experience? Are they in the right city? Do they have a Master’s degree?).
  2. Serial Reasoning: Performing multi-hop inference for "job-competency" (e.g., Can a "Product Manager" from the gaming industry transfer their skills to "User Operations" in e-commerce?).

Existing benchmarks treat recruitment as a generic text-matching task, failing to provide the diagnostic "capability map" needed to build production-ready HR systems.

Methods

The authors propose PJB v1.0, a benchmark that formalizes PJ matching as a "reasoning-aware" task. Their approach introduces two novel taxonomies for diagnostic slicing:

  • Domain Taxonomy: Queries are grouped into six "domain families" (e.g., Tech R&D, Sales & Support, HR/Admin) based on shared competency dimensions.
  • Reasoning Taxonomy: Queries are scored on two axes:
  • Parallel Width: The number of explicit constraints to be checked.
  • Serial Depth: The number of semantic inference steps required to judge competency.

To build the dataset, they used a three-stage pipeline: sourcing real-world data, using a "candidate recall" ensemble (BM25 + dense retrieval) to find high-potential pairs, and finally employing a two-stage LLM-as-a-Judge process (using Doubao-1.5 and Kimi-2.5) supplemented by human spot-checks to determine relevance.

Datasets

The paper introduces PJB v1.0, which consists of:

  • 297 Queries: Complete, real-world job descriptions (JDs).
  • 197,674 Documents: De-identified, complete resumes (CVs).
  • 2,242 Positive Judgments: A sparse-label, multi-positive retrieval set.
  • Data Source: Internal search logs from 2025, spanning six major industry domains.

Unlike previous datasets that use short snippets, PJB uses full-length documents, capturing the structural complexity of real recruitment.

Key Insights and Ideas

  • Domain Adaptation is Non-Negotiable: The in-house model (CRE-T1), which was pre-trained on recruitment data, outperformed a general-purpose model (Qwen3-Embedding) of the same size by roughly 3.5x. General-purpose embeddings simply lack the "vocabulary" of professional competencies.
  • The "Query Understanding" (QU) Penalty: Surprisingly, adding a query rewriting module (using Qwen3-8B) consistently degraded performance. The authors suggest that rewriting may lose the rigid structural constraints (like specific years of experience) inherent in the original job description.
  • Reranking is the Only Stable Win: While reranking provided a steady +8.9% gain for the domain-adapted model, it actually hurt the general-purpose model. This suggests that post-processing modules cannot compensate for a fundamentally weak base retriever; they only amplify existing quality.
  • Heterogeneity Matters: A model might perform excellently in "Tech R&D" but fail in "Sales & Support." PJB shows that aggregate scores hide these critical business-level blind spots.

Key Results

The primary metric used was nDCG@10.

  • Baseline Performance: The domain-specific CRE-T1-0.6B achieved an nDCG@10 of 0.2070, while the general-purpose Qwen3-Embedding-0.6B trailed at 0.0592.
  • Module Ablation: For the CRE-T1 model, adding a Reranker boosted the score to 0.2253. However, combining Query Understanding (QU) and Reranking dropped the score significantly to 0.1423.
  • Reasoning Slices: On "serial-dominant" queries (those requiring deep inference), the Reranker provided a massive boost, nearly doubling the nDCG from 0.1879 to 0.3809. This proves that rerankers are specifically better at the "reasoning" part of the task than standard bi-encoders.

Limitations and Future Work

  • Retrieval Scope: The current study focuses exclusively on dense retrieval. The authors acknowledge that results might differ for BM25 or hybrid (dense + sparse) pipelines.
  • Sample Size: Some diagnostic buckets (like Project Management) have very few queries, leading to potentially unstable statistics in those specific slices.
  • Label Sparsity: The dataset uses "positive-only" labels. Unlabeled pairs are treated as negatives, which is standard for TREC-style evaluation but may not capture the full picture of model behavior.
  • Future Directions: The team plans to expand the benchmark to include more natural-language query formulations and to test a wider variety of end-to-end system architectures.

Summary updated

Abstract

As retrieval models converge on generic benchmarks, the pressing question is no longer "who scores higher" but rather "where do systems fail, and why?" Person-job matching is a domain that urgently demands such diagnostic capability -- it requires systems not only to verify explicit constraints but also to perform skill-transfer inference and job-competency reasoning, yet existing benchmarks provide no systematic diagnostic support for this task. We introduce PJB (Person-Job Benchmark), a reasoning-aware retrieval evaluation dataset that uses complete job descriptions as queries and complete resumes as documents, defines relevance through job-competency judgment, is grounded in real-world recruitment data spanning six industry domains and nearly 200,000 resumes, and upgrades evaluation from "who scores higher" to "where do systems differ, and why" through domain-family and reasoning-type diagnostic labels. Diagnostic experiments using dense retrieval reveal that performance heterogeneity across industry domains far exceeds the gains from module upgrades for the same model, indicating that aggregate scores alone can severely mislead optimization decisions. At the module level, reranking yields stable improvements while query understanding not only fails to help but actually degrades overall performance when combined with reranking -- the two modules face fundamentally different improvement bottlenecks. The value of PJB lies not in yet another leaderboard of average scores, but in providing recruitment retrieval systems with a capability map that pinpoints where to invest.
BibTeX
@misc{wang2026pjb,
  author = {Guangzhi Wang and Xiaohui Yang and Kai Li and Jiawen He and Kai Yang and Ruixuan Zhang and Zhi Liu},
  title = {{PJB: A Reasoning-Aware Benchmark for Person-Job Retrieval}},
  year = {2026},
  eprint = {2603.17386},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.17386},
}
Figure