TL;DR
PJB (Person-Job Benchmark) is a new diagnostic evaluation framework designed to move recruitment AI beyond simple leaderboards. By categorizing real-world job-resume matching tasks by industry domain and reasoning complexity, it reveals that specialized domain-adapted models far outperform general-purpose ones, and that common "upgrades" like query rewriting can actually degrade performance in high-stakes retrieval scenarios.
What Problem Does This Paper Address?
In the world of Information Retrieval (IR), we are reaching a point of diminishing returns on general benchmarks like MTEB or BEIR. While a model might score high on average, these scores don't tell engineers where a system fails or why.
This is particularly problematic in Person-Job (PJ) retrieval. Matching a resume to a job description isn't just about semantic similarity; it requires two distinct types of logic:
- Parallel Reasoning: Verifying explicit, independent constraints (e.g., Does the candidate have 5 years of experience? Are they in the right city? Do they have a Master’s degree?).
- Serial Reasoning: Performing multi-hop inference for "job-competency" (e.g., Can a "Product Manager" from the gaming industry transfer their skills to "User Operations" in e-commerce?).
Existing benchmarks treat recruitment as a generic text-matching task, failing to provide the diagnostic "capability map" needed to build production-ready HR systems.
Methods
The authors propose PJB v1.0, a benchmark that formalizes PJ matching as a "reasoning-aware" task. Their approach introduces two novel taxonomies for diagnostic slicing:
- Domain Taxonomy: Queries are grouped into six "domain families" (e.g., Tech R&D, Sales & Support, HR/Admin) based on shared competency dimensions.
- Reasoning Taxonomy: Queries are scored on two axes:
- Parallel Width: The number of explicit constraints to be checked.
- Serial Depth: The number of semantic inference steps required to judge competency.
To build the dataset, they used a three-stage pipeline: sourcing real-world data, using a "candidate recall" ensemble (BM25 + dense retrieval) to find high-potential pairs, and finally employing a two-stage LLM-as-a-Judge process (using Doubao-1.5 and Kimi-2.5) supplemented by human spot-checks to determine relevance.
Datasets
The paper introduces PJB v1.0, which consists of:
- 297 Queries: Complete, real-world job descriptions (JDs).
- 197,674 Documents: De-identified, complete resumes (CVs).
- 2,242 Positive Judgments: A sparse-label, multi-positive retrieval set.
- Data Source: Internal search logs from 2025, spanning six major industry domains.
Unlike previous datasets that use short snippets, PJB uses full-length documents, capturing the structural complexity of real recruitment.
Key Insights and Ideas
- Domain Adaptation is Non-Negotiable: The in-house model (CRE-T1), which was pre-trained on recruitment data, outperformed a general-purpose model (Qwen3-Embedding) of the same size by roughly 3.5x. General-purpose embeddings simply lack the "vocabulary" of professional competencies.
- The "Query Understanding" (QU) Penalty: Surprisingly, adding a query rewriting module (using Qwen3-8B) consistently degraded performance. The authors suggest that rewriting may lose the rigid structural constraints (like specific years of experience) inherent in the original job description.
- Reranking is the Only Stable Win: While reranking provided a steady +8.9% gain for the domain-adapted model, it actually hurt the general-purpose model. This suggests that post-processing modules cannot compensate for a fundamentally weak base retriever; they only amplify existing quality.
- Heterogeneity Matters: A model might perform excellently in "Tech R&D" but fail in "Sales & Support." PJB shows that aggregate scores hide these critical business-level blind spots.
Key Results
The primary metric used was nDCG@10.
- Baseline Performance: The domain-specific CRE-T1-0.6B achieved an nDCG@10 of 0.2070, while the general-purpose Qwen3-Embedding-0.6B trailed at 0.0592.
- Module Ablation: For the CRE-T1 model, adding a Reranker boosted the score to 0.2253. However, combining Query Understanding (QU) and Reranking dropped the score significantly to 0.1423.
- Reasoning Slices: On "serial-dominant" queries (those requiring deep inference), the Reranker provided a massive boost, nearly doubling the nDCG from 0.1879 to 0.3809. This proves that rerankers are specifically better at the "reasoning" part of the task than standard bi-encoders.
Limitations and Future Work
- Retrieval Scope: The current study focuses exclusively on dense retrieval. The authors acknowledge that results might differ for BM25 or hybrid (dense + sparse) pipelines.
- Sample Size: Some diagnostic buckets (like Project Management) have very few queries, leading to potentially unstable statistics in those specific slices.
- Label Sparsity: The dataset uses "positive-only" labels. Unlabeled pairs are treated as negatives, which is standard for TREC-style evaluation but may not capture the full picture of model behavior.
- Future Directions: The team plans to expand the benchmark to include more natural-language query formulations and to test a wider variety of end-to-end system architectures.
Summary updated