Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026

Warre Veys, Luis Gasco, Jens-Joris DecortearXiv 2026 · arXiv

This paper from the WorkRB Skill Extraction Challenge at RecSys-HR 2026 argues that we’ve been measuring skill extraction all wrong. By moving from "right-or-wrong" binary labels to a nuanced 0–4 grading system, the authors reveal that the real challenge isn't just finding the "gold" skill—it's cleaning up the "nonsense" that models often rank alongside it.

TL;DR

Standard skill extraction benchmarks are "blind" to semantically nonsensical predictions because they only reward finding a few human-annotated positives. This paper introduces a graded relevance framework (0–4) using an LLM-judge cascade, revealing that top-performing models win by eliminating "nonsense" rather than finding more positives—but they currently require massive, inefficient scale to do so.

What Problem Does This Paper Address?

In HR technology, skill extraction models (which map job descriptions or profiles to a standard vocabulary like ESCO) are usually evaluated using binary metrics: a prediction is either a "hit" (it matches a human label) or a "miss."

However, ESCO contains nearly 14,000 skills. If a model misses the "gold" skill but suggests a highly plausible alternative, binary metrics treat it as a total failure. Conversely, if a model ranks the gold skill first but fills the rest of the top-10 with completely unrelated "nonsense" (e.g., suggesting "design hair style" for a mechanical engineering job because both contain the word "design"), binary metrics don't care. This "simplicity bias" rewards models that rely on lexical shortcuts rather than true semantic understanding, which can lead to unfair or nonsensical automated hiring decisions.

Methods

The authors propose a graded relevance rubric (0 to 4) to provide a higher-resolution view of model performance:

  • 0 (Nonsense): Unrelated domain.
  • 1 (Plausible): Right domain, but the activity isn't actually mentioned.
  • 2 (Recommendable): Correct domain and activity, but wrong granularity (too broad).
  • 3 (Additional): Correct and core to the text, but not in the original gold set.
  • 4 (Gold/Replacement): Semantically equivalent to a human-annotated positive.

To make this objective, they developed an LLM-as-a-judge cascade. Instead of asking an LLM for a 0–4 grade directly (which is inconsistent), they use a tree of four simple binary questions (e.g., "Is it the same domain?"). The grade is determined by how far the candidate skill "descends" through these gates.

The annotation cascade used to determine graded relevance.

Datasets

The study utilizes five benchmarks, all mapped to the ESCO v1.1.0 vocabulary:

  1. House, Tech, TechWolf, and SkillSkape: Four existing skill extraction datasets consisting of vacancy sentences, re-annotated with the new graded labels.
  2. SkillNorm: A new dataset specifically for skill normalization. It uses "alternative labels" from ESCO that are lexically distant but semantically identical to the preferred label (e.g., "manage gum conditions" vs. "provide periodontal treatment") to test if models can move beyond simple keyword matching.

Key Insights and Ideas

  • Binary metrics hide the truth: At the top of the leaderboard, the ranking of teams by binary metrics was almost entirely uncorrelated with the graded metrics (Kendall $\tau = 0.06$). The model that was "best" at finding gold positives was only 16th when it came to overall window quality.
  • Winning is about "Tail Hygiene": The best models don't necessarily find more "Grade 4" skills; everyone is already pretty good at that. The winners distinguish themselves by having far fewer "Grade 0" (nonsense) predictions in their top-100 results.
  • The "Brute Force" Gap: Participants closed the performance gap using massive scale. Every system that beat the baselines used at least 42x (and often >200x) more parameters. We are currently "buying" accuracy with massive compute rather than smarter architectures.
  • Taxonomy is under-baked: Despite ESCO being a rich, hierarchical graph, most winning teams treated it as a flat list of text descriptions. The structural relationships in the taxonomy provided almost no measurable lift.

Key Results

  • Leaderboard: The winner (thylinao) achieved a macro-averaged nDCG@100 of 0.8458, significantly outperforming the trained baselines (0.60–0.62).
  • Nonsense Reduction: As shown in the grade composition analysis, the winning model only allowed 9 "nonsense" (Grade 0) items into its top 100 slots, whereas a standard BM25 lexical search filled 79 slots with nonsense.
Figure 4: Grade composition of the 100 slots that nDCG@100 reads (mean per query, macro over the four extraction tasks) for the ideal ranking, the best and median valid submissions, and the organiser baselines. The ideal window contains essentially no grade-0 items, and the ranking systems differ mostly in how many grade-0 items they let in (9 for the winner, 17 at the median, 34–37 for the trained baselines, 79 for BM25) while everyone finds nearly all ≈2.5 grade-4 slots.
  • Scale vs. Score: There is a clear "empty middle" in system sizes. Baselines sit at ~110M parameters, while the smallest competitive submission was 4.6B. However, above 10B parameters, increasing size further yielded diminishing returns.

Limitations and Future Work

  • Circularity: Since the "ground truth" labels were generated by an LLM, and the top models are also LLM-based, there is a risk that the metric simply rewards models that share the same "biases" as the judge.
  • Efficiency: The field currently lacks a "deployable" solution. A 13-stage cascade of frontier LLMs is too slow and expensive for a real-time job board. The next frontier is achieving this "tail hygiene" in small, efficient models.
  • Multilinguality: The current benchmarks are English-only, whereas the labor market (and ESCO itself) is inherently multilingual.

Summary updated

Abstract

Skill extraction benchmarks score models against a few human-annotated positives and treat every other skill in the vocabulary as equally wrong. At the scale of ESCO (13,939 skills) this binary view can reward models that rank the annotated positive high for the wrong reason, and hides semantically nonsensical predictions. The WorkRB Challenge at RecSys-HR 2026 replaces it with graded relevance: every candidate skill in a model's ranking window carries a 0 to 4 grade, and the leaderboard scores nDCG@100 against these grades. Since ternary judgements proved too coarse for objective annotation, relevance is decomposed into a cascade of four binary questions (same domain, activity mentioned, right granularity, replacement of a gold positive) that a smaller LLM judge can answer reliably; the labels are validated with blind human and frontier-model re-annotation. This paper documents the annotation methodology, the construction of five benchmark datasets (four with vacancy sentences as queries, one with lexically distant skill surface forms), the validation of the LLM cascade, and insights from the submissions. The field converged on multi-stage retrieve-and-rerank pipelines, whose gains come from clearing nonsense out of the ranking window rather than finding more positives: at the top of the leaderboard, the graded ordering is uncorrelated with what binary evaluation would have said. Graded labels expose much bigger gaps between top submissions and baselines than binary evaluation does, and show that semantic-understanding skill recommendation is still wide open. The clear next direction is bringing efficiency back: every submitted system we could size uses at least 42x, typically several hundred times, more parameters than the baselines it beats.
BibTeX
@misc{veys2026grading,
  author = {Warre Veys and Luis Gasco and Jens-Joris Decorte},
  title = {{Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026}},
  year = {2026},
  eprint = {2611.03525},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03525},
}
Figure