Claims Versus Evidence: Correcting Self-Report Inflation in Text-Based Soft Skill Scoring

Systems that score soft skills from text, such as résumé-to-job matchers and skill extractors, treat what a candidate writes about themselves as ground truth. A résumé is a self-report, and self-reports about desirable traits are inflated in predictable ways. This note proposes splitting soft-skill language into two kinds of span. Claims are self-descriptions like "excellent communicator, strong team player." Evidence is described actions like "mediated a dispute between two suppliers and renegotiated delivery terms." The proposal treats claims as a biased self-report and evidence as a sparse but more trustworthy anchor, and asks how much weight to give the anchor before candidates learn to game it. The prediction is concrete. Matching models fail on soft-skill-heavy roles largely because claim language dominates the input. Down-weighting claims in favour of evidence should close part of that gap, but only up to a point.

The design is shaped by an empirical regularity borrowed from heteroatom doping. In that field, foreign atoms such as nitrogen are inserted into a carbon lattice to create reactive sites. Doping helps at low concentration and destroys the material at high concentration. We use this only as a hypothesis about the shape of the curve, not as a mechanism. We also borrow a methodological lesson from that field's own admitted failures: always build the undoped control under identical conditions.

Core Claim

Text-based soft-skill scores inherit self-report inflation. Separating claims from evidence and relying on the evidence spans will improve prediction of external outcomes for soft-skill roles, with a peak at an intermediate evidence weight. The second half of the claim concerns gaming. When candidates or a language model rewrite text to imitate evidence phrasing, an evidence-heavy scorer loses more than a balanced one. That loss defines the "over-doping" regime. As far as the file shows, no prior scorer separates these two kinds of span or measures this trade-off.

Where This Sits in Prior Work

The motivating gap comes from the person-job fit literature. Person-Job Fit: Adapting the Right Talent for the Right Job with Joint Representation Learning matches résumés to postings with a two-sided convolutional network. Its authors report weaker accuracy on soft-skill-dominated roles such as marketing and operations, and on generic requirement statements. Enhancing Person-Job Fit for Talent Recruitment: An Ability-aware Neural Network Approach adds layered attention over abilities. Both models take résumé text at face value. Our proposal keeps their architectures and changes what they are allowed to trust.

The extraction layer we build on is SkillSpan: Hard and Soft Skill Extraction from English Job Postings. It annotates skill spans in job postings and finds that skills appear as long verb-noun phrases, distinct from knowledge terms. That observation matters here because evidence spans are exactly such action phrases, while claims tend to be adjectival. SkillSpan does not separate the two, and it works on postings rather than candidate self-descriptions.

The self-report side comes from two sources. Linguistic analysis of empathy in medical school admission essays mined essays to sidestep social desirability in empathy scales. It found only a weak correlation (r = .074) and explicitly warned that publicized markers would be gamed. We take that warning as a design constraint and measure gaming directly. Investigating the Reliability of Self-report Data in the Wild: The Quest for Ground Truth showed that self-reported engagement diverges from wristband physiology. It offered no model for correcting the divergence. The telemetry studies, Skill Assessment Using Behavior Data in Virtual World and Providing objective metrics of team communication skills via interpersonal coordination mechanisms, replace self-report with behaviour. They do so on 26 students and 6 pilot dialogues respectively, and in settings where such logging exists. Most hiring decisions have no such logging, which is why we stay in text.

The doping papers play a constraining, analogical role. Oxygen reduction reactions on pure and nitrogen-doped graphene: a first-principles modeling finds that 4% nitrogen lowers the reaction barrier from 1.09 to 0.19 eV, while 50% nitrogen irreversibly poisons the surface. Boosting CO2 reduction on Fe-N-C with sulfur incorporation admits two things: it lacks solid evidence for its claimed higher active-site density, and it could not synthesize a matched iron-free control. That paper is itself an instance of a confident claim outrunning evidence, and it is the cautionary model for our ablations.

Research Program

The first step is to build the distinction. We will add a claim-versus-evidence layer to the soft-skill spans in SKILLSPAN, and to a matched sample of candidate-side text such as résumés or professional self-descriptions. The annotation guideline will be written so that agreement can be compared against SkillSpan's reported Fleiss' κ of 0.70–0.75. We will then train a span tagger initialized from JobSpanBERT.

The second step is to model the bias. For candidate $i$ and skill $k$, let $\theta_{ik}$ be the latent skill level. Let the claim signal be $c_{ik} = \theta_{ik} + b_i + \epsilon_{ik}$, where $b_i$ is a person-level inflation term. Let the evidence signal be $e_{ik} = \theta_{ik} + \eta_{ik}$, which is noisier and sparser but carries no shared inflation term. The scorer then combines the two with a mixing weight $\rho$:

$$\hat{\theta}_{ik} = \rho\, e_{ik} + (1-\rho)\,(c_{ik} - \hat{b}_i),$$

Here $\hat{b}_i$ is estimated from the gap between a candidate's claims and their evidence across all of their skills.

In the matching models, $\rho$ is applied as a gate on span representations before pooling. Sweeping $\rho$ from 0 to 1 produces a dose-response curve. The linear form follows from treating claims and evidence as two noisy measurements of the same latent skill. Under that model the optimal $\rho$ depends on their relative noise. Gaming raises the noise of evidence once evidence is rewarded, which predicts an interior optimum.

The third step measures the poisoning regime. We will rewrite held-out résumés with a language model instructed to convert claims into evidence-like phrasing without adding new facts. For each $\rho$ we then measure how much scores inflate. This tests the empirical worry raised in the essay study.

Baselines

The naive reference point is a matcher trained on raw résumé text with no span separation. This is the PJFNN-style input, and it corresponds to $\rho$ effectively undefined, with claims and evidence mixed. It is run on the same splits, with the same encoder and tuning budget as every variant. Beating it means a reliably higher held-out AUC on the soft-skill-role stratum, without loss on technical roles.

The strongest existing method is APJFNN from Enhancing Person-Job Fit for Talent Recruitment, which reports an AUC of 0.8316 on its million-record recruitment data. PJFNN from Person-Job Fit reports 0.758 AUC on real hiring logs, against 0.715 for gradient-boosted trees. These numbers come from different proprietary datasets and are not directly comparable to ours. We will therefore reimplement APJFNN on our own splits with an equal hyperparameter search budget, and report its number alongside ours. Beating it means a gain on the soft-skill stratum that holds across at least three random seeds and is larger than seed variance.

The key ablation is an "undoped control" in the spirit of the lesson from the Fe-NS-C paper. It removes the same number of tokens as the claim down-weighting, but chosen at random from soft-skill spans. If this random removal helps as much as targeted removal, the gain comes from shortening or regularizing the input, not from addressing self-report bias. A second ablation uses claims only, which isolates the value of the evidence channel.

For the tagger itself, the reference is JobSpanBERT's 56.64 F1 on skill spans from SkillSpan. There is no reported result found on Lacuna for claim-versus-evidence tagging.

Why not just…?

Why not just fine-tune a larger encoder end-to-end and let it learn to discount claims? This is the most serious threat. A large model trained on hire outcomes might implicitly down-weight boilerplate. Our experiments settle it directly: the large encoder is a baseline with the same budget. If it matches the claim/evidence model on the soft-skill stratum, the explicit decomposition is not needed for accuracy. The proposal would then survive only on the gaming experiment and on interpretability, and we will say so. The prediction we are betting on is narrower. An end-to-end model also rewards claim-to-evidence rewriting, because it has no notion of which phrasing is cheap to fake.

Why not just filter adjectives with a lexicon? A word list approximates claims cheaply. It misses claims phrased as clauses ("known for building consensus"), and it deletes adjectives inside genuine evidence. The random-removal and lexicon-filter variants will both be run. If the lexicon filter captures most of the gain, we will report the trained tagger as unnecessary.

Why not just collect objective telemetry or physiology instead of text? The behavioural studies in the file show this works where logging exists. Skill Assessment Using Behavior Data in Virtual World found a −0.54 correlation between shortcut use and teacher ratings. The communication study found r = 0.73 between prosodic accommodation and expert ratings. But both rest on tiny samples. Investigating the Reliability of Self-report Data in the Wild shows that wristband signals are themselves context-dependent. Doped-carbon sensors may eventually make such signals cheaper, but hiring and admissions decisions are made on text today. That is where the bias currently enters.

Why not just weight criteria by expert pairwise judgment, using the analytic hierarchy process? Expert weighting, as in Satria Abadi et al., fixes how much each skill matters. It does not detect whether a stated skill is inflated, so it complements rather than replaces our approach.

Proposed Experiments

The outcome data is the main practical risk. The recruitment logs used by PJFNN and APJFNN are proprietary. We will seek comparable logs through an industry partner. If access fails, the fallback outcome is a held-out set of soft-skill ratings from trained raters who score structured interview or work-sample transcripts. These raters will be blind to the résumé. The work-sample transcripts are themselves a stronger "evidence" signal than the résumé they are compared against.

We will stratify roles into technical and soft-skill-dominated groups, following the split along which PJFNN reported degradation. The primary metric is AUC per stratum, with seed variance reported.

The first decisive check is the dose-response curve over $\rho$. Gains that rise and then fall support the core claim. A monotonic curve means evidence should simply replace claims. A flat curve means the decomposition carries no signal. In each case we also check whether claim-only or random removal matches targeted removal.

The second decisive check is the gaming test. Our prediction is that score inflation under language-model rewriting grows with $\rho$. If inflation is flat, the "over-doping" concern is unfounded for current rewriters.

The third check tests the bias model outside hiring, on data that has an objective comparison. In the Figshare engagement data from Investigating the Reliability of Self-report Data in the Wild, we estimate a person-level offset $\hat{b}_i$ and ask whether subtracting it improves agreement with physiology for held-out reports. This dataset has only 23 students, so the result is suggestive at best. We will try the same check on the mood and Fitbit pairing from Assessing Real-Time Moderation for Developing Adaptive Mobile Health Interventions for Medical Interns if access is granted. A person-level offset that fails to transfer there would weaken the claim that inflation is a stable trait worth modeling.