TL;DR
This paper tackles the challenge of finding approximate matches between job skill descriptions, moving beyond exact keyword matching. They propose a novel semantic role-based approach that outperforms standard statistical methods and achieves human-level agreement by focusing on the core components of skills.
What Problem Does This Paper Address?
In today's knowledge-intensive industries, efficiently matching skilled individuals to job requirements is crucial. Current systems often rely on exact skill statement matches, which can miss qualified candidates whose skills are described using slightly different phrasing. For example, someone skilled in "maintaining a database" might be overlooked if a job explicitly asks for "maintaining DB2." This paper addresses the need for a system that can understand and compute semantic similarity between skill statements to enable approximate matching, thereby expanding the pool of potential candidates and improving the accuracy of skill-based recruitment.
Methods
The authors explore two main approaches:
- Statistical Information-Theoretic Approach (Lin's Measure): This baseline method treats skill statements as "bags of words" and calculates similarity based on shared vocabulary and corpus frequencies. It uses unigram features and a stop-word list, stemming words to group related terms.
- Semantic Role Matching Approach: This is the paper's novel contribution. It leverages natural language parsing to extract structural information from skill statements. Specifically, they identify semantic roles (like "action," "theme," "purpose," and a crucial "concept" role representing the core skill component) within verb phrases. The similarity is then computed by matching these identified semantic roles, giving more weight to the key "concept" role.
To support the semantic approach, they also:
- Preprocess Skill Statements: This involved expanding abbreviations, converting words to lowercase (except for acronyms and sentence beginnings), and adding a subject ("Employees") to convert phrases into full sentences to improve parsing accuracy.
- Evaluate Parsers: They compared four parsers (Charniak, Stanford, ESG, MINIPAR) for their ability to accurately parse the domain-specific skill statements, focusing on bracketing and syntactic labels. The Charniak parser was found to be the best for their task.
- Extract Semantic Role Patterns: They manually identified patterns for 18 skill verbs to extract semantic roles from the parsed trees.
Datasets
The primary dataset used is a corpus of 10,667 skill statements extracted from an enterprise-wide expertise taxonomy. For evaluation, they created a benchmark of 75 skill pairs, which were judged by three human raters to establish an upper bound for accuracy.
Key Insights and Ideas
- Structure Matters: Standard "bag-of-words" approaches, while useful, ignore the underlying structure of language. For skill matching, understanding how words relate (e.g., "database" vs. "DB2" as the theme of "maintaining") is critical.
- The "Concept" Role is King: The "concept" role within the "theme" of a skill statement (e.g., "CAD functionality" or "Process" in "Advise Business Knowledge of...") is the most important component for determining semantic similarity. Matching on this core element significantly improves accuracy.
- Domain-Specific Challenges: Parsing domain-specific text with abbreviations, acronyms, and specialized terminology is difficult. Preprocessing steps are essential for achieving good parsing performance.
- Human Agreement as a Benchmark: Establishing human agreement on skill similarity provides a realistic upper bound for automated systems, guiding the evaluation of their proposed methods.
Key Results
- Statistical Approach Performance: Lin's information-theoretic measure achieved 70.67% accuracy on the 75-pair test set, significantly underperforming human agreement.
- Semantic Role Matching Performance: The semantic role matching approach, focusing on the "concept" role and using a Tversky-like similarity criterion, achieved 80% accuracy. This significantly outperforms the statistical method and is very close to the human agreement upper bound of 81%.
- Parser Performance: The Charniak parser was identified as the best performer for parsing the skill statements, outperforming Stanford, ESG, and MINIPAR.
Limitations and Future Work
- Domain-Specific Lexicon: While the semantic role approach is promising, it still struggles with distinguishing very similar-sounding but conceptually different terms (e.g., "Field Force Automation" vs. "Sales Force Automation"). The authors suggest that a domain-specific ontology or lexicon could help overcome this.
- Skill Verb Similarity: The paper notes that skill verb similarity is also important (e.g., "implement" vs. "develop for"). While they explored this using WordNet, they found that general-purpose taxonomies might not fully capture domain-specific nuances. Future work could involve using domain-specific verb lexicons or exploring other similarity measures for verbs.
- Generalizability: The authors believe their approach of using domain-specific pattern analysis on parse trees to identify structural elements can generalize to other domains where short texts need comparison.
Summary updated