SKILL: A System for Skill Identification and Normalization

Meng Zhao, Faizan Javed, Ferosh Jacob, Matt McNairAAAI 2015 · DOI

TL;DR

This paper introduces SKILL, an automated system for identifying and normalizing professional skills from resumes and job descriptions. By leveraging Wikipedia for taxonomy generation and word embeddings for skill tagging, SKILL achieves high accuracy in bridging the gap between job market demands and candidate qualifications, aiding in workforce analytics and career projections.

What Problem Does This Paper Address?

The modern labor market faces a growing "skills gap," where a significant number of job openings remain unfilled despite a large pool of job seekers. This disconnect stems from the difficulty in accurately matching individuals' skills to the specific requirements of available positions. Traditional methods struggle with the inherent ambiguity of language, where the same skill can be expressed in various ways (e.g., "C#" vs. "C sharp") and the same term can have different meanings depending on context (e.g., "Java" for programming vs. coffee). This paper addresses the need for a robust system that can automatically identify, categorize, and normalize these skills, making human capital data more meaningful for recruitment, workforce training, and labor market analysis.

Methods

SKILL employs a two-component approach:

  1. Skills Taxonomy Generation:
    • Data Collection: Vocational skill sections from resumes and job postings are collected.
  2. Noise Reduction: A predefined dictionary of common non-skill terms (stopwords, adverbs, adjectives, etc.) is used to filter out irrelevant phrases.
  3. Wikipedia Integration: The cleaned "seed skill phrases" are then processed using Wikipedia. This involves:
  4. De-duplication and Normalization: Using Wikipedia's vast knowledge base to identify and consolidate similar skill terms.
  5. Category Tagging: Leveraging Wikipedia's category tags and decision rules, informed by the Standard Occupational Classification (SOC) system, to validate if a phrase represents a professional skill.
  6. Word Sense Disambiguation: For terms with multiple meanings (Type-II ambiguity), Google Search API rankings are used to determine the most relevant skill context, based on expert-approved relevancy.
  7. Skills Tagging:
    • Word Embeddings (Word2Vec): The system utilizes word2vec to create vector representations of words. This allows for capturing semantic relationships between words.
  8. Contextual Relevance: Instead of just matching exact words, word2vec helps in identifying skills by considering their surrounding words in a resume or job description, improving accuracy.
  9. Surface Form Trained Vectors: For production, vectors are trained specifically on the identified skill phrases (surface forms) to enhance semantic applicability and business congruity.
  10. N-gram Conversion: Spaces in multi-word skill phrases are replaced with underscores to treat them as single units for word2vec training, improving relevancy.
  11. Relevancy Score: A relevancy score is calculated for each matched surface form based on its co-occurrence with other matched surface forms within the text, using a defined formula. A threshold is applied to recognize and normalize skills.

Datasets

  • Training Data: Over 100 million English resumes from CareerBuilder (as of March 2014), containing approximately 46 million unique skill phrases. Additionally, half a billion job offers were analyzed.
  • Evaluation Data:
    • Gold Standard Taxonomy: The European Skills, Competences, Qualifications and Occupations (ESCO) system, a multilingual classification system, was used as a gold standard for evaluating recall.
  • User Survey: Approximately 3,000 responses from active CareerBuilder users (job applicants) were collected for evaluating the tagging system.

Key Insights and Ideas

  • Wikipedia as a Skill Knowledge Base: The paper highlights the effectiveness of Wikipedia not just for entity linking but also for building and validating a comprehensive skills taxonomy, especially when guided by established occupational classification systems like SOC.
  • Word Embeddings for Contextual Skill Recognition: Moving beyond simple keyword matching, the use of word2vec demonstrates the power of distributional semantics in understanding the context of skills, leading to more accurate identification and normalization.
  • The Importance of "Surface Forms": The system's focus on "surface forms" (how skills are actually written) and their normalization to formal entities is crucial for bridging the gap between raw text and structured data.
  • User Validation is Key: The paper emphasizes the value of user surveys for evaluating the practical accuracy of a skill tagging system, as end-users are the ultimate arbiters of their own skills.

Key Results

  • Taxonomy Generation: Achieved 91% accuracy and 76% recall. The generated taxonomy contains 50,000 surface forms mapped to 30,000 skill entities.
  • Skills Tagging: Achieved 82% accuracy and 70% recall.
  • User Approval: Over 80% of participated users approved at least 70% of the identified skills.

Limitations and Future Work

  • Hierarchical Skill Structures: The paper notes that hierarchical structures of skills will be analyzed in future work.
  • Word Sense Disambiguation Robustness: While Google Search API was used for disambiguation, the authors mention they are working on a more robust and reliable approach.
  • Taxonomy Expansion Complexity: The chosen "surface form trained vectors" approach, while effective, complicates taxonomy expansion, requiring retraining of the entire vector space if new surface forms are added.
  • API Service Deployment: The SKILL system has been exposed as an API service and is currently used in various workforce analytics and business intelligence projects at CareerBuilder.

Summary updated

BibTeX
@inproceedings{zhao2015skill,
  author = {Meng Zhao and Faizan Javed and Ferosh Jacob and Matt McNair},
  title = {{SKILL: A System for Skill Identification and Normalization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i2.19064},
}
Figure