TL;DR
Intel AI Lab introduces SetExpander, an iterative, corpus-based system designed to expand a small set of "seed" terms into a comprehensive list of semantically similar items. By combining multiple types of word embeddings with a neural classifier, it captures functional similarities that standard models miss, significantly speeding up domain-specific data extraction.
What Problem Does This Paper Address?
In Natural Language Processing (NLP), Term Set Expansion (TSE) is the task of taking a few examples (e.g., "Siri" and "Cortana") and finding others that belong to the same group (e.g., "Alexa" or "Google Assistant").
The challenge lies in the nuance of "similarity." Most standard models use a "bag-of-words" approach, which captures topical similarity. For instance, if you give a model the word "Python," a topical model might suggest "bytecode" or "compiler." However, for set expansion, we usually want functional similarity—other programming languages like "Java" or "C++."
Furthermore, the system must respect granularity. If your seeds are "orange" and "yellow," you want colors; if they are "orange" and "grapefruit," you want citrus fruits. Existing tools often struggle to balance these fine-grained distinctions in a user-friendly, end-to-end workflow.
Methods
SetExpander moves beyond simple word vectors by using a multi-context embedding approach. The researchers train five different types of embedding models for every term:
- Linear Bag-of-Words: The standard approach looking at neighboring words.
- Explicit Lists: Captures terms appearing in actual lists (e.g., "A, B, and C").
- Syntactic Dependency (Dep): Looks at how a term relates to verbs and other nouns in a sentence structure.
- Symmetric Patterns (SP): Identifies terms in patterns like "X and Y" or "X rather than Y."
- Unary Patterns (UP): Looks at specific n-grams where the term acts as a placeholder.
To combine these, they use a Multilayer Perceptron (MLP) classifier. This neural network takes ten similarity scores (two different scoring methods across the five context types) to predict whether a candidate term truly belongs in the expanded set.
The system is designed to be iterative: a user selects seeds, views the expansion, validates the good ones, and re-expands the set to refine the results.
Datasets
The authors evaluated SetExpander using:
- English Wikipedia: Used as a general-purpose corpus to calculate Mean Average Precision (MAP).
- Recruitment Corpus: A dataset of resumes and job descriptions used for a real-world "automated recruitment" use case.
- Software Defect Repository: A massive repository of millions of software defects used to identify duplicate issues.
Key Insights and Ideas
- Functional > Topical: The core "aha" moment is that different context types capture different semantic relations. While bag-of-words is great for general themes, syntactic dependencies and symmetric patterns are far superior for finding "co-hyponyms" (members of the same category).
- Context Complementarity: No single context type wins everywhere. For example, "Explicit Lists" worked best for the recruitment use case (where skills are often listed), while "Unary Patterns" excelled in the technical domain of software defects.
- Human-in-the-Loop: By making the system iterative, SetExpander allows the user to steer the "granularity" of the expansion, solving the ambiguity problem (e.g., deciding if "orange" means a color or a fruit based on user validation).
Key Results
- General Performance: On the Wikipedia dataset, the system achieved a MAP@10 of 0.83, meaning the top 10 results were highly accurate.
- Recruitment: When used to generate skills lists for job roles (like "Firmware Engineer"), the system achieved precision scores between 70.5% and 98.0% for the top 100 candidates.
- Efficiency: In the software defect use case, SetExpander improved duplicate detection precision by over 10% and reduced the time needed to generate term lists from several weeks to just a few hours.
Limitations and Future Work
The paper notes that the system relies heavily on the quality and size of the underlying corpus; if a term doesn't appear frequently in specific contexts (like symmetric patterns), the model's performance may degrade.
While not explicitly detailed as a "future work" section, the architecture suggests potential for integrating even more context types or using more advanced transformer-based embeddings (like BERT, which was emerging around the time of this paper's revision) to further refine the multi-context approach.
Summary updated