Abstract
Ryosuke Kohita, Issei Yoshida, Hiroshi Kanayama, Tetsuya Nasukawa. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
2020.
Introduction: The Need for Fine-Grained Dictionaries
In text analytics, researchers often rely on dictionaries—predefined lists of terms representing specific concepts—to extract insights from large corpora. While standard dictionaries for broad categories (like "countries" or "medicine names") are common, real-world analysis frequently requires "user-centric" dictionaries. These are highly specific, niche sets of terms tailored to an analyst's unique intent, such as "words describing a camera's appearance" versus "words describing its functionality."
Traditional methods like Set Expansion take a few seed words and find similar items. However, these tools often struggle with the subtle, overlapping boundaries of fine-grained categories. Manually building these lists is labor-intensive and suffers from a "cold start" problem: it is difficult for a machine to provide useful suggestions when given only one or two initial examples.
The Interactive Construction Framework
To address these challenges, the authors propose an interactive dictionary construction process. This human-in-the-loop framework functions as a continuous cycle: the system suggests candidate words, the user selects relevant terms, and the system updates its model based on this feedback to provide better suggestions in the next iteration.
Technical Innovations
The system incorporates several key technical strategies to improve performance and user experience:
- Feature Weighting with Predefined Similarity (FWPS): Standard word embeddings (like Word2Vec) may not always capture the user's specific intent. FWPS evaluates multiple feature spaces (e.g., TF-IDF, n-grams) and learns to weight the space that most accurately reflects the user's current selections.
- Min-Max Optimization: To prevent the system from getting "stuck" on a narrow cluster of very similar words, the algorithm employs a min-max approach. It seeks to maximize the similarity of the least similar positive term to the center of the group. This forces the system to explore the "edges" of the category, ensuring broader coverage of the user's intent.
- Feedback Denoising: Human feedback can be inconsistent. The system uses K-Means clustering to identify and ignore "outlier" terms selected by the user, preventing the model from being derailed by accidental or noisy input.
- The Hybrid Model (HB): The researchers found that similarity-based methods (like FWPS) excel during the "cold start" phase with very few labels. However, standard Logistic Regression (LR) becomes more effective as the number of labeled examples grows. The Hybrid Model starts with FWPS and automatically switches to LR after a few iterations to leverage the strengths of both approaches.
Evaluation and Results
The framework was tested using the Amazon review corpus across five domains: Apparel, Baby, Camera & Photo, Health & Personal Care, and Sports & Outdoors. To establish a ground truth, the authors used crowdsourcing to categorize terms into 22 fine-grained dictionaries.
Key Performance Metrics
The study introduced a metric called Weighted Coverage per Iteration (WCpI). This metric prioritizes finding correct terms in the earlier rounds of interaction, acknowledging that human analysts have limited patience and that early successes are more valuable for maintaining engagement.
Findings
- Superior Efficiency: The Hybrid Model (HB) consistently outperformed baselines, including standalone Word2Vec and the state-of-the-art SetExpan algorithm.
- Cold Start Robustness: Starting with only a single seed term, the proposed method achieved significantly higher "hit ratios" (the proportion of correct suggestions) in the first five iterations compared to traditional machine learning approaches.
- Importance of Diversity: The "min-max" approach proved essential for "stretching" the search area to find the full scope of the user's category, rather than just synonyms of the seed words.
Conclusion
This work demonstrates that an interactive, human-in-the-loop approach can significantly lower the barrier to creating high-quality, niche dictionaries for text analytics. By combining similarity-based methods for the cold start with robust classification for later stages, the system allows analysts to build precise tools with minimal manual effort. Future research may explore more complex negative feedback loops and conduct large-scale user studies to further refine the interaction design.
Summary updated