Abstract
Mihai Surdeanu, John Hungerford, Yee Seng Chan, Jessica MacBride, Benjamin Gyori, Andrew Zupon, Zheng Tang, Haoling Qiu, Bonan Min, Yan Zverev, Caitlin Hilverman, Max Thomas, Walter Andrews, Keith Alcock, Zeyu Zhang, Michael Reynolds, Steven Bethard, Rebecca Sharp, Egoitz Laparra. Proceedings of the Second Workshop on Bridging Human--Computer Interaction and Natural Language Processing.
2022.
Introduction
In the field of Information Extraction (IE), taxonomies serve as the essential knowledge backbones that categorize and ground extracted data. However, building these hierarchies is traditionally a manual, labor-intensive process that struggles to keep pace with the volume of new data. Taxonomy Builder addresses this bottleneck by providing a data-driven, human-in-the-loop tool designed to accelerate the creation and extension of domain-specific taxonomies.
System Architecture and Methods
The tool utilizes a hybrid architecture that balances heavy computational tasks with a responsive user interface, divided into offline and online phases.
Offline Pre-processing
To filter noise and identify relevant concepts, the system first performs extractive summarization on the corpus using a Luhn-inspired technique. Noun and verb phrases are then extracted from these salient sentences and ranked via an extended TextRank algorithm. Finally, the top-ranked phrases are encoded using DistilBERT contextualized embeddings and grouped using the Clustering By Committee (CBC) algorithm. This ensures that suggested concepts are semantically coherent and relevant to the specific domain.
Online Interactive Session
The web-based UI allows users to curate the generated suggestions. The workflow is split between two main components:
- Cluster Viewer: Users review suggested clusters, selecting high-quality phrases and rejecting noise.
- Taxonomy Editor: Users place the accepted concepts into the existing hierarchical structure.
The Curation Workflow
The core of the user experience is the Cluster Viewer, which presents ranked concepts for human evaluation. By keeping the human "in the loop," the system ensures high precision while significantly reducing the manual effort required to discover new nodes in a large corpus.
Evaluation and Results
The tool was tested on real-world scenarios involving food insecurity and regional security. Key findings include:
- Efficiency: Experts were able to process an average of 200 taxonomy candidates per hour.
- Quality: Approximately 16% of suggested clusters were deemed high-quality enough to be added to formal taxonomies.
- Improved Grounding: In food insecurity studies, the updated taxonomy led to a 7.6% relative increase in the correlation between grounding confidence and correctness.
- Downstream Impact: For regional security, the tool enabled the discovery of 33.8% more nodes and 27.6% more causal edges in knowledge graphs.
Conclusion
Taxonomy Builder demonstrates that combining automated summarization and clustering with human curation can significantly streamline taxonomy construction. By leveraging contextualized embeddings and a user-centric design, the tool provides a scalable solution for maintaining the knowledge structures necessary for modern IE systems.
Summary updated