Abstract
Concept extraction is a fundamental task for building knowledge bases from digital text. While deep learning models have advanced the field, they often require large amounts of labeled data and struggle with rare terms. This paper presents the Guided Attention Concept Extraction Network (GACEN), a novel framework that mimics human concept identification by utilizing document titles, global topics, and linguistic "clue words." By integrating a dual attention mechanism—topic-aware and position-aware—GACEN achieves superior performance and data efficiency compared to state-of-the-art models like BERT-CRF across diverse datasets.
Introduction
In the vast ocean of digital text, identifying core concepts such as "logistic regression" or "infinite set" is crucial for structuring information. Traditional neural models often treat text as a flat sequence of tokens, neglecting the rich meta-information available to human readers. For instance, a chapter title provides significant context, and specific linguistic markers like "is defined as" or "we introduce" act as "clue words" signaling nearby concepts. GACEN aims to bridge this gap by incorporating these signals into a structured neural architecture.
Methodology
The GACEN architecture consists of four primary components designed to capture both global context and local linguistic cues:
- Topic-based Encoder: Utilizes Latent Dirichlet Allocation (LDA) to generate topic distributions for documents and words, providing a "topic-aware" representation that situates concepts within a broader subject area.
- Soft Matching Module: A pre-trained module that identifies semantically similar clue words (e.g., "is called" vs. "is referred to as"), offering more flexibility than rigid rule-based systems.
- Dual Attention Layer:
- Topic-aware Attention: Aggregates information from the title and global topic for each word.
- Position-aware Attention: Focuses on text surrounding identified clue words, explicitly modeling the spatial relationship between a clue and a potential concept.
- CRF Layer: A Conditional Random Field layer models dependencies between output labels to ensure sequence consistency.
Experiments and Results
GACEN was evaluated on three datasets: CSEN (MOOC captions), KP-20K (scientific abstracts), and MTB (math textbooks). The model consistently outperformed baselines, including BERT-CRF. Notably, on the CSEN dataset, GACEN achieved an F1-score of 64.60%, a nearly 10% absolute improvement over BERT-CRF (55.26%).
A key finding is GACEN's remarkable data efficiency. The model reached performance levels with only 20% of the training data that standard Bi-LSTM-CRF models required 70% of the data to achieve. Furthermore, ablation studies showed that clue words primarily enhance precision, while topic information improves recall.
Conclusion
GACEN demonstrates that mimicking human intuition—using global context and local linguistic signals—can significantly improve concept extraction, especially in data-scarce scenarios. While the current model relies on manual clue word collection and LDA, future work could explore automated clue discovery and more robust topic modeling techniques to further enhance generalizability across languages and domains.
Summary updated