Guided Attention Network for Concept Extraction

Songtao Fang, Zhenya Huang, Ming He, Shiwei Tong, Xiaoqing Huang, Ye Liu, Jie Huang, Qi Liu

Abstract

Concept extraction is a fundamental task for building knowledge bases from digital text. While deep learning models have advanced the field, they often require large amounts of labeled data and struggle with rare terms. This paper presents the Guided Attention Concept Extraction Network (GACEN), a novel framework that mimics human concept identification by utilizing document titles, global topics, and linguistic "clue words." By integrating a dual attention mechanism—topic-aware and position-aware—GACEN achieves superior performance and data efficiency compared to state-of-the-art models like BERT-CRF across diverse datasets.

Introduction

In the vast ocean of digital text, identifying core concepts such as "logistic regression" or "infinite set" is crucial for structuring information. Traditional neural models often treat text as a flat sequence of tokens, neglecting the rich meta-information available to human readers. For instance, a chapter title provides significant context, and specific linguistic markers like "is defined as" or "we introduce" act as "clue words" signaling nearby concepts. GACEN aims to bridge this gap by incorporating these signals into a structured neural architecture.

Methodology

The GACEN architecture consists of four primary components designed to capture both global context and local linguistic cues:

  1. Topic-based Encoder: Utilizes Latent Dirichlet Allocation (LDA) to generate topic distributions for documents and words, providing a "topic-aware" representation that situates concepts within a broader subject area.
  2. Soft Matching Module: A pre-trained module that identifies semantically similar clue words (e.g., "is called" vs. "is referred to as"), offering more flexibility than rigid rule-based systems.
  3. Dual Attention Layer:
  4. Topic-aware Attention: Aggregates information from the title and global topic for each word.
  5. Position-aware Attention: Focuses on text surrounding identified clue words, explicitly modeling the spatial relationship between a clue and a potential concept.
  6. CRF Layer: A Conditional Random Field layer models dependencies between output labels to ensure sequence consistency.
The overall architecture of GACEN, highlighting the Topic Based Encoder, Soft Matching Module, Attention Layer, and CRF Layer.

Experiments and Results

GACEN was evaluated on three datasets: CSEN (MOOC captions), KP-20K (scientific abstracts), and MTB (math textbooks). The model consistently outperformed baselines, including BERT-CRF. Notably, on the CSEN dataset, GACEN achieved an F1-score of 64.60%, a nearly 10% absolute improvement over BERT-CRF (55.26%).

A key finding is GACEN's remarkable data efficiency. The model reached performance levels with only 20% of the training data that standard Bi-LSTM-CRF models required 70% of the data to achieve. Furthermore, ablation studies showed that clue words primarily enhance precision, while topic information improves recall.

Performance analysis showing (a) F1-score vs. percent of training data and (b) the impact of clue word percentage on F1-score.

Conclusion

GACEN demonstrates that mimicking human intuition—using global context and local linguistic signals—can significantly improve concept extraction, especially in data-scarce scenarios. While the current model relies on manual clue word collection and LDA, future work could explore automated clue discovery and more robust topic modeling techniques to further enhance generalizability across languages and domains.

Summary updated

Abstract

Concept extraction aims to find words or phrases describing a concept from massive texts. Recently, researchers propose many neural network-based methods to automatically extract concepts. Although these methods for this task show promising results, they ignore structured information in the raw textual data (e.g., title, topic, and clue words). In this paper, we propose a novel model, named Guided Attention Concept Extraction Network (GACEN), which uses title, topic, and clue words as additional supervision to provide guidance directly. Specifically, GACEN comprises two attention networks, one of them is to gather the relevant title and topic information for each context word in the document. The other one aims to model the implicit connection between informative words (clue words) and concepts. Finally, we aggregate information from two networks as input to Conditional Random Field (CRF) to model dependencies in the output. We collected clue words for three well-studied datasets. Extensive experiments demonstrate that our model outperforms the baseline models with a large margin, especially when the labeled data is insufficient.
BibTeX
@inproceedings{fang2021guided,
  author = {Songtao Fang and Zhenya Huang and Ming He and Shiwei Tong and Xiaoqing Huang and Ye Liu and Jie Huang and Qi Liu},
  title = {{Guided Attention Network for Concept Extraction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/200},
}
Figure