Hot Keyword Extraction of Sci-tech Periodicals Based on the Improved BERT Model

Bing Liu, Lv Zhi-jun, Nan Zhu, Dongyu Chang, Mengxin LüKSII 2022 · DOI

TL;DR

Researchers have developed i-BERT, an enhanced version of the BERT language model specifically designed to extract "hot" keywords from scientific journals. By optimizing how the model calculates semantic distance and handles compound phrases, i-BERT achieves significantly higher accuracy (77.5%) and recall (82.1%) than traditional methods like TF-IDF or standard BERT.

What Problem Does This Paper Address?

Scientific fields move fast. For journal editors and researchers, identifying emerging "hot topics" is crucial for planning special issues or choosing research directions. However, traditional manual extraction is slow and prone to bias.

Existing automated methods struggle with two main things:

  1. Compound Words: Most scientific keywords are complex phrases (e.g., "innovation incubation performance"). Standard models often treat these as single, sparse vectors, losing the nuanced meaning of the individual components.
  2. The Semantic Distance Gap: In standard BERT, the Euclidean distance between word vectors doesn't always map cleanly to human semantic understanding. Two "very similar" words might still appear mathematically distant, leading to poor clustering of related topics.

Methods

The authors propose the i-BERT (Improved BERT) framework, which introduces several clever tweaks to the standard NLP pipeline:

  • Multi-Element Semantic Sets: Instead of treating a compound keyword as one block, they use the "Jieba" segmentation tool to split it into a set of simple and overlapping words (e.g., "ABC" becomes {A, B, C, AB, BC, ABC}).
  • Nonlinear Distance Mapping: This is the "Improved" part. The authors found that standard BERT distances are often non-intuitive. They introduced a weight function based on a normal distribution to "correct" these distances—effectively pulling synonyms closer together and pushing unrelated words further apart.
  • Density Clustering (CFSFDP): They use the "Clustering by Fast Search and Find of Density Peaks" algorithm. This allows the model to identify "hotspots" in the data—clusters of keywords that appear frequently and are semantically related—without needing to pre-define the number of clusters.
  • Automated Label Generation: Once a cluster is found, the model uses a logic-based step to select the most representative "central word" for that topic based on frequency and position.

Datasets

The study utilized a substantial real-world dataset:

  • Source: 21 different social science and management periodicals collected from the CNKI (China National Knowledge Infrastructure) database.
  • Volume: 14,420 articles published between 2017 and 2019.
  • Evaluation: The model’s performance was benchmarked against manually marked "hot words" and compared to four baseline models: TF-IDF, Word2vec, standard BERT, and Decision Trees.

Key Insights and Ideas

  • Synonyms Attract: The core "aha moment" is the realization that standard vector spaces are too "rigid." By applying a nonlinear mapping to the weights of matching elements, the authors created a "magnetic" effect where semantically similar terms are more likely to aggregate into a single hot topic.
  • Context is King, but Structure Matters: While BERT is famous for understanding context, the authors show that for keyword extraction, the structural decomposition of compound phrases is just as important as the deep learning architecture itself.
  • Timeliness: The model is specifically tuned to ensure "timeliness," meaning it can capture a rising trend even if authors haven't yet settled on a single, unified term for a new phenomenon.

Key Results

The i-BERT model outperformed all baseline methods across the board:

Method Accuracy Recall
TF-IDF 66.4% 62.3%
Word2vec 68.4% 67.5%
BERT (Standard) 74.4% 73.2%
i-BERT (Proposed) 77.5% 82.1%
Decision Tree 66.3% 70.1%

The jump in Recall (82.1%) is particularly noteworthy, suggesting that i-BERT is much better at "finding" the hot topics that other models miss due to variations in wording.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Data Source: The current study relies on keywords provided by authors. Future iterations could apply this model to abstracts or full-text content, which would provide a much richer (though noisier) dataset for discovery.
  • Long-term Evolution: The current study looks at a three-year window. The authors suggest that future research should look at larger time spans to study how hot topics evolve, merge, or die out over decades.
  • Cross-Disciplinary Mining: Testing the model on interdisciplinary fields where terminology is even more fragmented would be a rigorous next step.

Summary updated

Abstract

With the development of the economy and the improvement of living standards, the hot issues in the subject area have become the main research direction, and the mining of the hot issues in the subject currently has problems such as a large amount of data and a complex algorithm structure. Therefore, in response to this problem, this study proposes a method for extracting hot keywords in scientific journals based on the improved BERT model.It can also provide reference for researchers,and the research method improves the overall similarity measure of the ensemble,introducing compound keyword word density, combining word segmentation, word sense set distance, and density clustering to construct an improved BERT framework, establish a composite keyword heat analysis model based on I-BERT framework.Taking the 14420 articles published in 21 kinds of social science management periodicals collected by CNKI(China National Knowledge Infrastructure) in 2017-2019 as the experimental data, the superiority of the proposed method is verified by the data of word spacing, class spacing, extraction accuracy and recall of hot keywords. In the experimental process of this research, it can be found that the method proposed in this paper has a higher accuracy than other methods in extracting hot keywords, which can ensure the timeliness and accuracy of scientific journals in capturing hot topics in the discipline, and finally pass Use information technology to master popular key words.
BibTeX
@inproceedings{liu2022hot,
  author = {Bing Liu and LV Zhi-jun and Nan Zhu and Dongyu Chang and Mengxin Lü},
  title = {{Hot Keyword Extraction of Sci-tech Periodicals Based on the Improved BERT Model}},
  booktitle = {KSII},
  year = {2022},
  doi = {10.3837/tiis.2022.06.002},
}
Figure