Scientific and Technological Text Knowledge Extraction Method of based on Word Mixing and GRU

Suyu Ouyang, Yingxia Shao, Junping Du, Ang LiarXiv 2022 · DOI · arXiv

Abstract

Knowledge extraction from Chinese scientific and technological texts is hindered by linguistic nuances and complex technical jargon. Traditional pipeline methods often suffer from error propagation and lose correlations between entity recognition and relation extraction. This paper proposes MBGAB, a joint extraction model that integrates character and word-level embeddings ("Word Mixing") with a Bi-GRU encoder and self-attention mechanism. A key innovation is the introduction of a biased loss function to prioritize meaningful entities over background text. Experimental results on a Chinese technical corpus demonstrate that MBGAB significantly outperforms baseline models, achieving an F1 score of 56.1%.

Introduction

Knowledge extraction—the process of transforming unstructured text into structured (entity-relation-entity) triples—is fundamental for building domain-specific knowledge graphs. In the context of Chinese scientific literature, two primary challenges exist. First, the "pipeline problem" in traditional systems leads to slow processing and a lack of feedback between entity and relation identification. Second, the absence of spaces in Chinese text causes "boundary segmentation errors," where technical terms are incorrectly split, leading to semantic loss.

Methodology: The MBGAB Model

The proposed MBGAB (Word Mixing, Bi-GRU, Attention, and Bias weight) model addresses these challenges through a multi-layered architecture designed for joint extraction.

Architecture of the MBGAB model showing the flow from word mixing to the output layer.

Word Mixing Vector Mapping

To mitigate segmentation errors, the model employs a hybrid embedding approach. It combines character-level vectors with word-level vectors derived from pre-trained Word2Vec models. This ensures that even if a word boundary is ambiguous, the character-level information preserves the underlying semantic meaning.

Bi-GRU Encoder and Self-Attention

A bidirectional Gated Recurrent Unit (Bi-GRU) processes the text to capture bidirectional context. To handle long-distance dependencies common in complex technical sentences, a self-attention layer is integrated. This layer allows the model to relate distant words regardless of their position in the sequence.

GRU Decoder with Bias Weight

The decoder uses a GRU-based sequence labeling approach. A critical component is the introduction of a bias weight ($\alpha$) in the loss function. Since most tokens in a sentence are "Outside" (O) tags, models often become biased toward predicting "O." The $\alpha$ parameter forces the model to pay more attention to the rarer but more significant entity and relation tags.

Experimental Results and Discussion

The model was evaluated using a Chinese corpus from the Baidu Language and Intelligent Technology Competition, containing over 24,000 training sets and 19 relationship types.

Impact of Bias Weight ($\alpha$)

The study found that the bias weight is crucial for balancing precision and recall. As shown in the performance analysis, the model's F1 score and recall improve as $\alpha$ increases from 1.

Performance metrics (Precision, Recall, and F1) as a function of the bias weight alpha.

The "sweet spot" was identified at $\alpha = 3$, where the model achieves the best balance. Beyond this point, precision begins to decline as the model may over-predict relations in its attempt to avoid "O" tags.

Comparative Performance

MBGAB achieved an F1 score of 56.1% and an accuracy of 65.3%, outperforming the pipeline-based FCM model (41.3% F1) and standard Bi-GRU joint models (51.3% F1). Qualitative analysis showed the model's high proficiency in identifying professional vocabulary and complex relationships in technical case studies.

Conclusion

The MBGAB model demonstrates that joint extraction combined with hybrid embeddings and biased loss functions is highly effective for technical Chinese text. Future work will focus on scaling the model to handle thousands of niche relations and addressing overlapping relations where a single entity participates in multiple triples.

Summary updated

Abstract

The knowledge extraction task is to extract triple relations (head entity-relation-tail entity) from unstructured text data. The existing knowledge extraction methods are divided into "pipeline" method and joint extraction method. The "pipeline" method is to separate named entity recognition and entity relationship extraction and use their own modules to extract them. Although this method has better flexibility, the training speed is slow. The learning model of joint extraction is an end-to-end model implemented by neural network to realize entity recognition and relationship extraction at the same time, which can well preserve the association between entities and relationships, and convert the joint extraction of entities and relationships into a sequence annotation problem. In this paper, we propose a knowledge extraction method for scientific and technological resources based on word mixture and GRU, combined with word mixture vector mapping method and self-attention mechanism, to effectively improve the effect of text relationship extraction for Chinese scientific and technological resources.
BibTeX
@misc{ouyang2022scientific,
  author = {Suyu Ouyang and Yingxia Shao and Junping Du and Ang Li},
  title = {{Scientific and Technological Text Knowledge Extraction Method of based on Word Mixing and GRU}},
  year = {2022},
  eprint = {2203.17079},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.17079},
}
Figure