AttenSy-SNER: software knowledge entity extraction with syntactic features and semantic augmentation information

Mingjing Tang, Tong Li, Wei Gao, Yu XiaComplex 2022 · DOI

This paper, published in Complex & Intelligent Systems, introduces AttenSy-SNER, a specialized framework designed to extract named entities (like APIs, libraries, and bug exceptions) from the messy, unstructured world of software engineering (SE) forums like StackOverflow.

TL;DR

AttenSy-SNER is a novel named entity recognition (NER) model that combines domain-specific BERT embeddings with syntactic dependency graphs (via GCNs) and a unique semantic augmentation strategy. It significantly outperforms standard models, particularly in identifying rare or "out-of-vocabulary" software entities that typically trip up general-purpose AI.

What Problem Does This Paper Address?

Software knowledge communities like StackOverflow are goldmines of information, but the data is "noisy." Developers use abbreviations (e.g., "JS" for JavaScript), encounter rare libraries that aren't in standard dictionaries, and use words that are highly ambiguous (e.g., "Mac" could be an operating system or a specific library).

Traditional NER models, often trained on formal news text, struggle with this "dev-speak." They face four major hurdles:

  1. Entity Variation: Multiple names for the same thing.
  2. Entity Sparsity: Many software-specific terms appear rarely.
  3. Ambiguity: The same word meaning different things in different contexts.
  4. Lack of Labeled Data: There aren't many high-quality, human-annotated datasets for software-specific entities.

Methods

The authors propose a multi-layered architecture to tackle these challenges:

  • Domain-Specific BERT: Instead of using a "vanilla" BERT, they pre-trained the model on a massive corpus of 43.5 million sentences from StackOverflow. This ensures the model understands the technical context of words from the start.
  • Syntactic Feature Encoding (GCN): The model doesn't just look at the order of words; it looks at their structural relationship. Using a Graph Convolutional Network (GCN) and dependency parsing, the model understands how a verb relates to its object, even if they are far apart in a sentence.
  • Semantic Augmentation (The "AttenSy" part): This is the core innovation. To help the model recognize rare words, it looks up "similar words" from a domain thesaurus (Sethesaurus) and the pre-trained BERT space. It then uses an attention mechanism to weight these similar words, enriching the representation of the target word before classification.
  • BiLSTM-CRF Backbone: It uses the industry-standard Bidirectional Long Short-Term Memory (BiLSTM) for sequence context and a Conditional Random Field (CRF) layer to ensure the final output tags (like "B-API" or "I-API") follow logical sequences.

Datasets

The study utilized two primary data sources:

  1. Unlabeled Pre-training Data: 2.6M questions and 5.5M answers from StackOverflow (2008–2018) used to fine-tune the BERT model.
  2. Annotated Dataset: The authors manually labeled a new dataset of 6,371 sentences containing 40 different entity categories (e.g., Object-oriented, Query language, Bug exception). This dataset is a significant contribution to the field, as it provides a fine-grained benchmark for SE-NER.

Key Insights and Ideas

  • Syntax is a Signal: By using GCNs to map the "dependency graph" of a sentence, the model can identify entities based on their grammatical role, which is more stable than just looking at neighboring words.
  • The Power of "Neighborhoods": If the model sees a rare library it doesn't recognize, the semantic augmentation strategy looks at semantically similar words (like "jQuery" or "React"). By seeing how those similar, well-known words behave, the model can make a much better guess about the rare entity.
  • Domain Knowledge is Essential: The jump in performance from standard BERT to StackOverflow-pre-trained BERT confirms that general language models are "illiterate" when it comes to specialized technical jargon.

Key Results

The AttenSy-SNER model showed dominant performance across the board:

  • Overall Performance: It achieved an F1 score of 74.17%, significantly higher than the baseline BiLSTM-CRF (63.94%) and even BERT-BiLSTM-CRF (69.25%).
  • Handling the Unknown: The most impressive result was in Out-of-Vocabulary (OOV) recognition. AttenSy-SNER achieved a 59.48% recall rate for OOV entities, whereas the standard BiLSTM-CRF managed only 35.41%.
  • Ablation Success: Removing the GCN or the semantic augmentation layers led to clear drops in accuracy, proving that both structural and semantic features are necessary for high-quality extraction.

Limitations and Future Work

While powerful, the authors noted a few areas for improvement:

  • Complexity vs. Overfitting: They found that adding too many GCN layers (more than two) actually decreased performance, likely due to overfitting on the specific sentence structures of the training set.
  • Relationship Extraction: This paper focuses on finding the entities. The next logical step, which the authors suggest for future work, is "Relationship Extraction"—automatically determining how these entities interact (e.g., "Library A is a dependency of Framework B").
  • Computational Overhead: The multi-layered approach (BERT + GCN + Attention) is computationally heavier than simpler models, which might be a consideration for real-time applications.

Summary updated

Abstract

Abstract Software knowledge community contains a large scale of software knowledge entity information, complex structure and rich semantic correlations. It is significant to recognize and extract software knowledge entity from software knowledge community, as it has great impact on entity-centric tasks such as software knowledge graph construction, software document generation and expert recommendation. Since the texts of the software knowledge community are unstructured by user-generated texts, it is difficult to apply the traditional entity extraction method in the domain of the software knowledge community due to the problems of entity variation, entity sparsity, entity ambiguity, out-of-vocabulary (OOV) words and the lack of annotated data sets. This paper proposes a novel software knowledge entity extraction model, named AttenSy-SNER, which integrates syntactic features and semantic augmentation information, to extract fine-grained software knowledge entities from unstructured user-generated content. The input representation layer utilizes Bidirectional Encoder Representations from Transformers (BERT) model to extract the feature representation of the input sequence. The contextual coding layer leverages the Bidirectional Long Short-Term Memory (BiLSTM) network and Graph Convolutional Network (GCN) for contextual information and syntactic dependency information, and a semantic augmentation strategy based on attention mechanism is introduced to enrich the semantic feature representation of sequences as well. The tag decoding layer leverages Conditional Random Fields (CRF) to solve the dependency between the output tags and obtain the global optimal label sequence. The results of model comparison experiments show that the proposed model has better performance than the benchmark model in software engineering domain.
BibTeX
@inproceedings{tang2022attensy,
  author = {Mingjing Tang and Tong Li and Wei Gao and Yu Xia},
  title = {{AttenSy-SNER: software knowledge entity extraction with syntactic features and semantic augmentation information}},
  booktitle = {Complex},
  year = {2022},
  doi = {10.1007/s40747-022-00742-5},
}
Figure