TL;DR
This paper introduces a novel Multi-Feature Fusion Transformer (MFT) model for Named Entity Recognition (NER) specifically tailored for the aerospace domain. By integrating character radicals, word-level information, and a dual Feed-Forward Neural Network (FFN) structure, MFT significantly improves entity recognition accuracy, achieving an F1 score of 86.10% on a newly created Chinese aerospace dataset.
What Problem Does This Paper Address?
The aerospace industry is rapidly integrating artificial intelligence, creating a vast amount of specialized text data. Extracting key information from this data is crucial, and Named Entity Recognition (NER) is a fundamental NLP task for this purpose. However, existing NER models often struggle with the complexities of Chinese text, particularly the lack of explicit word separators, which necessitates robust word segmentation. Furthermore, there's a lack of publicly available, domain-specific datasets for aerospace NER in Chinese. This paper aims to address these challenges by developing a more effective Chinese NER model for aerospace and by creating a relevant dataset.
Methods
The core of this work is the Multi-Feature Fusion Transformer (MFT) model. MFT builds upon the FLAT (Flat-Lattice-Transformer) architecture, which already incorporates a lattice structure to handle Chinese word segmentation ambiguities by considering both characters and potential words matched from a lexicon.
The key innovations in MFT are:
- Radical Feature Module: This module leverages the inherent semantic information within Chinese character radicals. It uses a 1D Convolutional Neural Network (1D-CNN) to extract embeddings from character radicals, which are then fused with word and character embeddings. This is particularly useful as radicals often carry semantic clues.
- Double Feed-Forward Neural Network (DFMS) Encoder: MFT employs a Transformer encoder with a "sandwich" structure. It incorporates a Double Feed-Forward Neural Network with a multi-head self-attention module in between. This design, inspired by Conformer, is shown to enhance feature encoding capabilities and improve performance.
- Multi-Feature Fusion: The model fuses embeddings from the Flat-Lattice module (character and word information) and the Radical Feature Module. This combined representation is then fed into the DFMS encoder.
The model uses a Conditional Random Field (CRF) decoder for the final label sequence prediction.
Datasets
The authors introduce a newly created Chinese aerospace domain dataset. This dataset was constructed by:
- Crawling data from Wikipedia and China Aerospace News.
- Screening the corpus to remove irrelevant content.
- Segmenting the corpus into sentences, ensuring each contained at least two aerospace entities.
- Manually labeling the data in the BIO (Begin, Inside, Outside) format using Label Studio.
The dataset contains 30,000 Chinese sentences and is divided into seven entity types: Aerospace Companies and Organizations (ACAO), Airports and Spacecraft Launch Sites (AASLS), Type of Aerospace Vehicle (TOAV), Constellations and Satellites (CAS), Space Missions and Projects (SMAP), Scientists and Astronauts (SAA), and Aerospace Technology and Equipment (ATAE). The dataset is split into 80% for training, 10% for validation, and 10% for testing.
The model was also evaluated on two public Chinese NER datasets:
- Weibo dataset: Primarily from social media, containing entities like Person, Location, Organization, and Geopolitic.
- Resume dataset: Primarily from Sina Finance, used for named entity recognition.
Key Insights and Ideas
- Radicals Matter for Chinese NER: The inclusion of character radical information significantly enhances semantic understanding, proving beneficial for Chinese NER tasks where character structure is rich in meaning.
- Dual FFN Boosts Encoding: The double Feed-Forward Neural Network structure within the Transformer encoder provides a more robust feature encoding mechanism compared to single FFNs, leading to better performance.
- Fusion is Key: Combining character, word, and radical features into a unified representation before feeding it into the Transformer encoder is crucial for MFT's success.
- Domain-Specific Datasets are Essential: The creation of a dedicated aerospace NER dataset was necessary to train and evaluate models effectively for this specialized domain, as general-purpose datasets may not capture the nuances of aerospace terminology.
- MFT's Attention is Broader: Visualizations show that MFT's self-attention mechanism has a broader focus than FLAT, allowing it to capture more semantic features.
Key Results
- On the newly created aerospace dataset, MFT achieved an F1 score of 86.10%.
- MFT outperformed baseline models like FLAT (F1: 80.24%) and Lattice-LSTM (F1: 71.33%) on the aerospace dataset.
- When pre-trained with BERT, MFT achieved an even higher F1 score of 86.10% on the aerospace dataset, outperforming FLAT + BERT (F1: 85.41%).
- On the Weibo dataset, MFT achieved an F1 score of 64.38%.
- On the Resume dataset, MFT achieved an F1 score of 95.78%.
- Ablation studies confirmed that both the Radical Feature Module and the Double FFN contribute positively to MFT's performance across all tested datasets. Removing either component led to a drop in F1 scores.
Limitations and Future Work
- Dataset Size: While 30,000 sentences is a good start, larger datasets could further improve model performance.
- Entity Type Distribution: The paper notes that the AASLS entity type had the lowest recognition rate, which correlated with it having the fewest entities in the dataset. This highlights the impact of data imbalance.
- Future Work: The authors suggest incorporating more diverse Chinese features, such as pronunciation and graphics of characters, for a multimodal approach. However, they acknowledge that this could increase model complexity and parameters, necessitating feature filtering for efficiency. They also propose exploring different FFN weight connection methods.
Summary updated