Geographic Named Entity Recognition by Employing Natural Language Processing and an Improved BERT Model

Liufeng Tao, Zhong Xie, Dexin Xu, Kai Ma, Qinjun Qiu, Shengyong Pan, Bo HuangISPRS Int. J. Geo Inf 2022 · DOI

TL;DR

This paper tackles the challenge of accurately identifying Chinese place names (toponyms) in social media text, which is often messy and informal. They propose a novel hybrid deep learning model, ALBERT-BiLSTM-CRF, that leverages a pre-trained ALBERT model for better word understanding and a BiLSTM-CRF layer for sequence labeling, achieving state-of-the-art results on a new benchmark dataset.

What Problem Does This Paper Address?

Geographic Named Entity Recognition (GeoNER) is crucial for tasks like disaster response, urban planning, and information retrieval. However, recognizing toponyms in social media presents a significant challenge due to informal language, abbreviations, misspellings, and complex sentence structures. Existing methods often struggle with these linguistic irregularities, especially for Chinese, which has a vast number of place names and unique naming conventions. This paper specifically aims to improve Chinese toponym recognition in social media by addressing these issues.

Methods

The core of the proposed solution is a hybrid deep learning model called ALBERT-BiLSTM-CRF. Here's a breakdown:

  • Input Layer: Processes individual words from the input text.
  • ALBERT Layer: This is a key innovation. Instead of using standard BERT, the authors employ a pre-trained ALBERT (A Lite BERT) model. ALBERT is a more lightweight version of BERT that achieves comparable performance with fewer parameters. Crucially, they fine-tune ALBERT on a corpus derived from social media messages, allowing it to better capture the nuances of informal language and vernacular terms common in these platforms. This layer generates enhanced word embeddings that represent dynamic, contextualized semantics.
  • BiLSTM Layer: A Bidirectional Long Short-Term Memory (BiLSTM) layer is used to capture contextual semantic features from the text in both forward and backward directions. This helps in understanding the relationships between words in a sequence, especially for longer place names.
  • CRF Layer: A Conditional Random Field (CRF) layer is employed to output the optimal sequence of labels. CRF models are effective for sequence labeling tasks as they consider dependencies between adjacent labels, improving the accuracy of entity boundary detection.

The model uses a BIO (Begin, Inside, Outside) tagging scheme for labeling entities.

Datasets

The authors introduce a new, large-scale, self-annotated corpus for Chinese geographic named entity recognition called TPCNER.

  • TPCNER: This corpus was built using data from Baidu Encyclopedia and the Chinese Encyclopedia of Chinese Geography, containing approximately 2 million words. It was further extended and annotated to include 7 fine-grained entity categories relevant to the geographic domain (e.g., Water System, Residential land and facilities, Transportation, Pipelines, Boundaries/Regions/Other Areas, Landforms, Organization). TPCNER contains 64,063 annotated samples.
  • Public Datasets: For evaluation and comparison, the study also utilized three publicly available datasets: Boson, MSRA, and RenMinRiBao. These datasets are commonly used for Chinese Named Entity Recognition tasks.

Key Insights and Ideas

  • Social Media Specific Pre-training is Crucial: Standard pre-trained models like BERT, while powerful, might not fully capture the linguistic peculiarities of social media. Fine-tuning ALBERT on social media data significantly improved its ability to understand informal language and vernacular terms, leading to better toponym recognition.
  • Hybrid Models Excel: Combining the strengths of pre-trained language models (ALBERT for semantic understanding) with sequence modeling architectures (BiLSTM for context and CRF for labeling) proves highly effective for GeoNER.
  • The Challenge of Long and Ambiguous Toponyms: The paper highlights that long, complex, or ambiguously named geographic entities remain a significant challenge. The proposed model's ability to handle these better is a key contribution.
  • New Benchmark Dataset is Valuable: The creation of TPCNER, a large and fine-grained corpus for Chinese GeoNER, is a significant contribution that can facilitate future research in this area.

Key Results

The proposed ALBERTours-BiLSTM-CRF model achieved state-of-the-art performance across various datasets:

  • On TPCNER: The model achieved an impressive 97.8% accuracy, 96.1% precision, 96.2% recall, and 96.1% F1-score. This significantly outperformed seven baseline models.
  • On Public Datasets (Boson, MSRA, RenMinRiBao): The model consistently achieved the highest F1-scores, often exceeding 95%, demonstrating its robustness and generalizability.
  • Comparison with Baselines: The ALBERT-BiLSTM-CRF model showed substantial improvements over previous methods, including a notable 7.8% improvement over the original ALBERT model when fine-tuned on their social media corpus.

Limitations and Future Work

  • Data Imbalance: The authors acknowledge that data imbalance in certain entity categories (e.g., TRA) can lead to lower precision for those categories.
  • Complex and Ambiguous Names: While improved, the model still faces challenges with extremely long, complex, or ambiguously named geographic entities, especially when they are not frequently seen in the training data.
  • Future Work: The authors plan to:
  • Include more features and refine the model's weights.
  • Make the TPCNER corpus publicly available.
  • Continue to explore more sensible modifications to feature weights.

Summary updated

BibTeX
@article{tao2022geographic,
  author = {Liufeng Tao and Zhong Xie and Dexin Xu and Kai Ma and Qinjun Qiu and Shengyong Pan and Bo Huang},
  title = {{Geographic Named Entity Recognition by Employing Natural Language Processing and an Improved BERT Model}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2022},
  doi = {10.3390/ijgi11120598},
}
Figure