geoGAT: Graph Model Based on Attention Mechanism for Geographic Text Classification

Weipeng Jing, Xianyang Song, Donglin Di, Houbing Song

TL;DR

The authors propose geoGAT, a Graph Attention Network (GAT) model specifically designed to classify Chinese text containing geographical information. By representing text as a graph and using an attention mechanism to weight the importance of neighboring nodes, the model achieves a 95% Macro-F score, significantly outperforming traditional convolutional and recurrent neural networks.

What Problem Does This Paper Address?

Extracting geographical information from the massive influx of online text (news, social media) is vital for applications like disaster relief, urban planning, and transportation. However, identifying which texts actually contain relevant geographical entities—and classifying them correctly—is a "time-consuming but valuable" task.

In the context of Chinese natural language processing (NLP), this is particularly challenging. Traditional models like CNNs and LSTMs often struggle with the non-Euclidean structure of text-based graphs and the nuances of Chinese word segmentation. While Graph Convolutional Networks (GCNs) have been used to bridge this gap, they treat all neighboring nodes (words) with equal importance, which limits their ability to capture the specific spatial and contextual correlations necessary for geographic classification.

Methods

The core innovation is the application of Graph Attention Networks (GAT) to the geographic text classification task. The process follows three main steps:

  1. Graph Construction: The authors convert the entire dataset into a large, heterogeneous graph.
  2. Nodes: Represent both individual documents (sequences) and unique words.
  3. Edges: The weight between word nodes is calculated using Pointwise Mutual Information (PMI), while the weight between a word node and a document node is determined by TF-IDF.
  4. Attention Mechanism: Unlike a standard GCN that uses a standardized operation to aggregate neighbors, geoGAT uses an attention mechanism to learn the relative importance of neighboring nodes. This allows the model to "focus" on specific keywords that signal geographical relevance.
  5. Multi-head Attention: To stabilize the learning process and expand the model's capacity, the authors employ multi-head attention. Each "head" learns features in different representation spaces, which are then averaged to produce the final output.

Datasets

The authors constructed a new Chinese dataset by extracting and filtering data from four major sources:

  • ByteDance_News: Short news samples.
  • THUCNews_Chinese_Text: Historical news data from Sina News.
  • Fudan University Chinese Text: Short documents across 20 categories.
  • SogouCA: News data from various domestic and overseas channels.

The resulting corpus contains 55,000 pieces of data divided into five categories: Geography, Games, Cars, Economy, and Entertainment. The "Geography" category was strictly curated to ensure texts contained clear, non-virtual geographical locations.

Key Insights and Ideas

  • GAT vs. GCN: The "aha moment" here is that not all words in a neighborhood are equally informative. By weighting neighbors, GAT compensates for GCN’s tendency to "blur" features across the graph, making it much better at capturing spatial information correlation.
  • Robustness to Segmentation: A major finding was that graph-based models (geoGAT and TextGCN) are less sensitive to Chinese word segmentation issues. While CNNs and RNNs performed significantly better with character-level segmentation than word-level (due to data sparsity and noise in segmentation tools), the graph models maintained high accuracy regardless of the segmentation method.
  • Global Context: By building a graph of the entire corpus, the model learns global word co-occurrence patterns that are often ignored by local window-based models like CNNs or LSTMs.

Key Results

The geoGAT model demonstrated superior performance across several metrics compared to baselines like TextGCN, BiLSTM-Att, and DPCNN:

  • Macro-F Score: Reached 95.20%, the highest among all tested models.
  • Precision and Recall: Achieved 98.34% Precision and 97.87% Recall on the specific geographic classification task.
  • Efficiency: While increasing the number of attention heads increases training time per epoch, the authors found that multi-head attention reached convergence faster (around 200 epochs) than simpler self-attention mechanisms.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Ambiguity: Some texts contain ambiguous toponyms (place names) that can lead to misidentification. The current model struggles with complex toponym resolution (e.g., distinguishing between a person's name and a place name when they are identical).
  • Graph Depth: The current model uses a single-layer graph structure. The authors suggest that extending this to a multi-layer network could provide better computing effects and deeper feature extraction.
  • Entity Disambiguation: Future work will focus on integrating entity disambiguation to improve accuracy and extracting even more detailed geolocation information from the text.

Summary updated

Abstract

In the area of geographic information processing, there are few researches on geographic text classification. However, the application of this task in Chinese is relatively rare. In our work, we intend to implement a method to extract text containing geographical entities from a large number of network texts. The geographic information in these texts is of great practical significance to transportation, urban and rural planning, disaster relief, and other fields. We use the method of graph convolutional neural network with attention mechanism to achieve this function. Graph attention networks (GAT) is an improvement of graph convolutional neural networks (GCN). Compared with GCN, the advantage of GAT is that the attention mechanism is proposed to weight the sum of the characteristics of adjacent vertices. In addition, We construct a Chinese dataset containing geographical classification from multiple datasets of Chinese text classification. The Macro-F Score of the geoGAT we used reached 95% on the new Chinese dataset.
BibTeX
@article{jing2021geogat,
  author = {Weipeng Jing and Xianyang Song and Donglin Di and Houbing Song},
  title = {{geoGAT: Graph Model Based on Attention Mechanism for Geographic Text Classification}},
  journal = {ACM Trans. Asian Low Resour. Lang. Inf. Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=K-57ZrD9dWU},
}
Figure