TL;DR
This paper tackles the challenge of automatically identifying the correct geographical entity a place name (toponym) refers to in news text. They developed an unsupervised machine learning approach that, after training on automatically annotated data, achieves 78.5% accuracy in disambiguating toponyms in human-annotated news, outperforming previous methods.
What Problem Does This Paper Address?
In natural language processing, accurately identifying and classifying place names (toponyms) is crucial for applications like Geographic Information Systems (GIS). A single toponym, like "Washington," can refer to multiple entities (e.g., the city, the state, or even a person). This ambiguity makes it difficult to ground information on a map or perform spatial reasoning. While distinguishing toponyms from other named entities is well-studied, resolving which specific place a toponym refers to, especially when multiple options exist in a gazetteer, has received less attention. This paper aims to bridge this gap by developing a method to automatically disambiguate toponyms in news text.
Methods
The authors propose a corpus-based, unsupervised machine learning approach. Their method involves:
- Gazetteer Merging: Combining information from two publicly available gazetteers (USGS Concise Gazetteer and a merged ATLAS/GAZ gazetteer) to create a richer knowledge source.
- Ambiguity Quantification: Measuring the degree of ambiguity for toponyms found in a news corpus against their gazetteer.
- Unsupervised Rule Development:
- Automatic Tagging: Toponyms in a large news corpus (MAC1) were automatically tagged with information from the gazetteer.
- Preference Heuristics: For ambiguous toponyms, preference heuristics (e.g., preferring the predominant sense in a corpus, or a more specific class like "Capital" over "Populated Place") were used to automatically disambiguate them.
- Machine Learning Training: This automatically tagged and disambiguated data was used to train a machine learner.
- Feature Engineering: They explored various features, including those internal to the toponym (e.g., abbreviations, all-caps) and contextual features (words within a window, document-level discourse topic, coreference information).
Datasets
- USGS Concise Gazetteer (GNIS): Used for initial analysis of US toponyms.
- ATLAS and GAZ: Publicly available websites used to create a worldwide gazetteer (WAG).
- English Gigaword (LDC 2003): A large corpus of news articles used for creating machine-annotated corpora.
- MAC1: A month of New York Times articles (6.51 million words) used for initial ambiguity quantification.
- MAC-DEV: Gigaword AFP articles (5.47 million words) used as a development corpus for feature exploration.
- MAC-ML: Gigaword AP Worldwide articles (6.21 million words) used as a machine learning corpus.
- Human Annotated Corpus (HAC): A small, manually annotated corpus (83,872 words) sampled from TimeBank 1.2 and June 2002 New York Times, used for blind testing.
Key Insights and Ideas
- Significant Ambiguity in News: A substantial portion (67.82%) of ambiguous toponyms found in news lacked immediate local discriminators in the text, highlighting the need for sophisticated disambiguation.
- Unsupervised Approach Viability: The paper demonstrates that an unsupervised approach, leveraging readily available gazetteers and large text corpora, can generate sufficient training data for a machine learner to achieve good performance.
- Importance of Discourse-Level Features: The
TagDiscoursefeature, which captures the overall topic of a document, proved critical for disambiguation, indicating that the broader context significantly influences toponym interpretation. - Coreference as a Strategy: The observation that many subsequent mentions of a toponym lack local discriminators suggests that coreference resolution is a valuable strategy for disambiguation.
- Trade-offs in Window Size: While larger windows (±20) improved performance on some metrics, they also led to a drop in accuracy on the same corpus (cross-validation), suggesting a balance is needed.
Key Results
- The developed unsupervised method, when trained on machine-annotated data, achieved 78.5% accuracy on a human-annotated test corpus (HAC).
- When training and testing on machine-annotated data (cross-validation), accuracies were even higher, reaching 88.39% for MAC-ML with a ±3 word window.
- The
TagDiscoursefeature was critical, dropping accuracy by nearly 9 percentage points when ignored. - Combining training data (MAC-ML + MAC-DEV) significantly improved accuracy on the human-annotated corpus (HAC), especially with larger window sizes.
Limitations and Future Work
- Limited Scope of Initial Gazetteer: The initial ambiguity analysis was limited to US toponyms from GNIS, excluding worldwide cities and countries.
- Potential for Spurious Tags: The automatic tagging process could misidentify common words or person names as toponyms.
- LexScan Limitations: The lexical lookup tool had limitations in handling abbreviations and case sensitivity, potentially missing some toponyms.
- Small Human-Annotated Corpus: The performance of the supervised approach was limited by the small size of the human-annotated corpus (HAC).
- Future Work: The authors plan to integrate their disambiguator with other gazetteers and geo-visualization systems, study the effect of different window sizes, and explore combinations with minimally-supervised approaches. They also aim to mitigate data sparseness by clustering terms based on stemming and semantic similarity.
Summary updated