A Coherent Unsupervised Model for Toponym Resolution

Ehsan Kamalloo, Davood Rafiei

TL;DR

Researchers have developed a new unsupervised model called Context-Hierarchy Fusion (CHF) that accurately maps place names in text (toponyms) to their precise geographic coordinates without requiring any labeled training data. By combining document-level context with spatial hierarchy rules, the model outperforms major commercial APIs and remains robust even when encountering locations it has never seen before.

What Problem Does This Paper Address?

When you read the word "Paris" in a news article, how do you know if it refers to the capital of France or a small town in Texas? This task is known as Toponym Resolution. While it sounds simple for humans, it is a nightmare for machines because of extreme ambiguity—the GeoNames database lists 97 different interpretations for "Paris" alone.

Existing solutions usually fall into two camps: supervised models that require massive amounts of hand-annotated data (which is scarce and expensive) and unsupervised models that often rely on simple heuristics. The gap the authors identify is that supervised models fail to generalize; if a model is trained on news from New York, it struggles to resolve place names in a document about rural Alberta. The authors sought a "coherent" model that relies only on the text itself and a standard geographic gazetteer.

Methods

The authors propose a hybrid approach called Context-Hierarchy Fusion (CHF), which merges two distinct unsupervised strategies:

  1. Context-Bound Hypotheses (CBH): This is a probabilistic model based on two ideas:
  2. Geo-centre inheritance: A document usually has a "geographic scope." If an article mentions several places in Ontario, a mention of "London" is likely London, Ontario.
  3. Near-location: Toponyms mentioned close to each other in text are often geographically close (e.g., "London’s Heathrow").
  4. Spatial-Hierarchy Sets (SHS): This model treats resolution as a "conflict-free set cover" problem. It looks for the smallest number of geographic "parent" regions (like counties or states) that can explain all the "child" locations mentioned in the text. This enforces the "one-sense-per-referent" rule—the idea that within one document, "Paris" shouldn't refer to two different cities.

The Fusion: The final CHF model uses a confidence threshold ($\tau$). If the probabilistic CBH model is very sure of an answer, it takes it. If not, it falls back on the structural logic of the SHS model to ensure spatial minimality.

Datasets

The model was evaluated on three distinct news corpora:

  • TR-News: A new dataset collected by the authors specifically to include "less dominant" interpretations (e.g., Edmonton, England instead of Edmonton, Canada) to test model robustness.
  • Local-Global Lexicon (LGL): A collection of local news articles that are notoriously difficult because they contain highly ambiguous, small-town toponyms.
  • CLUST: A large dataset containing over 13,000 documents from global and local news sources.

Key Insights and Ideas

  • The Generalization Trap: The authors found that supervised models are "brittle." When a test dataset contains more than 40% toponyms that weren't in the training set, the unsupervised CHF model actually outperforms the state-of-the-art supervised ones.
  • Minimality is Key: The "aha moment" here is that toponym resolution isn't just about individual words; it's about the coherence of the whole document. By assuming that an author is trying to be efficient and geographically consistent, the model can eliminate 99% of false interpretations.
  • Commercial Limitations: Surprisingly, the authors' unsupervised model outperformed heavyweights like Google Cloud Natural Language API and Reuters OpenCalais across all three datasets, suggesting that even "black-box" commercial models struggle with the long tail of ambiguous geographic mentions.

Key Results

  • Accuracy: On the TR-News dataset, CHF achieved a Precision of 79.3%, significantly higher than the previous state-of-the-art unsupervised model, TopoCluster (68.8%).
  • Error Distance: CHF maintained a low mean error distance (around 800-900km), which is impressive given that a single wrong "country-level" guess can throw an error off by thousands of kilometers.
  • Robustness: In head-to-head tests against commercial products on the LGL dataset (the hardest one), CHF's F1-score was 52.9%, while OpenCalais trailed at 42.1%.

Limitations and Future Work

The authors note a few areas for improvement:

  • Short vs. Long Text: The model relies on having enough "clues" in a document. Very short texts (like tweets) might not provide enough context for the spatial hierarchy logic to kick in.
  • Bounding Boxes: Currently, the model often resolves to a "centroid" (a single point). The authors suggest that using actual bounding boxes for regions could improve accuracy for large areas like states or countries.
  • Hybrid Potential: While they proved unsupervised models are better for "unseen" data, a future direction involves a smarter "switching" mechanism that uses supervised learning for common places and unsupervised logic for rare ones.

Summary updated

Abstract

Toponym Resolution, the task of assigning a location mention in a document to a geographic referent (i.e., latitude/longitude), plays a pivotal role in analyzing location-aware content. However, the ambiguities of natural language and a huge number of possible interpretations for toponyms constitute insurmountable hurdles for this task. In this paper, we study the problem of toponym resolution with no additional information other than a gazetteer and no training data. We demonstrate that a dearth of large enough annotated data makes supervised methods less capable of generalizing. Our proposed method estimates the geographic scope of documents and leverages the connections between nearby place names as evidence to resolve toponyms. We explore the interactions between multiple interpretations of mentions and the relationships between different toponyms in a document to build a model that finds the most coherent resolution. Our model is evaluated on three news corpora, two from the literature and one collected and annotated by us; then, we compare our methods to the state-of-the-art unsupervised and supervised techniques. We also examine three commercial products including Reuters OpenCalais, Yahoo! YQL Placemaker, and Google Cloud Natural Language API. The evaluation shows that our method outperforms the unsupervised technique as well as Reuters OpenCalais and Google Cloud Natural Language API on all three corpora; also, our method shows a performance close to that of the state-of-the-art supervised method and outperforms it when the test data has 40% or more toponyms that are not seen in the training data.
BibTeX
@inproceedings{kamalloo2018coherent,
  author = {Ehsan Kamalloo and Davood Rafiei},
  title = {{A Coherent Unsupervised Model for Toponym Resolution}},
  booktitle = {The Web Conference},
  year = {2018},
  doi = {10.1145/3178876.3186027},
}

Loading related content...

Figure