A Dataset and Evaluation Framework for Complex Geographical Description Parsing

Egoitz Laparra, Steven Bethard

Introduction

Traditional geoparsing typically involves toponym resolution—mapping a specific place name like "Catania" to a single point or polygon in a geographic database. However, natural language often describes locations through complex spatial relationships, such as "a region 10km north of the city" or "the area between the river and the highway." These compositional descriptions are critical for interpreting environmental reports or humanitarian data where named entities may be absent. This paper introduces a large-scale dataset and evaluation framework designed to move the field toward "complex geoparsing," the task of translating natural language descriptions into precise geometric shapes.

Example of complex geoparsing: translating a textual description of a location relative to reference towns into a specific geographic region.

Methodology and Data Acquisition

The researchers developed an automated pipeline to create a massive corpus of text-to-geometry pairs.

  1. Data Mining: The system extracts paragraphs from Wikipedia containing links to other locations and locative terms (e.g., north, between, close).
  2. Geometry Mapping: By linking Wikipedia entries to Wikidata and OpenStreetMap (OSM), the pipeline retrieves the actual geometries (polygons or linestrings) for both the target location and the mentioned reference points.
  3. Baseline Models: The study evaluates two approaches:
  4. Non-linguistic: Geometric operations like the Union or Intersection of all mentioned reference shapes.
  5. Linguistic: A semantic parser using a synchronous grammar that maps English phrases to geometric functions (e.g., "between" triggers a REGIONBETWEEN operation).

The Dataset

The resulting resource consists of two components:

  • Uncurated Set: A large-scale training corpus of 360,187 automatically extracted descriptions. While containing some noise, it provides the volume necessary for modern machine learning.
  • Curated Test Set: A high-quality benchmark of 1,000 manually reviewed examples, ensuring that descriptions are truly compositional and that all necessary reference geometries are present.

Key Insights and Results

The evaluation utilized area-overlap metrics (Precision, Recall, and F1) under strict and relaxed criteria.

The Importance of Linguistic Parsing

The linguistic baseline (F1: 0.221) significantly outperformed the best non-linguistic baseline (F1: 0.102). This highlights that complex geoparsing is fundamentally a semantic parsing task; simple geometric combinations of mentioned entities are insufficient to capture spatial logic. However, the grammar-based model was brittle, producing predictions for only 52.8% of cases.

The "Scale Trap" and Spatial Logic

A significant challenge identified is the "scale trap," where a model identifies a region that contains the target but is orders of magnitude too large (e.g., returning all of "northern Sydney" for a specific suburb). Furthermore, the study notes that spatial operators are context-dependent; the geometric interpretation of "between" varies significantly depending on whether the references are cities, rivers, or buildings.

Conclusion

While rule-based synchronous grammars provide a strong starting point, they struggle with the diversity of natural language. The introduction of this 360,000-example dataset paves the way for end-to-end neural models capable of learning nuanced spatial relationships directly from data. Future work will focus on improving the robustness of these models to handle the inherent ambiguity and complexity of human geographic descriptions.

Summary updated

Abstract

Much previous work on geoparsing has focused on identifying and resolving individual toponyms in text like Adrano, S.Maria di Licodia or Catania. However, geographical locations occur not only as individual toponyms, but also as compositions of reference geolocations joined and modified by connectives, e.g., “. . . between the towns of Adrano and S.Maria di Licodia, 32 kilometres northwest of Catania”. Ideally, a geoparser should be able to take such text, and the geographical shapes of the toponyms referenced within it, and parse these into a geographical shape, formed by a set of coordinates, that represents the location described. But creating a dataset for this complex geoparsing task is difficult and, if done manually, would require a huge amount of effort to annotate the geographical shapes of not only the geolocation described but also the reference toponyms. We present an approach that automates most of the process by combining Wikipedia and OpenStreetMap. As a result, we have gathered a collection of 360,187 uncurated complex geolocation descriptions, from which we have manually curated 1,000 examples intended to be used as a test set. To accompany the data, we define a new geoparsing evaluation framework along with a scoring methodology and a set of baselines.
BibTeX
@inproceedings{laparra2020dataset,
  author = {Egoitz Laparra and Steven Bethard},
  title = {{A Dataset and Evaluation Framework for Complex Geographical Description Parsing}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.81},
}
Figure