HeGeL: A Novel Dataset for Geo-Location from Hebrew Text

Tzuf Paz-Argaman, Tal Bauman, Itai Mondshine, Itzhak Omer, Sagi Dalyot, Reut TsarfatyACL 2023 · DOI

Introduction

Textual geolocation is the task of mapping a natural language description of a location to its corresponding geographic coordinates. While often simplified as a named entity lookup (e.g., matching "London" to a specific point), human communication frequently relies on complex spatial reasoning and relative descriptions. For instance, a person might describe a location as being "two blocks east of the pharmacy" or "near the sea at the end of the street."

Existing geolocation datasets suffer from several limitations: they are predominantly in English, often operate at a low resolution (city-level), and rarely require sophisticated spatial reasoning. Furthermore, Hebrew presents unique challenges as a morphologically rich language (MRL), where word forms vary significantly based on grammatical context, complicating standard NLP processing. To address these gaps, the HeGeL (Hebrew Geo-Location) dataset was developed.

Methodology: The Treasure Hunt Game

To collect high-quality, naturalistic spatial descriptions, the researchers designed a "Treasure Hunt" crowdsourcing task. In this setup, an "instructor" is shown a specific location on a map and tasked with describing its position to a "follower." Crucially, the instructor is forbidden from using the exact address or the name of the place itself, forcing the use of relative spatial markers and landmarks.

The rules of the Treasure Hunt game used to collect the HeGeL dataset.

The resulting descriptions were then validated by "followers" who attempted to identify the location on an interactive OpenStreetMap (OSM) interface. This process ensured that the descriptions were both natural and functional for human navigation.

The HeGeL Dataset

The HeGeL dataset comprises 5,649 validated Hebrew descriptions covering 167 unique locations across Tel Aviv, Haifa, and Jerusalem. The dataset is rich in diverse spatial phenomena, requiring models to understand various types of city elements such as nodes (squares), edges (coastlines), and paths (streets).

Linguistic and spatial phenomena present in the HeGeL dataset, categorized by established cognitive and spatial frameworks.

As shown in the analysis of the corpus, descriptions frequently utilize cardinal directions, references to unique entities, and different levels of spatial knowledge, ranging from route-based instructions to survey-level overviews. This diversity highlights the complexity of the task beyond simple keyword matching.

Experimental Results and the Human-Machine Gap

The researchers evaluated several modeling approaches against the HeGeL dataset, including Google Maps API, an Oracle NER baseline, and a neural Dual-Encoder architecture using AlephBERT. The primary metric for evaluation was Task Completion (TC) accuracy, defined as the percentage of descriptions resolved to within 300 meters of the target.

Model Mean Error (m) TC Accuracy (%)
Human (Follower) 553 70.81%
Oracle NER 2,373 37.79%
Google Maps API 2,811 27.66%
Dual-Encoder (80% Few-Shot) 983 15.7%
Dual-Encoder (Zero-Shot) 2,727 2.37%

The results revealed a significant performance gap. The Dual-Encoder's poor zero-shot performance (2.37%) highlights the difficulty of generalizing spatial reasoning across different cities. Models trained on Tel Aviv often failed in Jerusalem due to "Out-of-Vocabulary" (OOV) issues stemming from city-specific landmarks and Hebrew's morphological complexity.

Conclusion

The findings suggest that current NLP models lack a robust "cognitive map" of the world. While humans can easily synthesize relative descriptions into a mental spatial representation, models struggle with the multi-entity relationships required for fine-grained geolocation. Future research should focus on improving zero-shot generalization, perhaps through data augmentation techniques that generate synthetic grounded descriptions for new urban environments.

Summary updated

BibTeX
@inproceedings{pazargaman2023hegel,
  author = {Tzuf Paz-Argaman and Tal Bauman and Itai Mondshine and Itzhak Omer and Sagi Dalyot and Reut Tsarfaty},
  title = {{HeGeL: A Novel Dataset for Geo-Location from Hebrew Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.460},
}
Figure