Meteorologists and Students: A resource for language grounding of geographical descriptors

Alejandro Ramos-Soto, Ehud Reiter, Kees van Deemter, José Maria Alonso, Albert Gatt

TL;DR

This paper introduces a novel dataset designed to improve how computers understand and generate language describing geographical locations. By collecting drawings of geographical regions from both expert meteorologists and teenage students, the authors provide a unique resource for training and evaluating systems that ground language to spatial data, a crucial step for applications like weather forecasting and navigation.

What Problem Does This Paper Address?

The core challenge this paper tackles is "language grounding" – specifically, how to connect abstract linguistic descriptions of geographical areas (like "Northern Galicia" or "coastal regions") to concrete spatial data (like polygons on a map). This is a fundamental problem for Natural Language Generation (NLG) systems, especially those that need to produce accurate and meaningful descriptions based on data, such as weather reports. Existing methods often rely on heuristics or parallel corpora, but the authors argue for the need for more direct human input to map linguistic expressions to their spatial referents, especially considering the inherent vagueness and subjectivity in geographical descriptions. They aim to create models that capture this fuzziness, moving beyond simple binary mappings.

Methods

The paper's primary contribution is the creation and description of a new dataset. This dataset was collected through two surveys where participants were asked to draw polygons on a map representing given geographical descriptors.

  • Survey 1 (Students): Involved 99 teenage students from a high school. They were presented with 7 geographical descriptors. The purpose was to gather a large number of responses for evaluating modeling algorithms.
  • Survey 2 (Meteorologists): Involved 8 expert meteorologists from MeteoGalicia. They were given a list of 24 descriptors, including cardinal directions, proper names, and combinations of directions (e.g., "northwest"). The goal was to create fuzzy models for a future NLG system in the weather domain.

The data consists of these drawn polygons, which are then processed and visualized, often as contour maps showing the density or overlap of responses for a given descriptor. The authors also provide Python scripts for data loading and visualization.

Datasets

The paper introduces a new dataset comprising two parts:

  1. Student Dataset: Collected from 99 teenage students, containing responses for 7 geographical descriptors.
  2. Expert Dataset: Collected from 8 meteorologists, containing responses for 24 geographical descriptors.

Both datasets include polygon drawings on a Mercator-projected map of Galicia, Spain. The descriptors are a mix of cardinal directions, regions, and specific geographical features. The authors note that 6 descriptors are common to both groups, allowing for some comparative analysis. The data is available under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License and is provided in SQLite and JSON formats.

Key Insights and Ideas

  • Expert vs. Student Consistency: Experts are significantly more consistent in their drawings than students, likely due to their professional training and the smaller number of participants in the expert group. This highlights the potential for expert data to provide more reliable grounding for critical applications.
  • Fuzzy Semantics of Geographical Terms: The overlap in drawings, even for antonyms like "north" and "south," suggests that geographical descriptors often have fuzzy boundaries. A single point on the map might plausibly belong to multiple descriptors, which is crucial for building robust NLG systems that can handle ambiguity.
  • Potential for Confusing Descriptors: The study identified instances where descriptors were confused, particularly "west" and "east" by students, possibly due to phonetic similarity in Spanish ("oeste" and "este"). This points to the need for careful descriptor selection and potential disambiguation strategies in NLG.
  • Value for Multiple Research Areas: Beyond geographical referring expression generation, the dataset is deemed useful for testing algorithms that create geographical models, analyzing expert-nonexpert differences, and studying combinations of cardinal directions.

Key Results

  • Qualitative Differences: Visualizations clearly show experts drawing more concentrated polygons, indicating higher agreement. Students sometimes provided drawings with unexpected shapes, sizes, or locations.
  • Descriptor Overlap: For "Northern Galicia" and "Southern Galicia," there's a small area of overlap where points could belong to both, even though "north" and "south" are natural antonyms.
  • Descriptor Confusion: Some students confused "west" and "east" descriptors. A few experts also confused "Northeastern Galicia" with "Northwestern Galicia."
  • Data Availability: The resource is made publicly available, enabling further research.

Limitations and Future Work

  • Geographical Specificity: The current dataset is focused on the geography of Galicia, Spain. The authors suggest that future work should involve gathering similar data for other regions worldwide to enable generalization of geographical descriptor semantics.
  • Data Filtering: The authors acknowledge that some answers might be erroneous or misleading and suggest that users may need to filter data based on criteria like centroid deviation or size.
  • Descriptor Confusion: The paper highlights issues like descriptor confusion (e.g., west/east) as an inconvenience that users of the dataset should be aware of.
  • Future Data Collection: The authors propose performing similar data gathering tasks for diverse regions to create a more comprehensive resource and potentially establish a shared task for advancing research in this area.

Summary updated

Abstract

We present a data resource which can be useful for research purposes on language grounding tasks in the context of geographical referring expression generation. The resource is composed of two data sets that encompass 25 different geographical descriptors and a set of associated graphical representations, drawn as polygons on a map by two groups of human subjects: teenage students and expert meteorologists.
BibTeX
@inproceedings{ramossoto2018meteorologists,
  author = {Alejandro Ramos-Soto and Ehud Reiter and Kees van Deemter and José Maria Alonso and Albert Gatt},
  title = {{Meteorologists and Students: A resource for language grounding of geographical descriptors}},
  booktitle = {INLG},
  year = {2018},
  url = {https://openreview.net/forum?id=wkKyei51FT},
}
Figure