An Integrated Search System for Korea Weather Data

Jinkyung Jo, Dayeon Ki, Soyoung Yoon, Minjoon SeoEMNLP 2023 · OpenReview

TL;DR

This paper introduces WEATHERSEARCH, a novel system designed to simplify access to Korea's vast weather data by allowing users to query complex databases using natural language. By meticulously collecting expert-defined templates and leveraging advanced language models, they've created a system that significantly reduces the technical barrier for meteorological professionals.

What Problem Does This Paper Address?

Weather forecasting relies on massive datasets, but accessing this information is often cumbersome. For instance, the Korea Meteorological Administration (KMA) maintains a comprehensive system (COMIS) with a hierarchical, click-heavy interface, and a separate database requiring SQL proficiency. This complexity leads experts to stick to familiar data sources, limiting their ability to leverage the full spectrum of available information. This is particularly challenging in South Korea due to its diverse geography, which leads to complex weather patterns. The paper aims to bridge this gap by enabling natural language queries for both structured database retrieval (SQL) and unstructured web content (URLs).

Methods

The core of WEATHERSEARCH is built upon fine-tuning the mT5 (multilingual text-to-text transfer transformer) model. The novelty lies in the data collection and preparation pipeline, which heavily incorporates domain expertise:

  • Expert Surveys and Interviews: To understand how meteorological experts actually query data, the researchers conducted extensive surveys and interviews with 24 experts. This informed the creation of question templates and corresponding SQL query templates.
  • Template-Based Data Generation: Instead of manually creating millions of query-SQL pairs, they developed templates with placeholders (e.g., {date}, {region}, {attribute}). These templates were then populated with actual values and time expressions using techniques like template filling and data augmentation. This approach allowed for the efficient generation of a large dataset with minimal human labor.
  • Constrained Decoding: To improve the accuracy and adherence to specific formats in the generated SQL queries, they employed constrained decoding. This technique guides the language model's output using a prefix tree, ensuring the generated queries are syntactically correct and semantically relevant.
  • URL Dataset Creation: For accessing less structured information, they crawled KMA's COMIS website, collected URLs, and annotated them with relevant keywords. This dataset was refined through expert feedback and iterative improvements, including adding noise to keywords for robustness and using a synonym dictionary.

Datasets

The paper introduces two domain-specific datasets for training and evaluation:

  • SQL Dataset: This dataset consists of pairs of natural language queries and their corresponding SQL queries. It was generated using the template-based approach described above, resulting in approximately 4 million data points.
  • URL Dataset: This dataset comprises pairs of natural language keywords and corresponding URLs from the COMIS website. It was collected through crawling and refined with expert input, containing over 2 million records.

Key Insights and Ideas

  • Expert-Centric Data Collection is Crucial: The success of WEATHERSEARCH hinges on its deep integration of expert knowledge. Simply applying general-purpose NLP models to weather data wouldn't capture the nuances of expert queries and data structures.
  • Template-Based Generation Scales Efficiently: Manually creating large-scale, domain-specific datasets is often infeasible. The template-filling approach, combined with expert input, offers a scalable and cost-effective solution for generating millions of training examples.
  • Constrained Decoding Boosts SQL Accuracy: For structured query generation like SQL, simply relying on the model's raw output can lead to errors. Constrained decoding significantly improves the precision and reliability of generated SQL queries.
  • Robustness to Noisy Queries is Essential: Real-world search systems must handle imperfect user input. The URL dataset's preprocessing, including adding noise to keywords, reflects this need for robustness.

Key Results

The WEATHERSEARCH system, fine-tuned on mT5, achieved strong performance:

  • SQL Evaluation: An Exact Match (EM) score of 0.99 was achieved, indicating high accuracy in generating correct SQL queries.
  • URL Evaluation: The system achieved an average MRR (Mean Reciprocal Rank) of 0.66 and an average Recall of 0.82 with constrained decoding. Without constrained decoding, these figures were 0.64 MRR and 0.81 Recall, respectively. The slight dip in performance on the URL dataset compared to SQL is attributed to the inherent noise in keyword-based search.

Limitations and Future Work

  • Limitations:
    • Grammatical Inconsistencies: The template-based generation might introduce minor grammatical errors or inconsistencies, though these were found to not significantly impact model performance.
  • Evaluation Gap: The current evaluation datasets were created similarly to training datasets, potentially leading to results that are more favorable than real-world user experience.
  • Data Disclosure: Due to security concerns, the original training data cannot be publicly disclosed, limiting reproducibility for external researchers.
  • Future Work:
    • Human Evaluation: The authors plan to conduct human evaluations to qualitatively assess the system's performance and user experience.
  • Data Masking for Release: Exploring data masking techniques to potentially release the training data while preserving security.
  • Broader Applicability: The methodology is presented as a guideline for designing similar systems for different regions or languages, suggesting potential for generalization.

Summary updated

Abstract

We introduce WeatherSearch, an integrated search system deployed at the Korea Meteorological Administration (KMA). WeatherSearch enables users to retrieve all the relevant data for weather forecasting from a massive weather database with simple natural language queries. We carefully design and conduct multiple expert surveys and interviews for template creation and apply data augmentation techniques including template filling to collect 4 million data points with minimal human labors. We then finetune mT5 on the collected dataset and achieve an average MRR of 0.66 and an average Recall of 0.82. We also discuss weather-data-specific characteristics that should be taken into account for creating such a system. We hope our paper serves as a simple and effective guideline for those designing similar systems in other regions of the world.
BibTeX
@inproceedings{jo2023integrated,
  author = {Jinkyung Jo and Dayeon Ki and Soyoung Yoon and Minjoon Seo},
  title = {{An Integrated Search System for Korea Weather Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=9kmnkWgIpr},
}
Figure