An LLM-Based Inventory Construction Framework of Urban Ground Collapse Events with Spatiotemporal Locations

Yanan Hao, Jin Qi, Xiaowen Ma, Sensen Wu, Renyi Liu, Xiaoyi ZhangISPRS Int. J. Geo Inf 2024 · DOI

TL;DR

This paper introduces a novel framework using Large Language Models (LLMs) to automatically construct an inventory of urban ground collapse (UGC) events from news reports. By leveraging LLMs for event recognition and attribute extraction, the study creates the first open-access UGC inventory for Zhejiang province, China, offering valuable insights into the spatiotemporal patterns of these events.

What Problem Does This Paper Address?

Urban ground collapses (UGCs) are a growing concern, exacerbated by rapid urbanization and extensive underground construction. Understanding the frequency, location, and causes of these events is crucial for effective disaster prevention and management. However, collecting comprehensive data on UGCs is challenging due to their scattered nature and the lack of centralized official records. While news reports offer a rich source of information, manually sifting through vast amounts of text to identify and extract relevant details is time-consuming and labor-intensive. This paper aims to bridge this gap by developing an automated system to construct a structured UGC event inventory from unstructured news data.

Methods

The proposed framework consists of three main modules:

  1. News Report Crawling: This module uses a web crawler to systematically search news portals (specifically Baidu News) using predefined keywords related to ground collapse. It recursively navigates web links to gather comprehensive information, including titles, abstracts, and full article content.
  2. LLM-Aided Event Recognition: This module employs a Large Language Model (LLM), specifically ChatGLM-3b, to identify whether a news report describes a ground collapse event. This involves prompt engineering to guide the LLM, using zero-shot or few-shot learning with examples of both positive (UGC) and negative (non-UGC) cases. The LLM determines the event type based on core keywords and contextual understanding.
  3. LLM-Aided Event Extraction: Following event recognition, this module uses the LLM to extract specific attributes of the identified UGC events. These attributes include the event name, location (latitude and longitude), time, and potentially the cause and depth of the collapse. The LLM's ability to understand natural language ambiguity is leveraged for this task. Post-processing steps like geocoding and co-reference filtering are applied to refine the extracted information.

Datasets

The study utilizes news reports from the Baidu news portal as its primary data source. For the test region of Zhejiang province, China, the framework crawled 637 news reports. From these, 27 UGC events were identified and extracted. This collection forms the basis of the newly introduced UGC event inventory for Zhejiang province, which is made open-access. This inventory is distinct from existing statistical data by providing detailed spatiotemporal information and trigger factors derived from unstructured text.

Key Insights and Ideas

  • LLMs as powerful tools for disaster data extraction: The study demonstrates the significant potential of LLMs to automate the tedious process of extracting critical information about geological hazards from large volumes of unstructured text data like news reports.
  • The value of news data for UGC analysis: News media serves as a vital, albeit unstructured, source for understanding UGC events, especially in regions lacking comprehensive official records.
  • Zhejiang as a representative case study: The choice of Zhejiang province, known for its rapid development and geological hazards, makes the findings relevant to similar urbanized areas.
  • UGCs are driven by human activities: The analysis suggests that while geological conditions play a role, human activities like construction are significant contributors to UGCs, differentiating them from purely natural geo-hazards.
  • Spatiotemporal patterns are key: The constructed inventory allows for the visualization and analysis of UGC distribution, highlighting hotspots like eastern Hangzhou and revealing trends over time.

Key Results

The framework successfully identified 27 UGC events from 637 news reports in Zhejiang province. The event recognition module achieved a recall rate of over 60%, indicating its effectiveness in identifying relevant news. However, the precision was below 35%, suggesting room for improvement in distinguishing UGCs from other collapse events (e.g., bridge collapses). For event attribute extraction, the model showed a low precision of 34% for time extraction, attributed to the ambiguity of temporal descriptors in news reports. Location extraction had a higher precision of 49%. The resulting inventory covers events from May 2005 to May 2022 and includes detailed information on dates, locations, and potential causes.

Limitations and Future Work

The authors acknowledge several limitations:

  • Distinguishing event types: The LLM struggles to differentiate urban ground collapses from other types of collapses (e.g., tower crane, building, bridge). Future work could involve using ontology-type recognition methods to address this.
  • Accuracy of attribute extraction: The precision for time and location extraction needs improvement. The frequent use of relative time descriptors in news reports poses a challenge. The authors suggest a 3-day time window as an optimal choice for co-reference filtering.
  • Limited training data for fine-tuning: The current dataset is not large enough for effective fine-tuning of the LLM.

Future work could focus on:

  • Integrating knowledge graphs: Combining LLMs with knowledge graphs could enhance semantic representation and improve classification accuracy.
  • Improving prompt engineering: Further refinement of prompts and exploring different LLM architectures could lead to better performance.
  • Expanding the dataset: Collecting a larger and more diverse dataset of easily confused events would enable fine-tuning of the LLM for improved accuracy.

Summary updated

BibTeX
@article{hao2024llm,
  author = {Yanan Hao and Jin Qi and Xiaowen Ma and Sensen Wu and Renyi Liu and Xiaoyi Zhang},
  title = {{An LLM-Based Inventory Construction Framework of Urban Ground Collapse Events with Spatiotemporal Locations}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2024},
  doi = {10.3390/ijgi13040133},
}
Figure