A Spatial Information Extraction Method Based on Multi-Modal Social Media Data: A Case Study on Urban Inundation

Yilong Wu, Yingjie Chen, Rongyu Zhang, Zhenfei Cui, Xinyi Liu, Jiayi Zhang, Meizhen Wang, Yong WuISPRS Int. J. Geo Inf 2023 · DOI

TL;DR

This paper introduces MIST-SMMD, a novel method for extracting spatiotemporal information from social media, particularly focusing on urban inundation events. By integrating text, images, and videos, MIST-SMMD significantly improves the accuracy of spatial information extraction compared to text-only approaches, offering a valuable tool for urban disaster management.

What Problem Does This Paper Address?

Accurate spatiotemporal information is crucial for effective urban management and emergency response, especially during events like urban inundations. While social media platforms like Weibo are rich sources of real-time information, existing methods for extracting this data often rely solely on text, neglecting the wealth of information present in images and videos. This unimodal approach limits the precision of spatial information extraction. The paper aims to bridge this gap by developing a method that leverages multimodal social media data to achieve more accurate and comprehensive spatiotemporal event monitoring.

Methods

The proposed MIST-SMMD method operates in three main steps:

  1. Data Crawl and Pre-Processing: This involves collecting multimodal data (text, images, videos) from Weibo using keywords related to urban events. The data undergoes cleaning (removing noise, duplicates) and event classification to ensure relevance. For videos, stable frames are extracted using optical flow.
  2. Coarse-grained Extraction of Spatiotemporal Information: This step focuses on extracting initial spatiotemporal information primarily from text. It utilizes Named Entity Recognition (NER) to identify relevant entities and then applies a stringent standardization process for both temporal (Year-Month-Day) and spatial (Province-City-District-Specific Geographic Location) information. This standardized output is then geocoded into WGS1984 coordinates.
  3. Fine-grained Extraction of Spatial Information: This is the core innovation, where multimodal data is leveraged to refine spatial accuracy. It employs a cascaded computer vision model called LSGL (LOFTR-Seg Geo-Localization). LSGL uses the LOFTR (Local Feature Transformer) for robust feature matching, even with blurry images, and incorporates semantic segmentation (SS) and quantitative indicators (QIFM) to improve matching accuracy. The process involves matching user-uploaded images with street view images to pinpoint precise locations.

Datasets

The study uses Weibo data for the "20 July Zhengzhou Heavy Rainfall" incident as a case study. The data was collected from July 18 to July 20, 2021, using keywords related to waterlogging. For spatial localization, street view images from Baidu Maps are utilized. The paper also references a dataset of spatial distribution of flood inundation in Zhengzhou City from July 2021 by Lu et al. (2022) as ground truth.

Key Insights and Ideas

  • Multimodal Data is Key: The study strongly demonstrates that integrating image and video data with text significantly enhances the precision of spatial information extraction, a crucial aspect often overlooked by unimodal approaches.
  • Hierarchical Extraction: The coarse-to-fine-grained approach effectively breaks down the complex task of spatiotemporal extraction, first establishing a general location and then refining it with more detailed visual information.
  • Leveraging Social Media for Disaster Response: The research highlights the immense potential of social media data as a supplementary, low-cost, and widely available source for real-time urban disaster monitoring and management.
  • Challenges of Social Media Data: The paper acknowledges inherent challenges like data noise, heterogeneity, sparsity, and the potential misalignment between textual and visual information within social media posts.

Key Results

  • Coarse-grained Spatial Precision: The method achieved 65.88% spatial precision within a 52m buffer range and 87.54% within a 60m buffer range. Ultimately, it reached 100% spatial precision within a 201m range.
  • Fine-grained Spatial Error Reduction: Compared to coarse-grained text-only extraction, fine-grained extraction using multimodal data (Text + Images) significantly reduced spatial errors. MAESE improved by 95.53% (from 1491.13 to 66.63) and RMSESE by 93.62% (from 2068.43 to 131.88).
  • Effectiveness of LSGL Components: Ablation studies showed that the combination of Feature Matching (FM) and Semantic Segmentation (SS) performed best in reducing spatial error, though the inclusion of QIFM sometimes led to deterioration in specific scenarios.

Limitations and Future Work

  • Data Volume and Quality: The study acknowledges that the volume of high-quality, relevant multimodal social media data is currently limited, which impacts the robustness of findings. Future work should focus on larger and more diverse datasets.
  • Image-Text Alignment: A significant limitation is the potential misalignment between textual and visual information in social media posts, which can lead to anomalies in fine-grained results.
  • Operational Accuracy: The accuracy of the model's operational results is influenced by the density and timeliness of local street-view images.
  • Privacy Concerns: The collection of public media data may inadvertently include images from individual users, raising concerns about location privacy and image copyright infringement, which the authors attempted to mitigate by removing personal information.
  • Future Research Directions: The authors suggest exploring more advanced models for event classification and NER, integrating more unutilized social media information, and incorporating other data sources to enhance robustness. They also emphasize the potential for real-time extraction and processing to contribute to more efficient urban emergency systems.

Summary updated

BibTeX
@article{wu2023spatial,
  author = {Yilong Wu and Yingjie Chen and Rongyu Zhang and Zhenfei Cui and Xinyi Liu and Jiayi Zhang and Meizhen Wang and Yong Wu},
  title = {{A Spatial Information Extraction Method Based on Multi-Modal Social Media Data: A Case Study on Urban Inundation}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2023},
  doi = {10.3390/ijgi12090368},
}
Figure