Social Media Meets Big Urban Data: A Case Study of Urban Waterlogging Analysis

Ningyu Zhang, Huajun Chen, Jiaoyan Chen, Xi Chen

TL;DR

This paper tackles the challenge of urban waterlogging analysis, especially in smaller cities with limited sensor data, by proposing a novel multiview discriminant transfer learning approach. By leveraging social media data and physical sensor data from larger cities, they effectively transfer knowledge to improve waterlogging prediction in data-scarce regions, demonstrating significant improvements over traditional methods.

What Problem Does This Paper Address?

Urban waterlogging is a growing concern, exacerbated by climate change and increasing urbanization. Accurate analysis and prediction of waterlogging are crucial for traffic management and disaster mitigation. However, a significant challenge in developing smart city solutions is data sparsity, particularly in smaller cities where infrastructure for data collection (like dense sensor networks) is often lacking. While large cities generate abundant social media data that can be used for analysis, this is not the case for smaller urban areas. This paper aims to bridge this gap by transferring knowledge learned from data-rich (large) cities to data-scarce (small) cities for urban waterlogging analysis.

Methods

The core of the proposed solution lies in a multiview discriminant transfer learning framework. This framework treats urban waterlogging analysis as a problem with multiple "views" of data:

  • Social View: This view is constructed from social media data (tweets, Weibo posts). The method extracts word embeddings from these posts, representing phrases related to waterlogging severity. It then builds a feature vector by considering the average word embeddings and the frequency of specific waterlogging-related terms.
  • Physical View: This view incorporates traditional data sources such as precipitation, terrain elevation, road network characteristics (culverts, intersections), and Points of Interest (POIs). Each of these is processed into feature vectors.

The framework then employs multiview discriminant analysis to combine these views and learn a robust representation. Crucially, it uses transfer learning to transfer this learned knowledge from source cities (typically larger, data-rich cities) to target cities (smaller, data-scarce cities). An autoencoder is used to learn a feature representation that minimizes domain divergence, and a softmax regression classifier is trained on the transferred knowledge. The method is designed to handle the different distributions of features across cities, which is a common issue in transfer learning.

Datasets

The study utilizes real-world datasets from various cities in China and India. Specifically:

  • Social Media Data: Collected from Twitter and Sina Weibo for 2013-2014. The paper notes a significant difference in the volume of social media data between cities like Beijing (massive) and Hangzhou (much less).
  • Precipitation Data: Obtained from the National Oceanic and Atmospheric Administration (NOAA) web service.
  • Road Network Data: Gathered from Openmaps.
  • Terrain Data: Also sourced from Openmaps.
  • Points of Interest (POIs): Collected from Baidu Maps.

The paper uses a specific experimental setup where Beijing (China) is used as a source city and Hangzhou (China) as a target city. For Hangzhou, 30 blocks with label data are used for training and 35 for testing. For Beijing, 121 blocks with label data are used for training.

Key Insights and Ideas

  • Social Media as a Valuable Data Source: The paper strongly advocates for the use of social media data, alongside traditional physical data, for urban waterlogging analysis, especially in situations where sensor data is scarce.
  • Transfer Learning for Data Sparsity: The core idea that knowledge learned from data-rich environments can be effectively transferred to data-scarce environments is central to solving real-world problems in urban computing.
  • Multiview Learning Advantage: Combining different "views" of data (social and physical) provides a more comprehensive understanding of waterlogging phenomena than relying on a single view.
  • Domain Adaptation is Crucial: The distinct characteristics and data distributions of different cities necessitate domain adaptation techniques to ensure successful knowledge transfer.

Key Results

The paper demonstrates the effectiveness of its proposed multiview discriminant transfer learning method.

  • Single City Transfer: When transferring knowledge from a source city to a target city, the transfer learning method consistently outperforms the baseline method that uses only the target city's data. For example, the enhancement (Fst/F1) is shown to be significant, especially when transferring from larger cities with more social media activity to smaller ones.
  • Multiple City Transfer: Transferring knowledge from multiple source cities to a single target city also shows improvement compared to single-city transfer and baselines.
  • Quantitative Improvements: Table 2 shows that the proposed "Multiview transfer learning with autocoder (feature-transfer)" method achieves higher F1 scores across various target cities (Beijing, Shanghai, Hangzhou, Mumbai, New Delhi) compared to GBRT, ANN, and TrAdaBoost baselines, particularly when considering both social and physical views. For instance, for Hangzhou, the proposed method achieves an F1 score of 0.894, significantly higher than GBRT (0.711) and ANN (0.814).

Limitations and Future Work

  • Data Distribution Differences: The paper acknowledges that features in different cities have different distributions, which is a primary motivation for transfer learning. However, the effectiveness of the transfer can still be influenced by the degree of these differences.
  • Scalability to More Cities: The authors express a desire to apply their approach to a wider range of cities to further validate its generalizability.
  • Broader Applications: They hypothesize that the proposed algorithm could be successful in transferring knowledge for other urban data sparsity problems, such as air quality index (AQI) analysis or traffic prediction, suggesting future research directions.
  • Feature Transfer vs. Instance Transfer: The paper argues that feature-transfer methods (like theirs) are more suitable than instance-transfer methods (like TrAdaBoost) when feature distributions differ significantly across domains, which is a key insight for future work in similar transfer learning scenarios.

Summary updated

BibTeX
@inproceedings{zhang2016social,
  author = {Ningyu Zhang and Huajun Chen and Jiaoyan Chen and Xi Chen},
  title = {{Social Media Meets Big Urban Data: A Case Study of Urban Waterlogging Analysis}},
  booktitle = {Comput. Intell. Neurosci},
  year = {2016},
  url = {https://openreview.net/forum?id=wKDXaXJXng},
}
Figure