StreamCollab: A Streaming Crowd-AI Collaborative System to Smart Urban Infrastructure Monitoring in Social Sensing

Yang Zhang, Lanyu Shang, Ruohan Zong, Zeng Wang, Ziyi Kou, Dong Wang

Abstract

Monitoring urban infrastructure damage via social media presents significant challenges due to the high volume of data and the inherent limitations of both AI and human crowdsourcing. This paper introduces StreamCollab, a novel framework that integrates the speed of deep learning with the nuanced judgment of human crowds in a streaming environment. By utilizing a "duo-branch" neural network to estimate prediction uncertainty, the system dynamically allocates human resources only when necessary. Furthermore, a recursive knowledge fusion model handles noisy crowd labels in real-time without the need for computationally expensive model retraining. Experimental results on a real-world Twitter dataset demonstrate that StreamCollab achieves superior accuracy and an order-of-magnitude speedup compared to existing hybrid systems.

Introduction

Modern cities face accelerating infrastructure decay, necessitating rapid and scalable monitoring solutions. Social media offers a rich, "infrastructure-free" data source, yet processing this stream in real-time is difficult. Traditional AI models are fast but often brittle, missing critical damage in complex images (false negatives). Conversely, human crowdsourcing is highly accurate but too slow and expensive for continuous data streams. StreamCollab addresses these gaps by dynamically identifying which images require human intervention and efficiently fusing noisy crowd responses on-the-fly.

Methodology

StreamCollab consists of two core modules designed to optimize the trade-off between accuracy and computational cost:

  1. Uncertainty-driven Dynamic Quality Estimation (UDQE): This module employs a "duo-branch" neural network. While the first branch performs standard classification (damaged vs. intact), the second branch—the Uncertainty Inference Subnetwork (UIS)—estimates the AI's confidence in its own prediction. High-uncertainty images are "triaged" and sent to human workers for verification.
  2. Streaming Crowd Knowledge Fusion (SCKF): To manage conflicting human labels without the lag of model retraining, SCKF uses a recursive estimation model. It treats worker reliability as a hidden variable and updates "truth" estimations using a sliding window. This allows the system to maintain high throughput by fixing the output in real-time rather than attempting to immediately retrain the underlying AI model.
The StreamCollab system architecture, illustrating the flow from social media image streams through AI uncertainty detection to crowd knowledge fusion.
The duo-branch neural network architecture, showing the classification branch (Output Branch 1) and the uncertainty inference branch (Output Branch 2).

Evaluation and Results

The system was evaluated using a real-world dataset of 1,200 urban infrastructure images collected from Twitter, featuring damage such as broken signs and collapsed roads. Ground truth was established by domain experts, and crowd responses were simulated using Amazon Mechanical Turk (AMT) workers.

  • Superior Accuracy: At a 10% crowd query ratio, StreamCollab achieved a Cohen’s Kappa (K-Score) of 0.6076, significantly outperforming both AI-only baselines (0.5424) and existing hybrid models (0.5428).
  • Efficiency: By avoiding the computational overhead of frequent model retraining, StreamCollab processed images in approximately 0.02 seconds. This is an order of magnitude faster than other hybrid systems, which ranged from 0.16 to 0.29 seconds.
  • Robustness: The system remained stable across varying numbers of crowd workers, proving its ability to handle the inherent noise and variability of human input.

Conclusion

StreamCollab provides a scalable, efficient solution for real-time urban monitoring. By treating AI uncertainty as a resource-allocation signal and implementing a recursive fusion approach, it successfully balances high accuracy with the speed required for social sensing. Future work will explore budget-sensitive constraints, scaling to more diverse urban environments, and developing mechanisms to mitigate the impact of misinformation in social media data.

Summary updated

Abstract

Social sensing has emerged as a pervasive and scalable sensing paradigm to collect observations of the physical world from human sensors. A key advantage of social sensing is its infrastructure-free nature. In this paper, we focus on a streaming urban infrastructure monitoring (Streaming UIM) problem in social sensing. The goal is to automatically detect the urban infrastructure damages from the streaming imagery data posted on social media by exploring the collective power of both AI and human intelligence from crowdsourcing systems. Our work is motivated by the limitation of current AI and crowdsourcing solutions that either fail in many critical time-sensitive UIM application scenarios or are not easily generalizable to monitor the damage of different types of urban infrastructures. We identify two critical challenges in solving our problem: i) it is difficult to dynamically integrate AI and crowd intelligence to effectively identify and fix the failure cases of AI solutions; ii) it is non-trivial to obtain accurate human intelligence from unreliable crowd workers in streaming UIM applications. In this paper, we propose StreamCollab, a streaming crowd-AI collaborative system that explores the collaborative intelligence from AI and crowd to solve the streaming UIM problem. The evaluation results on a real-world urban infrastructure imagery dataset collected from social media demonstrate that StreamCollab consistently outperforms both state-of-the-art AI and crowd-AI baselines in UIM accuracy while maintaining the lowest computational cost.
BibTeX
@inproceedings{zhang2021streamcollab,
  author = {Yang Zhang and Lanyu Shang and Ruohan Zong and Zeng Wang and Ziyi Kou and Dong Wang},
  title = {{StreamCollab: A Streaming Crowd-AI Collaborative System to Smart Urban Infrastructure Monitoring in Social Sensing}},
  booktitle = {HCOMP},
  year = {2021},
  doi = {10.1609/hcomp.v9i1.18950},
}
Figure