Abstract
Monitoring urban infrastructure damage via social media presents significant challenges due to the high volume of data and the inherent limitations of both AI and human crowdsourcing. This paper introduces StreamCollab, a novel framework that integrates the speed of deep learning with the nuanced judgment of human crowds in a streaming environment. By utilizing a "duo-branch" neural network to estimate prediction uncertainty, the system dynamically allocates human resources only when necessary. Furthermore, a recursive knowledge fusion model handles noisy crowd labels in real-time without the need for computationally expensive model retraining. Experimental results on a real-world Twitter dataset demonstrate that StreamCollab achieves superior accuracy and an order-of-magnitude speedup compared to existing hybrid systems.
Introduction
Modern cities face accelerating infrastructure decay, necessitating rapid and scalable monitoring solutions. Social media offers a rich, "infrastructure-free" data source, yet processing this stream in real-time is difficult. Traditional AI models are fast but often brittle, missing critical damage in complex images (false negatives). Conversely, human crowdsourcing is highly accurate but too slow and expensive for continuous data streams. StreamCollab addresses these gaps by dynamically identifying which images require human intervention and efficiently fusing noisy crowd responses on-the-fly.
Methodology
StreamCollab consists of two core modules designed to optimize the trade-off between accuracy and computational cost:
- Uncertainty-driven Dynamic Quality Estimation (UDQE): This module employs a "duo-branch" neural network. While the first branch performs standard classification (damaged vs. intact), the second branch—the Uncertainty Inference Subnetwork (UIS)—estimates the AI's confidence in its own prediction. High-uncertainty images are "triaged" and sent to human workers for verification.
- Streaming Crowd Knowledge Fusion (SCKF): To manage conflicting human labels without the lag of model retraining, SCKF uses a recursive estimation model. It treats worker reliability as a hidden variable and updates "truth" estimations using a sliding window. This allows the system to maintain high throughput by fixing the output in real-time rather than attempting to immediately retrain the underlying AI model.
Evaluation and Results
The system was evaluated using a real-world dataset of 1,200 urban infrastructure images collected from Twitter, featuring damage such as broken signs and collapsed roads. Ground truth was established by domain experts, and crowd responses were simulated using Amazon Mechanical Turk (AMT) workers.
- Superior Accuracy: At a 10% crowd query ratio, StreamCollab achieved a Cohen’s Kappa (K-Score) of 0.6076, significantly outperforming both AI-only baselines (0.5424) and existing hybrid models (0.5428).
- Efficiency: By avoiding the computational overhead of frequent model retraining, StreamCollab processed images in approximately 0.02 seconds. This is an order of magnitude faster than other hybrid systems, which ranged from 0.16 to 0.29 seconds.
- Robustness: The system remained stable across varying numbers of crowd workers, proving its ability to handle the inherent noise and variability of human input.
Conclusion
StreamCollab provides a scalable, efficient solution for real-time urban monitoring. By treating AI uncertainty as a resource-allocation signal and implementing a recursive fusion approach, it successfully balances high accuracy with the speed required for social sensing. Future work will explore budget-sensitive constraints, scaling to more diverse urban environments, and developing mechanisms to mitigate the impact of misinformation in social media data.
Summary updated