Abstract
Urban environments generate a constant stream of "micro-events"—localized issues such as graffiti, broken streetlights, or illegal dumping—that significantly impact city livability. While citizen sensing platforms allow residents to report these issues, the resulting data is often heterogeneous and ambiguous. This paper introduces a multimodal machine learning framework developed at the University of Amsterdam that fuses text, imagery, geolocation, and weather data. By employing hybrid fusion and multimodal graph embeddings, the framework achieves superior classification performance compared to unimodal approaches, effectively routing reports to the appropriate municipal services.
Introduction
Cities are complex ecosystems where small-scale events can have large-scale impacts on resident satisfaction. Traditional monitoring methods, such as satellite imagery or broad social media sentiment analysis, often miss these "micro-events." Consequently, many cities have implemented "citizen sensing" systems (e.g., 311 apps) to crowdsource urban maintenance. However, the data generated by these systems is notoriously messy, consisting of blurry photos, cryptic text like "Please remove this," and varying GPS accuracy. To address this, we propose a system capable of integrating these diverse data streams to automate the classification and routing of urban reports.
System Architecture
The proposed framework follows a structured pipeline for processing and classifying multimodal reports.
The methodology involves four primary stages as illustrated above:
- Feature Extraction: Textual data is processed using TF-IDF and word2vec. Visual features are extracted from a pre-trained ResNet50. Geo-spatial "fingerprints" are generated based on the proximity to over 500,000 known city objects, while temporal and weather data are one-hot encoded.
- Additional Data Sources: The model incorporates external context, including historical weather patterns and ImageNet pre-training to enhance feature depth.
- Fusion Approaches: The researchers explored several fusion strategies, including Early Fusion (concatenation), Late Fusion (stacking meta-classifiers), and a novel Graph Embedding approach using node2vec to capture latent associations between words, locations, and visual concepts.
- Classification Output: The system predicts at two levels: 8 broad "Main Classes" (e.g., Garbage, Nuisance) and 57 specific "Urban Micro-events" (e.g., Speeding boat, Dead animals).
The Amsterdam Dataset
The framework was validated using a comprehensive real-world dataset from the City of Amsterdam, comprising over 500,000 reports. A multimodal subset of approximately 29,000 reports was used for training, with a further 9,362 reports manually labeled by experts to serve as the ground truth.
As shown in the example above, individual modalities are often insufficient. The text "It is still there" provides no category information without the accompanying image of the bicycle wreckage and the geolocation data.
Results and Key Insights
The multimodal approach demonstrated significant improvements over unimodal baselines:
- Main Class Level: The Hybrid Fusion model achieved a weighted F1-score of 0.882, surpassing the text-only baseline of 0.865.
- Micro-event Level: The system reached an F1-score of 0.700, compared to 0.675 for text-only.
- Human-Level Performance: In a real-world test, the automated textual classifier's 89% accuracy was remarkably close to the 91% accuracy achieved by human domain experts.
Key findings highlight that while text remains the "anchor" modality, geolocation and time provide critical context. For instance, "speeding boats" are geographically tied to canals, while "noise nuisance" follows specific temporal patterns. Visual data proved most beneficial for categories with distinct visual signatures, such as "sunken boats" or "dead animals."
Conclusion
This research demonstrates the power of multimodal fusion in managing urban environments. By representing the city as a heterogeneous graph and utilizing hybrid fusion techniques, municipal systems can more accurately interpret citizen reports. Future work will focus on addressing class imbalance in reporting and exploring more dynamic data sources, such as real-time sensor feeds, to further refine the classification of urban micro-events.
Summary updated