Multimodal Classification of Urban Micro-Events

Maarten Sukel, Stevan Rudinac, Marcel Worring

Abstract

Urban environments generate a constant stream of "micro-events"—localized issues such as graffiti, broken streetlights, or illegal dumping—that significantly impact city livability. While citizen sensing platforms allow residents to report these issues, the resulting data is often heterogeneous and ambiguous. This paper introduces a multimodal machine learning framework developed at the University of Amsterdam that fuses text, imagery, geolocation, and weather data. By employing hybrid fusion and multimodal graph embeddings, the framework achieves superior classification performance compared to unimodal approaches, effectively routing reports to the appropriate municipal services.

Introduction

Cities are complex ecosystems where small-scale events can have large-scale impacts on resident satisfaction. Traditional monitoring methods, such as satellite imagery or broad social media sentiment analysis, often miss these "micro-events." Consequently, many cities have implemented "citizen sensing" systems (e.g., 311 apps) to crowdsource urban maintenance. However, the data generated by these systems is notoriously messy, consisting of blurry photos, cryptic text like "Please remove this," and varying GPS accuracy. To address this, we propose a system capable of integrating these diverse data streams to automate the classification and routing of urban reports.

System Architecture

The proposed framework follows a structured pipeline for processing and classifying multimodal reports.

Overview of the multimodal classification pipeline.

The methodology involves four primary stages as illustrated above:

  1. Feature Extraction: Textual data is processed using TF-IDF and word2vec. Visual features are extracted from a pre-trained ResNet50. Geo-spatial "fingerprints" are generated based on the proximity to over 500,000 known city objects, while temporal and weather data are one-hot encoded.
  2. Additional Data Sources: The model incorporates external context, including historical weather patterns and ImageNet pre-training to enhance feature depth.
  3. Fusion Approaches: The researchers explored several fusion strategies, including Early Fusion (concatenation), Late Fusion (stacking meta-classifiers), and a novel Graph Embedding approach using node2vec to capture latent associations between words, locations, and visual concepts.
  4. Classification Output: The system predicts at two levels: 8 broad "Main Classes" (e.g., Garbage, Nuisance) and 57 specific "Urban Micro-events" (e.g., Speeding boat, Dead animals).

The Amsterdam Dataset

The framework was validated using a comprehensive real-world dataset from the City of Amsterdam, comprising over 500,000 reports. A multimodal subset of approximately 29,000 reports was used for training, with a further 9,362 reports manually labeled by experts to serve as the ground truth.

An example of a multimodal citizen report featuring text, image, and geolocation.

As shown in the example above, individual modalities are often insufficient. The text "It is still there" provides no category information without the accompanying image of the bicycle wreckage and the geolocation data.

Results and Key Insights

The multimodal approach demonstrated significant improvements over unimodal baselines:

  • Main Class Level: The Hybrid Fusion model achieved a weighted F1-score of 0.882, surpassing the text-only baseline of 0.865.
  • Micro-event Level: The system reached an F1-score of 0.700, compared to 0.675 for text-only.
  • Human-Level Performance: In a real-world test, the automated textual classifier's 89% accuracy was remarkably close to the 91% accuracy achieved by human domain experts.

Key findings highlight that while text remains the "anchor" modality, geolocation and time provide critical context. For instance, "speeding boats" are geographically tied to canals, while "noise nuisance" follows specific temporal patterns. Visual data proved most beneficial for categories with distinct visual signatures, such as "sunken boats" or "dead animals."

Conclusion

This research demonstrates the power of multimodal fusion in managing urban environments. By representing the city as a heterogeneous graph and utilizing hybrid fusion techniques, municipal systems can more accurately interpret citizen reports. Future work will focus on addressing class imbalance in reporting and exploring more dynamic data sources, such as real-time sensor feeds, to further refine the classification of urban micro-events.

Summary updated

Abstract

In this paper we seek methods to effectively detect urban micro-events. Urban micro-events are events which occur in cities, have limited geographical coverage and typically affect only a small group of citizens. Because of their scale these are difficult to identify in most data sources. However, by using citizen sensing to gather data, detecting them becomes feasible. The data gathered by citizen sensing is often multimodal and, as a consequence, the information required to detect urban micro-events is distributed over multiple modalities. This makes it essential to have a classifier capable of combining them. In this paper we explore several methods of creating such a classifier, including early, late, hybrid fusion and representation learning using multimodal graphs. We evaluate performance on a real world dataset obtained from a live citizen reporting system. We show that a multimodal approach yields higher performance than unimodal alternatives. Furthermore, we demonstrate that our hybrid combination of early and late fusion with multimodal embeddings performs best in classification of urban micro-events.
BibTeX
@inproceedings{sukel2019multimodal,
  author = {Maarten Sukel and Stevan Rudinac and Marcel Worring},
  title = {{Multimodal Classification of Urban Micro-Events}},
  booktitle = {ACM Multimedia},
  year = {2019},
  doi = {10.1145/3343031.3350967},
}
Figure