MiCrowd: Vision-Based Deep Crowd Counting on MCU

Sungwook Son, Ahreum Seo, Gyeongseon Eo, Kwangyeon Gill, Taesik Gong, Hyung-Sin Kim

TL;DR

This paper introduces MiCrowd, a novel system for on-device crowd counting using microcontrollers (MCUs). By developing a lightweight deep neural network (MiCrowdNet) and a specialized dataset, they enable accurate crowd estimation on resource-constrained devices, addressing privacy and communication cost concerns.

What Problem Does This Paper Address?

The proliferation of Internet of Things (IoT) devices, often powered by microcontrollers (MCUs), presents an opportunity for ubiquitous sensing. However, MCUs have severe limitations in computational power, memory, and storage, making it challenging to run complex deep neural networks (DNNs) for tasks like computer vision. Traditional methods for crowd counting, such as using smartphone location data from mobile network operators (MNOs), are expensive, can be inaccurate (e.g., not distinguishing pedestrians from drivers), and raise privacy concerns. This paper tackles the problem of performing accurate, on-device crowd counting on MCUs, specifically in the context of e-scooter usage for urban planning and business insights, while mitigating privacy and communication overhead.

Methods

The core of MiCrowd is MiCrowdNet, a custom-designed, lightweight deep neural network for crowd counting. To make it suitable for MCUs, several optimizations were employed:

  • Lightweight Architecture: MiCrowdNet is designed with fewer layers and smaller kernel sizes compared to existing crowd-counting models like MCNN, aiming to reduce computational and memory requirements.
  • Patch-based Processing: To handle high-resolution images within MCU memory constraints, the input images are divided into smaller patches (64x64 pixels) which are processed sequentially. This allows for inference on images that would otherwise be too large to fit in memory.
  • MobileNetV2 Blocks: The network leverages MobileNetV2 convolution blocks, known for their efficiency on mobile and embedded devices.
  • Quantization: Both weights and activations of MiCrowdNet are quantized to 8-bit integers. This significantly reduces model size and speeds up inference, especially crucial for MCUs that often lack floating-point units.
  • Adaptive Gaussian Kernel Sizes: To better represent varying head sizes in images, the paper uses adaptive Gaussian kernel sizes in the density map generation, making the model more robust to scale variations.
  • Memory Optimization: An algorithm for reordering operations within the network graph is implemented to minimize peak memory usage during inference on the MCU.

Datasets

The paper utilizes the Person Re-identification in the Wild (PRW) dataset for training and evaluation. However, they note that existing crowd-counting datasets are often unsuitable for their "mobility view" scenario (i.e., images taken from e-scooter height, around 1-1.5m) as they typically feature dense crowds or aerial views. Therefore, they re-labeled the PRW dataset to create ground-truth density maps for crowd counting. This involved estimating head positions from bounding boxes and applying adaptive Gaussian kernels.

Key Insights and Ideas

  • MCUs are Viable for Vision Tasks: The paper demonstrates that with careful network design and optimization, even resource-constrained MCUs can perform complex vision tasks like crowd counting.
  • E-scooters as Ubiquitous Sensors: E-scooters, with their cameras and connectivity, can serve as a distributed network of sensors for gathering valuable real-world data on floating populations.
  • Privacy and Communication are Key: By performing inference directly on the MCU, MiCrowd significantly reduces the amount of data transmitted, addressing both privacy concerns (no raw images sent) and communication costs.
  • Dataset Matters: The choice of dataset and its adaptation to the specific application scenario (mobility view, sparse crowds) is critical for achieving good performance. Standard crowd-counting datasets are not directly transferable.
  • Quantization is a Game-Changer for MCUs: The significant speedup achieved through 8-bit quantization highlights its importance for deploying DNNs on embedded systems.

Key Results

  • Model Size: MiCrowdNet is remarkably small, with only 0.07M parameters and a quantized model size of 137 kB, making it significantly smaller (up to 600x) than other crowd-counting models.
  • Accuracy: MiCrowdNet achieves a Mean Absolute Error (MAE) of 1.12 on the PRW dataset, which is comparable to much larger models and a 29% improvement over MCNN.
  • Inference Speed: On the ESP32-S3 MCU, the quantized MiCrowdNet achieves a 2.2x speedup compared to its floating-point version, running in approximately 5.7 seconds per 64x64 patch.
  • Memory Efficiency: The patch-based approach and memory optimization techniques allow MiCrowdNet to run on an MCU with only 512 KB SRAM, far below what would be needed for full-image processing.
  • Domain Shift Impact: Experiments show that models trained on aerial view datasets (like ShanghaiTech) perform poorly when tested on mobility view data, underscoring the importance of domain-specific datasets.

Limitations and Future Work

  • Camera Placement: The paper acknowledges that optimal camera placement on e-scooters is crucial and requires further investigation, especially considering different lens types and the use of multiple cameras.
  • Dataset Realism: While PRW was adapted, the authors plan to collect their own real-world dataset to better reflect diverse outdoor conditions.
  • Multi-scooter Collaboration: The paper suggests exploring how multiple e-scooters in close proximity could collaborate to avoid redundant computations and potentially improve accuracy through diverse viewpoints.
  • Generalizability: While the core idea is not limited to e-scooters, the current implementation and dataset are tailored to that scenario. Future work could explore broader applications.

Summary updated

Abstract

Microcontrollers (MCUs) have been deployed on numerous IoT devices due to their compact sizes and low costs. MCUs are capable of capturing sensor data and processing them. However, due to their low computational power, applications processing sensor data with deep neural networks (DNNs) have been limited. In this paper, we propose MiCrowd, a floating population measurement system with a tiny DNNs running on MCUs since the data have essential value in urban planning and business. Moreover, MiCrowd addresses the following important challenges: (1) privacy issues, (2) communication costs, and (3) extreme resource constraints on MCUs. To tackle those challenges, we designed a lightweight crowd-counting deep neural network, named MiCrowdNet, which enables on-MCU inferences. In addition, our dataset is carefully chosen and completely re-labeled to train MiCrowdNet for counting people from an mobility view. Experiments show the effectiveness of MiCrowdNet and our relabeled dataset for accurate on-device crowd counting.
BibTeX
@inproceedings{son2023microwd,
  author = {Sungwook Son and Ahreum Seo and Gyeongseon Eo and Kwangyeon Gill and Taesik Gong and Hyung-Sin Kim},
  title = {{MiCrowd: Vision-Based Deep Crowd Counting on MCU}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=9iJM0ob3Bi6},
}

Loading related content...

Figure