On-sensor binarized CNN inference with dynamic model swapping in pixel processor arrays

Yanan Liu, Laurie Bose, Rui Fan, Piotr Dudek, Walterio Mayol‐CuevasFrontiers in Neuroscience 2022 · DOI

TL;DR

This paper introduces novel methods for running efficient Convolutional Neural Networks (CNNs) directly on resource-constrained Pixel Processor Arrays (PPAs). They achieve this by developing techniques for training purely binarized CNNs and implementing a dynamic model swapping system, enabling complex tasks like classification, localization, and segmentation on embedded vision systems.

What Problem Does This Paper Address?

Embedded systems, especially those with limited computational power and memory like Pixel Processor Arrays (PPAs), struggle to run complex deep learning models. Traditional CNNs require significant resources for storing weights, performing floating-point computations, and managing intermediate activations. This paper tackles the challenge of deploying efficient and accurate CNN inference directly on these "all-in-sensor" devices, aiming to reduce latency, energy consumption, and data bandwidth by processing data at the source.

Methods

The authors propose two primary methods:

  • Purely Binarized CNNs with Batch Normalization: They train CNNs where both weights and activations are constrained to binary values (+1 or -1). A key innovation is the integration of batch normalization during training, which is then converted into a simple bias matrix for efficient on-sensor implementation using add/subtract operations. This approach simplifies computations and reduces memory requirements.
  • Dynamic Model Swapping (CNN Trees): For tasks exceeding the PPA's capacity, they propose decomposing them into smaller sub-tasks. These sub-tasks are handled by individual CNN modules that can be dynamically loaded into the PPA's registers as needed. This "CNN tree" architecture allows for more complex applications by chaining simpler, loadable models, effectively managing limited on-sensor resources. They also introduce the first implementation of a Fully Convolutional Network (FCN) on a PPA.

Datasets

  • EMNIST dataset: Used for English letter classification. This dataset is an extension of MNIST and contains 37 classes of handwritten characters, including lower-case, upper-case, and mixed cases, posing a challenge due to similarities between certain characters.
  • Simulated datasets for road and grass segmentation, and vehicle localization: Collected using a Webots robot simulator and a simulated drone camera, these datasets were used to train and validate the FCN for segmentation and localization tasks.

Key Insights and Ideas

  • Batch Normalization to Bias Matrix: Converting batch normalization parameters into a bias matrix for binarized activations is a clever way to integrate a powerful regularization technique into a highly constrained hardware architecture without significant overhead.
  • Dynamic Model Swapping as a Resource Management Strategy: The "CNN tree" approach is a pragmatic solution for deploying complex AI on edge devices. By loading only necessary model components, it bypasses the limitations of fixed on-chip memory.
  • FCN on PPA: Implementing a Fully Convolutional Network (FCN) on a PPA for tasks like segmentation is a significant step, moving beyond simple classification and demonstrating the PPA's potential for more sophisticated computer vision tasks.
  • Binarization Benefits: The paper reinforces that binarization is crucial for on-sensor computing, not just for reducing memory but also for simplifying computations to bitwise operations and mitigating analog computing errors.

Key Results

  • EMNIST Classification: A CNN tree architecture achieved an accuracy of 86.74% on the 37-class EMNIST dataset, outperforming a single 4-layer CNN.
  • Object Localization: Achieved a localization accuracy of around 83% on the sensor, with a small performance gap (around 5%) compared to simulation, attributed to analog noise and bit-flipping errors.
  • Coarse Segmentation: Demonstrated Intersection over Union (IoU) scores of 69.3% for road segmentation and 72.9% for grass segmentation on the sensor, showing competitive performance with simulation.
  • Latency: Achieved a latency of 3.5 ms for each inference in segmentation and localization tasks.
  • Computational Efficiency: The binarization and dynamic swapping methods significantly reduce computational complexity and memory footprint, making them suitable for PPAs.

Limitations and Future Work

  • Analog Noise and Bit-Flipping: The authors acknowledge that analog noise in the processing elements and random bit-flipping in registers introduce errors, leading to a performance gap compared to simulations. They suggest further software or hardware solutions are needed.
  • Prototype Hardware Limitations: The current PPA prototype hardware limits the complexity of networks that can be deployed.
  • Future Hardware Development: The paper suggests that further advancements in PPA hardware are crucial for implementing even more sophisticated networks like AlexNet and VGG-16, and for improving latency, energy consumption, and data bandwidth.
  • Model Upload Time: While dynamic swapping is effective, the time to upload models from external flash memory into DREGs (around 26 ms) is noted as an issue that could be mitigated by pre-storing models.

Summary updated

Abstract

Many types of Convolutional Neural Network (CNN) models and training methods have been proposed in recent years aiming to provide efficiency for embedded and edge devices with limited computation and memory resources. The wide variety of architectures makes this a complex task that has to balance generality with efficiency. Among the most interesting camera-sensor architectures are Pixel Processor Arrays (PPAs). This study presents two methods that are useful for embedded CNNs in general but particularly suitable for PPAs. The first is for training purely binarized CNNs, the second is for deploying larger models with a model swapping paradigm that loads model components dynamically. Specifically, this study trains and implements networks with batch normalization and adaptive threshold for binary activations. Then, we convert batch normalization and binary activations into a bias matrix which can be parallelly implemented by an add/sub operation. For dynamic model swapping, we propose to decompose applications that are beyond the capacity of a PPA into sub-tasks that can be solved by tree networks that can be loaded dynamically as needed. We demonstrate our approaches to various tasks including classification, localization, and coarse segmentation on a highly resource constrained PPA sensor-processor.
BibTeX
@inproceedings{liu2022sensor,
  author = {Yanan Liu and Laurie Bose and Rui Fan and Piotr Dudek and Walterio Mayol‐Cuevas},
  title = {{On-sensor binarized CNN inference with dynamic model swapping in pixel processor arrays}},
  booktitle = {Frontiers in Neuroscience},
  year = {2022},
  doi = {10.3389/fnins.2022.909448},
}
Figure