HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor

Pankaj Bhowmik, Md Jubaer Hossain Pantho, Christophe BobdaSensors 2021 · DOI · OpenReview

TL;DR

This paper introduces HARP, a novel hardware architecture for vision sensors that performs hierarchical, attention-oriented processing directly on the sensor's focal plane. By intelligently identifying and processing only salient image regions, HARP significantly reduces data transmission, power consumption, and latency, enabling high-performance computation for time-critical applications on edge devices.

What Problem Does This Paper Address?

Modern vision systems, especially those using high-resolution CMOS image sensors, generate massive amounts of data. Traditionally, this data is offloaded to the cloud for processing, which creates a bottleneck for time-critical applications like autonomous driving and surveillance due to communication latency and privacy concerns. While some edge processing exists, it's often limited to low-level tasks. High-level inference, like Convolutional Neural Networks (CNNs), is computationally expensive and requires significant memory, making it challenging to implement directly on resource-constrained vision sensors. The paper aims to bridge this gap by enabling efficient, high-level computation directly at the sensor, leveraging the sensor's inherent bandwidth while minimizing data transfer.

Methods

The core of HARP is its Hierarchical Attention Oriented Region-Based Processing. This approach involves two main stages:

  • Attention-Based Preprocessing Layer (APL): This layer operates on the image sensor's focal plane. It divides the image into smaller regions and uses low-level image processing techniques (like edge detection and temporal saliency) to identify "relevant" regions. This is inspired by biological vision systems where only salient information is processed. Regions deemed irrelevant are effectively "blanked out," drastically reducing the amount of data that needs further processing.
  • Inference Computation Layer (ICL): This layer receives only the relevant regions identified by the APL. It then performs high-level inference, specifically CNN operations, using Region Inference Engines (RIEs). The RIEs are designed for efficient, partial data-based computation, processing convolutional layers in a pipelined manner to reduce on-chip memory requirements and off-chip communication. A Fully-Connected Neural Network (FcNN) module handles the final layers.

Key novelties include:

  • Sensor-level hierarchical processing: Moving from low-level attention to high-level inference directly on the sensor.
  • Attention-oriented pruning: Using saliency scores to discard redundant information before computationally intensive processing.
  • Partial data-based computation: Processing convolutional layers in stages to reduce memory footprint and data movement.
  • Pipelined RIEs: Enabling efficient execution of multiple convolutional layers with reduced reliance on external memory.

Datasets

The paper mentions using the following datasets for evaluation:

  • MOT17-SD1: Used to analyze inter-frame temporal redundancies in video sequences.
  • MNIST: Used to evaluate spatial redundancy and the impact of pruning on accuracy.
  • FashionMNIST: Similar to MNIST, used for evaluating spatial redundancy and pruning impact.
  • Ades and Culez mosquito species image dataset: Used as a case study for the VGG-16 classifier, chosen for its high redundant information and high image resolution.

No new benchmarks were introduced; existing datasets were used to demonstrate the effectiveness of the proposed pruning and processing strategy.

Key Insights and Ideas

  • Biological Inspiration Pays Off: Mimicking the human visual system's attention mechanism is a powerful strategy for efficient image processing, especially on resource-constrained edge devices.
  • Pruning is Key: Identifying and discarding irrelevant image regions early in the processing pipeline offers substantial gains in power, latency, and memory usage, without sacrificing accuracy.
  • Sensor-Level Intelligence is Achievable: HARP demonstrates that complex inference tasks can be effectively integrated into vision sensors, moving beyond simple data acquisition.
  • Trade-offs are Manageable: While HARP requires more area than a basic pixel circuit, the significant improvements in performance metrics (latency, energy, memory) justify this overhead for demanding applications.
  • Partial Data Processing is Crucial: For deep CNNs, processing data in chunks and pipelining operations is essential to manage memory and computational demands on-chip.

Key Results

  • Speedup: Achieved significant speedup, with improvements up to 70.1% in execution time when using attention-oriented processing with 25% relevant information.
  • Energy Efficiency: Demonstrated energy savings of up to 45.82% and power reduction by 76.34% in cases with high redundancy.
  • Latency Reduction: Latency was reduced significantly, with the first convolution layer being 15.82× faster when only 10% of regions were relevant.
  • Memory Optimization: The partial-data-based strategy reduced on-chip memory requirements by 1.99× for computing two convolutional layers.
  • FPGA/ASIC Prototyping: The architecture was prototyped on Virtex UltraScale+ FPGA and a 90nm ASIC, achieving high frequencies (320 MHz on FPGA, 430 MHz on ASIC).
  • Accuracy: No accuracy drop was observed when pruning irrelevant regions on MNIST, FashionMNIST, and the mosquito dataset.

Limitations and Future Work

  • Area Overhead: The APL layer introduces an area overhead (11.34% of the overall design) compared to a basic pixel circuit, though this is justified by performance gains.
  • Generalizability: While HARP is demonstrated with VGG-16, its adaptability to other complex CNN models with varying layer structures is a key aspect of its design. The paper suggests it can be used for other real-time applications.
  • Future Work: The paper implicitly suggests further exploration into optimizing the attention mechanisms, exploring different CNN architectures, and potentially integrating more advanced AI models directly onto the sensor. The authors also mention the possibility of using power gating to further reduce leakage power, which was not the primary focus.

Summary updated

Abstract

Cameras are widely adopted for high image quality with the rapid advancement of complementary metal-oxide-semiconductor (CMOS) image sensors while offloading vision applications’ computation to the cloud. It raises concern for time-critical applications such as autonomous driving, surveillance, and defense systems since moving pixels from the sensor’s focal plane are expensive. This paper presents a hardware architecture for smart cameras that understands the salient regions from an image frame and then performs high-level inference computation for sensor-level information creation instead of transporting raw pixels. A visual attention-oriented computational strategy helps to filter a significant amount of redundant spatiotemporal data collected at the focal plane. A computationally expensive learning model is then applied to the interesting regions of the image. The hierarchical processing in the pixels’ data path demonstrates a bottom-up architecture with massive parallelism and gives high throughput by exploiting the large bandwidth available at the image source. We prototype the model in field-programmable gate array (FPGA) and application-specific integrated circuit (ASIC) for integrating with a pixel-parallel image sensor. The experiment results show that our approach achieves significant speedup while in certain conditions exhibits up to 45% more energy efficiency with the attention-oriented processing. Although there is an area overhead for inheriting attention-oriented processing, the achieved performance based on energy consumption, latency, and memory utilization overcomes that limitation.
BibTeX
@article{bhowmik2021harp,
  author = {Pankaj Bhowmik and Md Jubaer Hossain Pantho and Christophe Bobda},
  title = {{HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor}},
  journal = {Sensors},
  year = {2021},
  doi = {10.3390/s21051757},
}
Figure