RaScaNet: Learning Tiny Models by Raster-Scanning Images

Jaehyoung Yoo, Dongwook Lee, Changyong Son, Sangil Jung, ByungIn Yoo, Changkyu Choi, Jae-Joon Han, Bohyung HanCVPR 2021 · OpenReview

Abstract

Deploying deep convolutional neural networks (CNNs) on ultra-low power systems is challenging due to extremely limited resources, particularly on-chip memory. Peak memory usage in lower layers often forces aggressive input downsampling, sacrificing accuracy. We propose RaScaNet, a novel Raster-Scanning Network inspired by image sensor operations. RaScaNet processes images row-by-row using a CNN-RNN hybrid, requiring 15.9–24.3× smaller peak memory than state-of-the-art tiny models. By fitting within 60 KB of SRAM/cache, RaScaNet significantly improves power efficiency and performance on Visual Wake Words and Pascal VOC datasets.

The Memory Bottleneck in TinyML

Deep learning on microcontrollers (MCUs) with 100–320 KB of SRAM is constrained by the "peak memory bottleneck." In standard CNNs, the first layers generate large activation maps because they process the entire high-resolution input simultaneously. To fit these models, researchers often downsample images, which destroys fine-grained features and reduces accuracy. Existing solutions like MCUNet or MobileNetV2 still require hundreds of kilobytes, often necessitating power-hungry fetches from external flash storage.

Methodology: Raster-Scanning Architecture

RaScaNet shifts the paradigm by mimicking how image sensors stream data. Instead of processing a full frame, it operates on a few rows of pixels (a "strip") at a time.

The RaScaNet pipeline processes image strips sequentially, integrating CNN feature extraction with RNN state updates directly from the image sensor's ISP.

Sequential Strip Processing

The architecture consists of two main components:

  1. Shallow Multi-head CNN: This component extracts local features from each horizontal strip.
  2. Convolutional RNN (GRU variant): The RNN maintains a hidden state that captures vertical context as strips are processed. This "memory" allows the network to understand the whole image without ever storing its full activation maps.

Attention and Early Termination

RaScaNet employs spatial and channel attention, using the previous RNN hidden state to highlight relevant features in the current strip. Furthermore, the Early Termination (RaScaNet-ET) variant allows the model to stop scanning once a confidence threshold is met, saving significant computational cycles (MACs) when a subject is identified early in the scan.

Key Results and Memory Efficiency

RaScaNet achieves a total footprint (peak memory + weights) of under 60 KB, allowing it to reside entirely within on-chip SRAM and cache.

Memory footprint comparison: RaScaNet is the only model that fits within the target 60KB on-chip memory limit (Cache + partial SRAM).
  • Memory Savings: Compared to MCUNet, RaScaNet reduces peak memory by up to 24.3× and weight memory by up to 12.9×.
  • Accuracy: On the Visual Wake Words (VWW) dataset, RaScaNet maintains a high accuracy of ~91.6%, proving that sequential processing does not compromise performance.
  • Efficiency: The Early Termination strategy reduces average MACs by 17.1% without accuracy loss.

Conclusion

RaScaNet demonstrates that hardware-algorithm co-design—specifically aligning neural network operations with the raster-scan nature of image sensors—can overcome the fundamental memory limits of ultra-low-power systems. This approach paves the way for sophisticated "on-sensor" AI applications that are both highly efficient and accurate.

Summary updated

Abstract

Deploying deep convolutional neural networks on ultra-low power systems is challenging due to the extremely limited resources. Especially, the memory becomes a bottleneck as the systems put a hard limit on the size of on-chip memory. Because peak memory explosion in the lower layers is critical even in tiny models, the size of an input image should be reduced with sacrifice in accuracy. To overcome this drawback, we propose a novel Raster-Scanning Network, named RaScaNet, inspired by raster-scanning in image sensors. RaScaNet reads only a few rows of pixels at a time using a convolutional neural network and then sequentially learns the representation of the whole image using a recurrent neural network. The proposed method operates on an ultra-low power system without input size reduction; it requires 15.9-24.3x smaller peak memory and 5.3-12.9x smaller weight memory than the state-of-the-art tiny models. Moreover, RaScaNet fully exploits on-chip SRAM and cache memory of the system as the sum of the peak memory and the weight memory does not exceed 60 KB, improving the power efficiency of the system. In our experiments, we demonstrate the binary classification performance of RaScaNet on Visual Wake Words and Pascal VOC datasets.
BibTeX
@inproceedings{yoo2021rascanet,
  author = {Jaehyoung Yoo and Dongwook Lee and Changyong Son and Sangil Jung and ByungIn Yoo and Changkyu Choi and Jae-Joon Han and Bohyung Han},
  title = {{RaScaNet: Learning Tiny Models by Raster-Scanning Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=kkLgn7Fl43h},
}
Figure