Abstract
Deploying deep convolutional neural networks (CNNs) on ultra-low power systems is challenging due to extremely limited resources, particularly on-chip memory. Peak memory usage in lower layers often forces aggressive input downsampling, sacrificing accuracy. We propose RaScaNet, a novel Raster-Scanning Network inspired by image sensor operations. RaScaNet processes images row-by-row using a CNN-RNN hybrid, requiring 15.9–24.3× smaller peak memory than state-of-the-art tiny models. By fitting within 60 KB of SRAM/cache, RaScaNet significantly improves power efficiency and performance on Visual Wake Words and Pascal VOC datasets.
The Memory Bottleneck in TinyML
Deep learning on microcontrollers (MCUs) with 100–320 KB of SRAM is constrained by the "peak memory bottleneck." In standard CNNs, the first layers generate large activation maps because they process the entire high-resolution input simultaneously. To fit these models, researchers often downsample images, which destroys fine-grained features and reduces accuracy. Existing solutions like MCUNet or MobileNetV2 still require hundreds of kilobytes, often necessitating power-hungry fetches from external flash storage.
Methodology: Raster-Scanning Architecture
RaScaNet shifts the paradigm by mimicking how image sensors stream data. Instead of processing a full frame, it operates on a few rows of pixels (a "strip") at a time.
Sequential Strip Processing
The architecture consists of two main components:
- Shallow Multi-head CNN: This component extracts local features from each horizontal strip.
- Convolutional RNN (GRU variant): The RNN maintains a hidden state that captures vertical context as strips are processed. This "memory" allows the network to understand the whole image without ever storing its full activation maps.
Attention and Early Termination
RaScaNet employs spatial and channel attention, using the previous RNN hidden state to highlight relevant features in the current strip. Furthermore, the Early Termination (RaScaNet-ET) variant allows the model to stop scanning once a confidence threshold is met, saving significant computational cycles (MACs) when a subject is identified early in the scan.
Key Results and Memory Efficiency
RaScaNet achieves a total footprint (peak memory + weights) of under 60 KB, allowing it to reside entirely within on-chip SRAM and cache.
- Memory Savings: Compared to MCUNet, RaScaNet reduces peak memory by up to 24.3× and weight memory by up to 12.9×.
- Accuracy: On the Visual Wake Words (VWW) dataset, RaScaNet maintains a high accuracy of ~91.6%, proving that sequential processing does not compromise performance.
- Efficiency: The Early Termination strategy reduces average MACs by 17.1% without accuracy loss.
Conclusion
RaScaNet demonstrates that hardware-algorithm co-design—specifically aligning neural network operations with the raster-scan nature of image sensors—can overcome the fundamental memory limits of ultra-low-power systems. This approach paves the way for sophisticated "on-sensor" AI applications that are both highly efficient and accurate.
Summary updated