TL;DR
This paper introduces CATS, a novel training approach that combines spatial (activation) and temporal event suppression to significantly reduce the computational load of deep neural networks (DNNs) on event-driven processors. By intelligently learning optimal suppression strategies, CATS achieves up to 6x higher event suppression than existing methods, leading to substantial improvements in inference speed, latency, and energy efficiency on specialized hardware.
What Problem Does This Paper Address?
Deep Neural Networks (DNNs) are powerful but computationally demanding, creating a mismatch with the needs of edge devices and specialized hardware that require low power consumption, high efficiency, and minimal latency. Event-driven processors, inspired by the brain's efficiency, offer a promising alternative by processing only "events" (changes in data) rather than entire frames. However, maximizing the efficiency of these processors requires further reducing the number of events processed. Prior work has focused on either spatial suppression (eliminating redundant activations within a single frame) or temporal suppression (eliminating redundant information across frames). This paper addresses the gap that simultaneously enhancing both types of suppression can lead to synergistic gains, a phenomenon not fully explored or exploited by existing methods.
Methods
The core of this work is the CATS (Combined Activation and Temporal Suppression) training approach. CATS integrates two key novel components:
- Differentiable Temporal Threshold Search (DTTS): This method automates the learning of optimal layer-wise temporal thresholds. Instead of manually tuning these thresholds, DTTS uses a differentiable approach inspired by neural architecture search. It defines a search space of power-of-two thresholds and uses a Gumbel-Softmax trick to allow gradients to flow through the selection process, enabling end-to-end training. This makes the thresholds hardware-friendly and avoids the manual effort and untrainable nature of previous methods.
- Combined Suppression: CATS synergistically combines DTTS with an enhanced activation suppression technique (STAR). The training objective is a weighted sum of the task loss, a sigma (spatial) sparsity penalty, and a delta (temporal) sparsity penalty. This allows the model to learn to suppress events in both spatial and temporal domains simultaneously, leading to greater overall event reduction than either method alone. Bayesian Optimization is used to efficiently find the optimal balance between these loss components.
Datasets
The paper uses the following datasets for training and evaluation:
- EgoHands: Used for object detection (MobileNetv1-SSD).
- MOT17: Used for object tracking (FairMOT-yolov5s).
- MPII: Used for pose estimation (MobileNetv1-PoseNet).
- Cityscapes: Used for semantic segmentation (ResNet18-DeepLabV3+).
Key Insights and Ideas
- Synergy of Spatial and Temporal Suppression: The most significant insight is that combining activation (spatial) and temporal suppression yields a multiplicative effect on event reduction, far exceeding the gains from either method individually. This is empirically demonstrated and forms the foundation of CATS.
- DTTS for Hardware-Friendly Thresholds: Automating the search for power-of-two temporal thresholds is crucial for efficient hardware implementation. DTTS provides a principled way to learn these thresholds during training, making them directly applicable to event-driven processors.
- Activation Suppression's Role in Temporal Gains: While temporal suppression is key for video data, activation suppression plays a vital role in further reducing events, especially in scenarios with spatial redundancy. It acts as a crucial precursor that amplifies the benefits of temporal suppression.
- End-to-End Training is Superior: Training approaches that incorporate suppression directly into the network training process (like CATS, DAL, Skip-Conv) significantly outperform post-training methods (like Delta-CNN, EvNet) in terms of event suppression.
Key Results
- Event Suppression: CATS achieves 2-6x higher event suppression compared to inherent ReLU suppression and consistently outperforms state-of-the-art methods across various vision tasks.
- Hardware Performance (GrAI-VIP): On the GrAI-VIP event-driven processor, CATS-optimized models show significant improvements:
- 2.5x increase in Frames Per Second (FPS).
- 2.1x reduction in latency.
- 3.8x reduction in energy consumption.
- These improvements are achieved while maintaining model accuracy.
- Comparison to GPUs: CATS on GrAI-VIP demonstrates superior performance compared to Nvidia Jetson Nano with TensorRT acceleration, suggesting event-driven processors can be a more efficient alternative for edge computing.
- Ablation Study: The study confirms that all components of CATS (activation quantization, sigma sparsity penalty, delta sparsity penalty, and differentiable threshold search) are influential in achieving maximal event suppression.
Limitations and Future Work
- Accuracy vs. Suppression Trade-off: While CATS aims to balance accuracy and suppression, aggressive suppression can still lead to minor accuracy drops. The paper uses Bayesian Optimization to find optimal trade-offs, but further research could explore more sophisticated methods for managing this balance.
- Hardware Mapping Complexity: The paper notes that while energy savings are proportional to event/MAC reduction, FPS and latency improvements don't always follow the same trend. This divergence is speculated to be due to hardware mapping complexities and segmentation issues. Future work could investigate how network architecture and hardware constraints interact more deeply.
- Generalization to Diverse Hardware: While evaluated on GrAI-VIP, further testing on a wider range of event-driven processors would be beneficial to confirm the broad applicability of CATS.
Summary updated