TL;DR
This paper introduces a novel, fused Winograd Convolution algorithm specifically optimized for ARMv8 processors. By tightly integrating all stages of the Winograd process and leveraging AArch64 assembly, they achieve significant speedups (up to 10.57x) over existing libraries on various ARM platforms, making deep learning inference more efficient on these widely used architectures.
What Problem Does This Paper Address?
Convolutional Neural Networks (CNNs) are computationally intensive, especially their convolution layers. Winograd convolution offers a way to reduce the number of multiplications needed, boosting efficiency. However, existing Winograd implementations on ARM processors struggle with issues like:
- Fragmented processing: Treating input/filter transformations, matrix multiplication, and output transformations as separate steps leads to inefficient cache usage.
- Transformation overhead: Converting data to and from the Winograd domain involves costly memory accesses.
- Suboptimal parallelization: Existing methods don't adapt well to the varying scales of different convolution layers.
- ARMv8 specific challenges: Leveraging the full potential of ARMv8's architecture, particularly its SIMD capabilities and memory hierarchy, requires careful, low-level optimization.
These challenges limit the performance of CNNs on ARM devices, which are prevalent in mobile and increasingly in high-performance computing.
Methods
The authors propose a "fused" Winograd Convolution algorithm that treats the entire process as a single pipeline. Key innovations include:
- Integrated Pipeline: Input transformation, filter transformation, matrix multiplication (GEMM), and output transformation are tightly coupled. This minimizes intermediate data movement and improves cache locality.
- AArch64 Assembly Optimization: All core kernels are hand-optimized in AArch64 assembly. This allows for precise control over registers, software prefetching to mitigate memory access latency, and fine-tuning of blocking parameters, surpassing what compiler intrinsics can achieve.
- Custom Data Layout: A "z-shaped" data layout is used for transformed data. This ensures consecutive memory access for the GEMM micro-kernel, further boosting efficiency.
- Optimized GEMM Micro-kernel: A "ping-pong" technique is employed within the GEMM micro-kernel to keep the pipeline full and maximize throughput.
- Multi-dimensional Parallel Strategy: An adaptive strategy dynamically chooses which dimensions (N, T, C/K) to parallelize based on the layer's characteristics, maximizing parallel efficiency across diverse network structures.
- Dual Micro-kernel Strategy: To handle edge cases arising from varying layer dimensions (especially as T decreases), they offer two micro-kernels with different parameter sets, balancing arithmetic intensity and edge-case handling.
Datasets
The paper evaluates its method on benchmark networks, specifically VGG-16 and FusionNet, which represent a range of network scales. The experiments use these networks to demonstrate performance across different convolution layers. No new datasets were introduced.
Key Insights and Ideas
- Fusing is Key: The most significant insight is that tightly integrating all Winograd stages, rather than treating them separately, is crucial for unlocking performance on modern architectures like ARMv8. This minimizes overhead and maximizes cache utilization.
- Assembly Matters for ARM: For architectures like ARMv8, hand-tuned assembly provides a level of control over hardware features (like SIMD registers and prefetching) that is essential for squeezing out maximum performance, especially for computationally intensive tasks like Winograd convolution.
- Data Layout is Critical for GEMM: A custom, "z-shaped" data layout is not just a minor tweak; it's fundamental to making the GEMM micro-kernel efficient by ensuring predictable, sequential memory access.
- Adaptive Parallelism is a Must: Convolutional layers vary wildly in their dimensions (channels, height, width). A one-size-fits-all parallelization strategy won't work; an adaptive, multi-dimensional approach is needed to balance workloads effectively.
- Edge Cases are Real: The paper acknowledges that Winograd's efficiency can be impacted by specific layer dimensions. Offering dual micro-kernels is a smart way to handle these edge cases without sacrificing overall performance.
Key Results
The proposed fused Winograd Convolution algorithm achieves substantial speedups across multiple ARMv8 platforms:
- Kunpeng 920: Up to 10.57x speedup over ACL, 4.74x over NCNN, 4.10x over NNPACK, and 4.72x over FastConv (with multiple threads).
- AWS Graviton2: Up to 7.80x speedup over ACL, 3.85x over NCNN, 2.81x over NNPACK, and 4.20x over FastConv.
- Phytium 2000+: Up to 9.28x speedup over ACL, 3.32x over NCNN, 3.68x over NNPACK, and 8.00x over FastConv.
- Single-Core Efficiency: The method achieves up to 94.81% of the Kunpeng 920's theoretical peak performance for F(2x2, 3x3) on certain layers, with GEMM routines maintaining over 90% efficiency.
- LLC Miss Rates: The approach consistently shows lower Last Level Cache (LLC) miss rates compared to other libraries, indicating more efficient cache utilization.
Limitations and Future Work
- Limitations:
- The paper notes that for smaller layers (e.g., VggNet5.2), the limited computational load can lead to memory-bound scenarios, reducing GFlop/s and hardware utilization, especially for larger Winograd configurations (F(4x4, 3x3) and F(6x6, 3x3)).
- While the method is highly efficient, the complexity of hand-optimized assembly increases development effort.
- Future Work:
- Extend the approach to 3-D Winograd Convolution, which presents greater complexity.
- Adapt the method for larger filter kernels.
- Explore applicability to other architectures, such as x86 and RISC-V.
Summary updated