Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tiles

Renzo Andri, Beatrice Bussolino, Antonio Cipolletta, Lukas Cavigelli, Zhe Wang

TL;DR

This paper tackles the challenge of efficiently running Winograd convolutions (specifically the F4 variant) on integer-based hardware accelerators. They introduce a novel "tap-wise quantization" method to overcome numerical instability and design custom hardware modules to accelerate the Winograd transformations, achieving significant speedups and energy efficiency gains with minimal accuracy loss.

What Problem Does This Paper Address?

Deep neural networks, particularly convolutional neural networks (CNNs), are computationally intensive, with convolution operations being a major bottleneck. The Winograd convolution algorithm offers a way to reduce the number of multiply-accumulate (MAC) operations required for convolutions. While the F2 variant (2x2 tiles) provides a modest speedup, the F4 variant (4x4 tiles) promises a much larger reduction in MACs (4x). However, the F4 algorithm suffers from numerical instability when using standard integer arithmetic, making it unsuitable for efficient integer-only hardware accelerators (DSAs). Additionally, the transformations required by Winograd introduce computational overhead. This paper aims to unlock the full potential of Winograd F4 by addressing these numerical and hardware challenges for efficient inference on DSAs.

Methods

The core contributions of this paper are:

  • Tap-wise Quantization: Instead of applying a single quantization scale factor across all elements in the Winograd domain, this method quantizes each "tap" (element) independently. This is crucial because the dynamic range of these elements varies significantly across different taps, as shown in Figure 1. This approach allows for integer-only inference without significant accuracy degradation.
  • Winograd-Aware Training: The authors employ a training methodology that propagates gradients through the Winograd domain to improve the accuracy of quantized networks. They also explore "learned power-of-two quantization" where scaling factors are learned to be powers of two, enabling efficient shift-and-add operations.
  • Custom Hardware Units: To address the computational overhead of Winograd transformations, the paper proposes custom hardware modules for processing these transformations efficiently. They explore two strategies: "row-by-row" and "tap-by-tap" engines, optimizing for area and power.
  • DSA Integration: These custom hardware units are integrated into an industrial-grade, programmable Domain-Specific Accelerator (DSA).

Datasets

The following datasets were used for training and evaluation:

  • CIFAR-10: Used for training and evaluating ResNet-20 and a light version of VGG.
  • ImageNet ILSVRC: Used for training and evaluating ResNet-34 and ResNet-50.

Key Insights and Ideas

  • The "Tap" is Key: The most significant insight is that the numerical issues with Winograd F4 stem from the highly non-uniform dynamic range of elements in the Winograd domain. A single quantization scale factor is insufficient; independent "tap-wise" quantization is necessary.
  • Hardware Co-design is Crucial: Simply having a more efficient algorithm (Winograd F4) isn't enough. The paper highlights the need for custom hardware to efficiently handle the Winograd transformation steps, which are otherwise a bottleneck.
  • Power-of-Two Scaling is Practical: Learning scaling factors that are powers of two allows for efficient hardware implementation using shifts and adds, avoiding costly multiplications.
  • Winograd F4 Benefits from Bandwidth: While Winograd F4 reduces computation, its performance can still be limited by external memory bandwidth, especially when the transformed weights cannot be fully stored on-chip.

Key Results

  • Accuracy: The tap-wise quantized Winograd F4 network achieves accuracy comparable to the FP32 baseline. For example, ResNet-34 on ImageNet shows only a 0.6% drop in Top-1 accuracy with the proposed method (72.0% vs. 72.6% FP32).
  • Speedup: The Winograd-enhanced DSA achieves up to 1.83x end-to-end speed-up for state-of-the-art segmentation and detection networks. For compute-intensive 3x3 convolutional layers, speedups of up to 3.42x were observed.
  • Energy Efficiency: The system achieves up to 1.85x gain in energy efficiency. The compute datapath is shown to be approximately 3x more energy efficient with Winograd F4 compared to the im2col approach.
  • Hardware Overhead: The proposed Winograd extensions add only about 6.1% to the core area and 17% to the power consumption of the MatMul engine.
  • Comparison with NVDLA: The proposed accelerator outperforms NVDLA with its Winograd F2 extension by 1.5 to 3.3x at the same peak throughput and external bandwidth.

Limitations and Future Work

  • Not All Layers Benefit: The paper acknowledges that not all convolutional layers can be efficiently implemented with the Winograd algorithm (e.g., 1x1 convolutions). The proposed system selectively replaces 3x3 layers.
  • Tiling Constraints: The spatial resolution of output activation tiles being a multiple of 4 can limit tiling choices and may require zero-padding.
  • Memory Bandwidth: While optimized, external memory bandwidth remains a factor influencing performance, especially for workloads with limited on-chip storage of transformed weights.
  • Future Work: The authors suggest exploring combining pruning with tap-wise quantization and assessing its benefits on a hardware accelerator. They also mention that the methodology can guide DSA designers in integrating Winograd support into other accelerators.

Summary updated

Abstract

Most of today’s computer vision pipelines are built around deep neural networks, where convolution operations require most of the generally high compute effort. The Winograd convolution algorithm computes convolutions with fewer multiply–accumulate operations (MACs) compared to the standard algorithm, reducing the operation count by a factor of 2.25x for 3x3 convolutions when using the version with 2x2-sized tiles F2. Even though the gain is significant, the Winograd algorithm with larger tile sizes, i.e., F4, offers even more potential in improving throughput and energy efficiency, as it reduces the required MACs by 4x. Unfortunately, the Winograd algorithm with larger tile sizes introduces numerical issues that prevent its use on integer domain-specific accelerators (DSAs) and higher computational overhead to transform input and output data between spatial and Winograd domains. To unlock the full potential of Winograd F4, we propose a novel tap-wise quantization method that overcomes the numerical issues of using larger tiles, enabling integer-only inference. Moreover, we present custom hardware units that process the Winograd transformations in a power- and area-efficient way, and we show how to integrate such custom modules in an industrial-grade, programmable DSA. An extensive experimental evaluation on a large set of state-of-the-art computer vision benchmarks reveals that the tap-wise quantization algorithm makes the quantized Winograd F4 network almost as accurate as the FP32 baseline. The Winograd-enhanced DSA achieves up to 1.85x gain in energy efficiency and up to 1.83x end-to-end speed-up for state-of-the-art segmentation and detection networks.
BibTeX
@inproceedings{andri2022going,
  author = {Renzo Andri and Beatrice Bussolino and Antonio Cipolletta and Lukas Cavigelli and Zhe Wang},
  title = {{Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tiles}},
  booktitle = {MICRO},
  year = {2022},
  url = {https://openreview.net/forum?id=TVGG4a8fYu},
}
Figure