Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networks

Kartik Gupta, Akshay Asthana

Abstract

Training ultra-low precision (3-bit and 4-bit) YOLO models is notoriously difficult due to weight and scale factor "jitter" or oscillations during training, which leads to significant accuracy degradation. This paper introduces two primary interventions: Exponential Moving Average (EMA) to stabilize training and a one-epoch "Quantization Correction" (QC) step to refine the final model. These methods effectively close the performance gap between ultra-low precision and full-precision object detectors on standard benchmarks.

Introduction: The Challenge of Low-Bit YOLO

Quantization-Aware Training (QAT) is essential for deploying deep learning models on edge devices with limited resources. By reducing 32-bit floating-point parameters to 3 or 4 bits, models can be significantly compressed. However, QAT typically relies on the Straight Through Estimator (STE) to approximate gradients for non-differentiable rounding operations.

In compact architectures like YOLO, the STE frequently induces oscillations. Weights and learnable scale factors fail to converge, instead jumping across quantization boundaries. This instability prevents the model from reaching an optimal state, resulting in a substantial accuracy penalty compared to full-precision counterparts.

Oscillations of latent weights and quantization thresholds during training steps, illustrating the instability caused by the Straight Through Estimator.

Key Insights: Beyond Weight Oscillations

A critical finding of this work is that oscillations are not limited to weights. Learnable scale factors for activations—which define the quantization range—also exhibit significant jitter. This instability in activation quantization is a major contributor to performance loss in YOLO architectures.

The activation quantization threshold exhibits persistent oscillations throughout the training process, hindering convergence.

Furthermore, analysis suggests that latent weights oscillating near boundaries may actually be closer to an "optimal" continuous state than if they were strictly forced to the nearest integer. This observation highlights the limitations of rigid per-tensor quantization schemes and the need for more flexible correction mechanisms.

Proposed Methodology

The authors propose a two-stage stabilization and correction pipeline:

  1. Exponential Moving Average (EMA): To mitigate noise, EMA is applied to both latent weights and learnable scale factors during training. This acts as a low-pass filter, smoothening the trajectories of these parameters and helping the model settle into a more stable, representative configuration.
  2. Post-hoc Quantization Correction (QC): Following standard QAT, a lightweight "tune-up" is performed. A learnable affine (scale and shift) transformation is added to the pre-activations of every layer. With the rest of the network frozen, these correction parameters are trained for a single epoch on a small calibration set. This step compensates for residual errors from weights that remained near quantization boundaries. Crucially, these parameters can be folded into existing Batch Normalization layers, incurring zero additional inference cost.

Experimental Results

The methodology was evaluated using the COCO dataset across various YOLOv5 and YOLOv7 models. The combination of EMA and QC consistently outperformed standard LSQ-based QAT:

  • YOLOv5-n (3-bit): Achieved a Mean Average Precision (mAP) of 18.2, up from 15.2 with standard methods.
  • YOLOv5-s (4-bit): Reached 34.0 mAP, coming within 3.4 points of the full-precision baseline (37.4 mAP).
  • Generalization: The improvements were robust across both object detection and semantic segmentation tasks, setting a new state-of-the-art for low-precision YOLO.

Conclusion and Future Work

This work demonstrates that taming oscillations in both weights and scale factors is key to effective low-bit quantization for YOLO. While the current per-channel QC is highly effective, future research may explore more granular correction factors or automated estimation techniques to further enhance accuracy. Additionally, applying this stabilization pipeline to other architectures like Transformers could broaden its impact on efficient deep learning.

Summary updated

Abstract

Quantized networks use less computational and memory resources and are suitable for deployment on edge devices. While quantization-aware training QAT is the well-studied approach to quantize the networks at low precision, most research focuses on over-parameterized networks for classification with limited studies on popular and edge device friendly single-shot object detection and semantic segmentation methods like YOLO. Moreover, majority of QAT methods rely on Straight-through Estimator (STE) approximation which suffers from an oscillation phenomenon resulting in sub-optimal network quantization. In this paper, we show that it is difficult to achieve extremely low precision (4-bit and lower) for efficient YOLO models even with SOTA QAT methods due to oscillation issue and existing methods to overcome this problem are not effective on these models. To mitigate the effect of oscillation, we first propose Exponentially Moving Average (EMA) based update to the QAT model. Further, we propose a simple QAT correction method, namely QC, that takes only a single epoch of training after standard QAT procedure to correct the error induced by oscillating weights and activations resulting in a more accurate quantized model. With extensive evaluation on COCO dataset using various YOLO5 and YOLO7 variants, we show that our correction method improves quantized YOLO networks consistently on both object detection and segmentation tasks at low-precision (4-bit and 3-bit).
BibTeX
@inproceedings{gupta2024reducing,
  author = {Kartik Gupta and Akshay Asthana},
  title = {{Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00244},
}
Figure