ClimbQ: Class Imbalanced Quantization Enabling Robustness on Efficient Inferences

Ting-An Chen, De-Nian Yang, Ming-Syan ChenNeurIPS 2022 · OpenReview

TL;DR

ClimbQ is a novel quantization framework designed specifically for class-imbalanced (long-tailed) datasets. By diminishing inter-class heterogeneity through variance scaling and a statistically-grounded "HomoVar Loss," it prevents majority classes from dominating the quantization process. This approach leads to massive accuracy gains for minority classes in low-bit models without sacrificing overall performance.

The Problem: Quantization on Long-Tailed Data

Deep learning models are often too computationally heavy for mobile or IoT devices. Quantization—the process of compressing 32-bit floating-point weights and activations into low-bit integers (like 2-bit or 4-bit)—is a standard solution to reduce memory and accelerate inference. However, most quantization research assumes "balanced" datasets where every class has roughly the same number of examples.

In the real world, data is almost always "long-tailed" or highly imbalanced. The authors of ClimbQ discovered that standard quantization fails miserably in these realistic scenarios. Because majority classes have more data and higher variance, the quantization parameters (like clipping ranges) are naturally tuned to fit them.

Impact of class imbalance on quantization accuracy and feature representation.

This "majority rules" dynamic leaves minority classes with massive quantization errors. As shown in the figure above, minority classes (e.g., classes 8 and 9) suffer accuracy drops as high as 14% for 2-bit models, even when the overall model accuracy looks acceptable. The feature space for these minority classes becomes squashed, destroying their discriminative features.

The ClimbQ Solution

To address this, the authors propose a two-part framework called ClimbQ (Class IMBalanced Quantization) that focuses on diminishing inter-class heterogeneity.

The ClimbQ framework architecture, featuring Distribution Scaling and HomoVar Loss.

1. Distribution Scaling & Projection

Instead of quantizing raw features directly, ClimbQ first scales the variance of each class. It "shrinks" the wide distributions of the majority classes and "stretches" the narrow distributions of the minority classes. It then uses a Cumulative Distribution Function (CDF) to project these adjusted distributions into a uniform space $[0, 1]$. This ensures that the subsequent uniform quantization step treats all classes more fairly, preventing majority classes from hogging all the available quantization "bins."

2. HomoVar Loss (Homogeneous Variance Loss)

To keep class variances consistent during training, the authors turned to statistics. Using Levene’s test (a standard statistical tool used to check if different groups have equal variances), they derived mathematical bounds ($n_L$ and $n_U$) for ideal class sizes. Based on this, they designed a reweighting loss—the HomoVar Loss. This loss penalizes minority classes more heavily to force the model to learn them better, while downweighting "enormous" majority classes that exceed the statistical upper bound. This provides a rigorous theoretical justification for their loss function, moving beyond the simple heuristics often found in reweighting papers.

Key Insights

  • Quantization is a "Zero-Sum" Game: In standard methods, the "bins" used to represent numbers are a finite resource. If the majority classes have high variance, they consume the available bins, leaving the minority classes to be rounded to the same few values.
  • Variance is the Culprit: It’s not just that there are fewer minority samples; it’s that their distributions are physically different. ClimbQ’s core insight is that we must force inter-class homogeneity (making classes look similar in distribution) before we can quantize them effectively.

Experimental Results

The researchers evaluated their method on several "Long-Tailed" (LT) versions of standard benchmarks, including CIFAR-10-LT, CIFAR-100-LT, and Syndigit-LT, with imbalance ratios ($\gamma$) up to 200 (where the largest class is 200x larger than the smallest).

ClimbQ+ (the full version using the HomoVar Loss) significantly outperformed state-of-the-art (SOTA) methods like LLSQ and BatchQuant:

  • MobileNet-V2 on CIFAR-100-LT: For a highly constrained 2-bit model with a severe imbalance ratio ($\gamma=200$), ClimbQ+ achieved 25.45% accuracy, while many SOTA methods failed to converge entirely or stagnated near 10-13%.
  • ResNet-56: On highly imbalanced data, ClimbQ+ improved accuracy by roughly 6% to 12% over previous quantization techniques.
  • Robustness on Balanced Data: Even on perfectly balanced datasets like standard ImageNet, ClimbQ+ remained highly competitive, proving it is a robust, general-purpose quantization strategy that doesn't sacrifice baseline performance.

Limitations and Future Work

The authors note a potential security trade-off: because the model is forced to pay more attention to minority classes with very little data, it might become more susceptible to adversarial attacks targeting those specific classes.

Additionally, while the paper focuses heavily on class imbalance, other types of data heterogeneity—such as domain shifts or non-Gaussian distributions—remain an open challenge. Exploring quantization on these other forms of "messy" real-world data is the next logical step for robust, efficient AI.

Summary updated

Abstract

Quantization compresses models to low bits for efficient inferences which has received increasing attentions. However, existing approaches focused on balanced datasets, while imbalanced data is pervasive in the real world. Therefore, in this study, we investigate the realistic problem, quantization on class-imbalanced data. We observe from the analytical results that quantizing imbalanced data tends to obtain a large error due to the differences between separate class distributions, which leads to a significant accuracy loss. To address this issue, we propose a novel quantization framework, Class Imbalanced Quantization (ClimbQ) that focuses on diminishing the inter-class heterogeneity for quantization error reduction. ClimbQ first scales the variance of each class distribution and then projects data through the new distributions to the same space for quantization. To guarantee the homogeneity of class variances after the ClimbQ process, we examine the quantized features and derive that the homogeneity satisfies when data size for each class is restricted (bounded). Accordingly, we design a Homogeneous Variance Loss (HomoVar Loss) which reweights the data losses of each class based on the bounded data sizes to satisfy the homogeneity of class variances. Extensive experiments on class-imbalanced and benchmark balanced datasets reveal that ClimbQ outperforms the state-of-the-art quantization techniques, especially on highly imbalanced data.
BibTeX
@inproceedings{chen2022climbq,
  author = {Ting-An Chen and De-Nian Yang and Ming-Syan Chen},
  title = {{ClimbQ: Class Imbalanced Quantization Enabling Robustness on Efficient Inferences}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=dBYwRfjydb},
}

Loading related content...

Figure