Abstract
We consider the problem of distributed feature quantization, where the goal is to enable a pretrained classifier at a central node to carry out its classification on features that are gathered from distributed nodes through communication constrained channels. We propose the design of distributed quantization schemes specifically tailored to the classification task: unlike quantization schemes that help the central node reconstruct the original signal as accurately as possible, our focus is not reconstruction accuracy, but instead correct classification. Our work does not make any apriori distributional assumptions on the data, but instead uses training data for the quantizer design. Our main contributions include: we prove NP-hardness of finding optimal quantizers in the general case; we design an optimal scheme for a special case; we propose quantization algorithms, that leverage discrete neural representations and training data, and can be designed in polynomial-time for any number of features, any number of classes, and arbitrary division of features across the distributed nodes. We find that tailoring the quantizers to the classification task can offer significant savings: as compared to alternatives, we can achieve more than a factor of two reduction in terms of the number of bits communicated, for the same classification accuracy.
Introduction
In the era of the Internet of Things (IoT) and distributed sensing, multiple nodes often collect data that must be transmitted to a central "brain" for analysis. A critical bottleneck in these systems is the communication constraint; transmitting high-precision raw data is expensive in terms of power and bandwidth. To address this, we use quantization—mapping continuous signal values to a small set of discrete bits.
Traditionally, quantization is designed for reconstruction, aiming to minimize distortion so the received signal looks as much like the original as possible. However, if the central node's only task is classification (e.g., "is this gesture A or B?"), perfect reconstruction is unnecessary. This paper argues that focusing on decision boundaries rather than signal fidelity allows for significant communication savings.
Methodology
The core philosophy of this work is that the decision boundary is everything. Traditional quantization methods, like k-means, place more resolution where the data is most dense. In contrast, classification-aware quantization concentrates resolution near the decision boundaries. If a data point is far from a boundary, the classifier needs very little detail to categorize it correctly.
Greedy Boundary Insertion (GBI)
We propose a data-driven algorithm called Greedy Boundary Insertion. GBI iteratively adds quantization boundaries where they most effectively reduce misclassification. A key innovation is the "Purity Criterion," a tie-breaking rule that favors boundaries creating homogeneous class regions, making it easier for subsequent iterations to separate classes.
Neural Representations
To handle complex data, we utilize two neural network-based approaches:
- NN-REG: Encoders are trained with a custom quantization loss that nudges outputs toward discrete levels, avoiding the "dead gradient" problem common in discrete optimization.
- NN-GBI: A hybrid approach where a continuous representation is learned first, followed by GBI to find optimal boundaries, and finally end-to-end fine-tuning.
Experimental Results
We evaluated our methods on two distinct datasets: surface electromyographic (sEMG) signals for hand gesture recognition and the CIFAR-10 image benchmark. In both cases, the features were distributed across multiple non-communicating nodes.
- Massive Bit Savings: On the sEMG dataset, the proposed NN-GBI method achieved 97.3% accuracy using only 4 bits per encoder. Traditional reconstruction-based k-means reached only 66.7% with the same bit budget.
- Efficiency Gains: On CIFAR-10, to reach ~50% accuracy, a centralized VQ-VAE required 200 bits. Our proposed methods achieved over 70% accuracy with only 12 bits total (3 bits per quadrant).
These results highlight that reconstruction-based methods often waste their limited bit budget on textures and colors that do not help distinguish between classes, whereas classification-aware methods remain robust even at extremely low bit rates.
Conclusion and Future Work
By tailoring quantization to the classification task, we can achieve more than a factor of two reduction in communication bandwidth for the same accuracy. While our current GBI algorithm creates axis-aligned quantization regions, future research will explore more complex, non-rectangular distributed boundaries and joint optimization where the classifier and distributed quantizers are trained simultaneously from scratch.
Summary updated