TL;DR
This paper introduces Low-Rank Representation Vector Quantization (LR2VQ), a novel method for compressing deep neural networks. LR2VQ decouples the compression ratio from the clustering dimensionality, allowing for more effective compression by learning low-rank representations of convolutional filters before applying vector quantization. This approach achieves state-of-the-art compression performance with minimal accuracy loss.
What Problem Does This Paper Address?
Deep neural networks have achieved remarkable success in various computer vision tasks, but their large memory footprint and computational overhead make them challenging to deploy on resource-constrained devices. Model compression techniques aim to reduce these requirements. Vector Quantization (VQ) is a popular method that compresses models by replacing original weights with codes from a smaller codebook. While existing VQ methods have focused on optimizing codebooks and codes under a fixed clustering dimensionality, they often overlook the impact of dimensionality itself on clustering performance. High-dimensional clustering is prone to the "curse of dimensionality," leading to increased clustering errors and, consequently, accuracy loss. This paper addresses the need for a VQ method that can effectively manage clustering dimensionality to improve compression efficiency and accuracy.
Methods
The proposed method, Low-Rank Representation Vector Quantization (LR2VQ), consists of three main steps:
- Learning Low-Rank Representations (LRRs): Instead of directly quantizing the original convolutional filters (represented as matrices $W$), LR2VQ approximates them using two matrices, $A$ and $B$, such that $W \approx A \times B$. Here, $A$ is the low-rank representation (LRR) and $B$ is a linear transformation. Crucially, $A$ and $B$ are learned end-to-end, making the LRR adaptive to the specific network and task. This step allows for decoupling the subvector size ($m$) from the clustering dimensionality ($\tilde{d}$).
- Quantizing Low-Rank Representations: Vector quantization is then applied to the learned LRRs ($A$). The subvectors within $A$ are clustered using a method like k-means. The key innovation here is that the clustering dimensionality ($\tilde{d}$) can be chosen independently of the subvector size ($m$). This allows for clustering in a lower-dimensional space, which is theoretically less susceptible to the curse of dimensionality and thus reduces clustering error.
- Global Fine-tuning: After quantization, the compressed network is fine-tuned on the task loss to recover any accuracy lost during compression. The paper also proposes a method to eliminate the linear transformation matrix $B$ during inference by merging it with the codebook, resulting in a more efficient computation using a simple look-up table.
Datasets
The paper evaluates LR2VQ on the following datasets:
- ImageNet: Used for evaluating classification performance on ResNet-18 and ResNet-50 architectures.
- COCO: Used for evaluating object detection and segmentation performance on Mask R-CNN.
The authors re-implement PQF and vanilla DKM baselines under the same code base for fair comparison, using standard configurations for these datasets.
Key Insights and Ideas
- Decoupling Compression Ratio and Clustering Dimensionality: The core idea is that the compression ratio is primarily determined by the subvector size ($m$), while the quantization accuracy is heavily influenced by the clustering dimensionality ($\tilde{d}$). By learning LRRs, LR2VQ separates these two factors, allowing for independent optimization.
- The Trade-off of $\tilde{d}$: The paper highlights a critical trade-off: increasing $\tilde{d}$ reduces the approximation error ($E_a$) of the LRR but increases the clustering error ($E_c$), and vice-versa. This inverse correlation suggests an optimal $\tilde{d}$ exists that minimizes the overall reconstruction error ($E_r$).
- Lower Intrinsic Dimensionality: LRRs learned by the method exhibit lower intrinsic dimensionality compared to the original filters, even when $\tilde{d}$ equals the subvector size $m$. This suggests that LRRs can automatically learn more compact representations that are more amenable to clustering.
- Analytical Estimation of $\tilde{d}$: The paper proposes a theoretical analysis and an empirical method to estimate a suitable $\tilde{d}$ by minimizing the determinant of the covariance matrix of the subvectors, which is related to clustering error. This helps reduce the computational overhead of searching for the optimal $\tilde{d}$.
Key Results
- ImageNet Classification:
- On ResNet-18 with a 43x compression factor, LR2VQ achieved a 2.8% higher top-1 accuracy than PQF.
- On ResNet-50 with a 31x compression factor, LR2VQ achieved a 1.0% higher top-1 accuracy than PQF.
- The optimal $\tilde{d}$ values found were significantly lower than the subvector sizes, confirming the benefit of low-dimensional clustering.
- COCO Object Detection:
- For Mask R-CNN with a 26x compression factor, LR2VQ achieved 0.91% higher box AP than PQF.
- Generalizability: LR2VQ demonstrates strong performance across different datasets (ImageNet, COCO) and architectures (ResNet, Mask R-CNN), indicating its broad applicability.
Limitations and Future Work
- Limitations:
- The paper notes that while the LRR pre-trained networks often approach the accuracy of the original uncompressed network, there can still be a gap, especially when the subvector size is limited. This is attributed to the inherent limitations of the LRR learning process.
- The analytical method for estimating $\tilde{d}$ is presented as a coarse estimation and a starting point for further search.
- Future Work:
- The authors suggest that the findings on the trade-off of $\tilde{d}$ and the benefits of low-dimensional clustering provide a solid foundation for future research in vector quantization.
- The paper implicitly suggests exploring more advanced clustering methods or adapting existing ones to leverage the learned LRRs more effectively.
Summary updated