LPRNet: Lightweight Deep Network by Low-rank Pointwise Residual Convolution

Bin Sun, Jun Li, Ming Shao, Yun FuarXiv 2019 · arXiv

Abstract

LPRNet introduces a novel Low-rank Pointwise Residual (LPR) convolution module designed to optimize deep neural networks for mobile and embedded deployment. By decomposing computationally intensive 1x1 pointwise convolutions into low-rank matrices and incorporating a specialized residual path, LPRNet achieves significant reductions in FLOPs and parameter counts. Experimental results on ImageNet and 3D face alignment tasks demonstrate that LPRNet maintains competitive accuracy while offering substantial efficiency gains over standard Depthwise Separable Convolution (DSC) architectures.

Introduction

The deployment of Deep Convolutional Neural Networks (DCNNs) on resource-constrained hardware remains a significant challenge due to their high computational and memory demands. While Depthwise Separable Convolutions (DSC) have successfully reduced spatial redundancy, the 1x1 pointwise convolutions have emerged as the primary bottleneck in modern lightweight architectures like MobileNet and ShuffleNet. LPRNet addresses this by further compressing these pointwise layers through low-rank approximation and error-correcting residuals.

The Low-rank Pointwise Residual (LPR) Module

The core innovation of LPRNet is the LPR module, which evolves the standard DSC approach. As illustrated in Figure 1, standard convolutions are first replaced by DSC to separate spatial and channel-wise operations. LPRNet then targets the pointwise step.

Conceptual comparison of (a) Standard Convolution, (b) Depthwise Separable Convolution (DSC), and (c) the proposed Low-rank Pointwise Residual (LPR) convolution.

Low-Rank Decomposition

The pointwise convolution matrix $P$ is treated as a large weight matrix and decomposed into two smaller, low-rank matrices ($P^{(1)}$ and $P^{(2)}$). For an $m \times n$ matrix, choosing a rank $r \ll m, n$ reduces the parameter count from $m \times n$ to $(m+n) \times r$. The authors found that a rank of $r = m/8$ provides an optimal balance between compression and representational capacity.

Pointwise Residual Connection and L2 Normalization

To mitigate the information loss inherent in low-rank approximations, LPRNet introduces a residual path that carries features from the depthwise layer directly to the output. This "rectifies" the compressed features, preserving spatial identity. Additionally, L2 normalization is applied across channels before the final summation to stabilize training and enhance performance.

Architectural Integration

The LPR module is designed as a "plug-and-play" component. It can replace standard blocks in existing architectures such as MobileNetV1 and ShuffleNetV2. Figure 2 shows the detailed implementation of the LPR block compared to its predecessors.

Experimental Results

The researchers evaluated LPRNet across two diverse domains:

  • Image Classification (ImageNet 2012): An LPRNet variant based on MobileNet achieved 70.6% Top-1 accuracy—identical to the original MobileNetV1—while utilizing 55% fewer FLOPs (260M vs 574M) and 45% fewer parameters (2.3M vs 4.2M).
  • 3D Face Alignment: On challenging datasets like AFLW2000-3D, LPRNet demonstrated exceptional efficiency. A compressed LPRNet-MobileNet x0.25 model achieved real-time performance of 95 FPS on a standard Intel-i7 CPU, significantly outperforming heavier state-of-the-art models in speed while maintaining high accuracy.

Conclusion

LPRNet demonstrates that the pointwise bottleneck in lightweight networks can be effectively addressed through low-rank decomposition combined with residual error correction. By significantly reducing the computational footprint without sacrificing accuracy, LPRNet provides a robust framework for deploying advanced computer vision models on edge devices. Future work may explore automated rank selection through Neural Architecture Search (NAS) to further optimize layer-specific compression.

Summary updated

Abstract

Deep learning has become popular in recent years primarily due to the powerful computing device such as GPUs. However, deploying these deep models to end-user devices, smart phones, or embedded systems with limited resources is challenging. To reduce the computation and memory costs, we propose a novel lightweight deep learning module by low-rank pointwise residual (LPR) convolution, called LPRNet. Essentially, LPR aims at using low-rank approximation in pointwise convolution to further reduce the module size, while keeping depthwise convolutions as the residual module to rectify the LPR module. This is critical when the low-rankness undermines the convolution process. We embody our design by replacing modules of identical input-output dimension in MobileNet and ShuffleNetv2. Experiments on visual recognition tasks including image classification and face alignment on popular benchmarks show that our LPRNet achieves competitive performance but with significant reduction of Flops and memory cost compared to the state-of-the-art deep models focusing on model compression.
BibTeX
@misc{sun2019lprnet,
  author = {Bin Sun and Jun Li and Ming Shao and Yun Fu},
  title = {{LPRNet: Lightweight Deep Network by Low-rank Pointwise Residual Convolution}},
  year = {2019},
  eprint = {1910.11853},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.11853},
}
Figure