Fast Jacobian-Vector Product for Deep Networks

Randall Balestriero, Richard G. BaraniukarXiv 2021 · arXiv

Abstract

Jacobian-vector products (JVPs) are fundamental to modern machine learning tasks, including adversarial robustness assessment, sharpness-aware regularization, and second-order optimization. However, standard automatic differentiation (AD) frameworks often introduce significant computational and memory overhead. We present a novel "Deep Network Cloning" approach specifically for Continuous Piecewise Affine (CPA) networks—such as those utilizing ReLU or Max-pooling—that bypasses AD entirely. By exploiting the local linearity of these architectures, our method achieves a consistent ~2x speedup across a wide range of standard deep learning models while significantly reducing memory consumption.

Introduction

The Jacobian matrix, which captures the sensitivity of model outputs to input perturbations, is a critical tool for understanding and improving deep neural networks. While the full Jacobian is typically too large to compute or store, many applications only require the Jacobian-vector product (JVP). Traditionally, JVPs are computed using forward-mode AD or the "double-backpropagation" trick. Despite their ubiquity, these methods are often computationally expensive and can lead to out-of-memory (OOM) errors, particularly in recurrent architectures.

Methodology: Deep Network Cloning

The proposed method leverages the geometric properties of Continuous Piecewise Affine (CPA) networks. In such networks, the mapping from input to output is locally linear; within a specific region of the input space, the network functions as a simple affine transformation: $f(x) = A_\omega x + b_\omega$, where $A_\omega$ is the Jacobian.

Schematic of the Deep Network Cloning process for computing JVPs.

As illustrated in Figure 1, the "cloning" algorithm operates in two primary phases:

  1. Observation Pass: The input $x$ is passed through the network to determine and "freeze" the states of all piecewise linear activations (e.g., identifying which ReLU units are active).
  2. Cloned Pass: A new vector is passed through a "cloned" version of the network where the nonlinearities are forced to maintain the states recorded in the first pass.

By combining these passes—for instance, by calculating the difference between cloned outputs—the JVP ($A_\omega u$) can be extracted directly. This approach eliminates the need for tracking computational graphs or derivative rules, making it highly efficient and easy to implement in environments without full AD support.

Experimental Results

The efficiency of the cloning method was evaluated across 13 diverse architectures, including RNNs, UNets, and various iterations of VGG, ResNet, and EfficientNet. Benchmarks were conducted on multiple hardware platforms, including NVIDIA Quadro RTX 8000 and TITAN X (Pascal) GPUs.

Comparative performance of the cloning method against standard AD techniques across various architectures.

As shown in the table above (Figure 6), the cloning method consistently outperformed all AD-based alternatives. Key findings include:

  • Speed: The method provides an average speedup factor of approximately 1.7x to 2.1x compared to the fastest alternative AD methods (such as jvp or double-VJP).
  • Memory Efficiency: In Recurrent Neural Networks (RNNs), where AD often fails due to the memory requirements of tracking long-term dependencies, the cloning method remained stable and successfully computed JVPs without OOM errors.
  • Scalability: The performance gains were consistent across different input resolutions ($100 \times 100$ and $400 \times 400$), demonstrating the method's viability for high-resolution computer vision tasks.

Discussion and Limitations

While highly effective for CPA networks, the cloning method is currently limited to piecewise affine nonlinearities. Architectures utilizing smooth activations like Sigmoid, Tanh, or Swish would require linear approximations to benefit from this technique. Additionally, while the method excels at forward-mode JVP computation, it does not replace the Vector-Jacobian Products (VJPs) used in standard backpropagation for weight updates.

Conclusion

Deep Network Cloning offers a simple yet powerful alternative to automatic differentiation for computing JVPs in modern neural networks. By bypassing the overhead of AD, it enables faster model analysis and more efficient implementation of robustness-enhancing regularization techniques. Future work may involve integrating this fast JVP directly into training loops to develop models that are inherently more resilient to adversarial noise.

Summary updated

Abstract

Jacobian-vector products (JVPs) form the backbone of many recent developments in Deep Networks (DNs), with applications including faster constrained optimization, regularization with generalization guarantees, and adversarial example sensitivity assessments. Unfortunately, JVPs are computationally expensive for real world DN architectures and require the use of automatic differentiation to avoid manually adapting the JVP program when changing the DN architecture. We propose a novel method to quickly compute JVPs for any DN that employ Continuous Piecewise Affine (e.g., leaky-ReLU, max-pooling, maxout, etc.) nonlinearities. We show that our technique is on average $2\times$ faster than the fastest alternative over $13$ DN architectures and across various hardware. In addition, our solution does not require automatic differentiation and is thus easy to deploy in software, requiring only the modification of a few lines of codes that do not depend on the DN architecture.
BibTeX
@misc{balestriero2021fast,
  author = {Randall Balestriero and Richard G. Baraniuk},
  title = {{Fast Jacobian-Vector Product for Deep Networks}},
  year = {2021},
  eprint = {2104.00219},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.00219},
}
Figure