Abstract
Jacobian-vector products (JVPs) are fundamental to modern machine learning tasks, including adversarial robustness assessment, sharpness-aware regularization, and second-order optimization. However, standard automatic differentiation (AD) frameworks often introduce significant computational and memory overhead. We present a novel "Deep Network Cloning" approach specifically for Continuous Piecewise Affine (CPA) networks—such as those utilizing ReLU or Max-pooling—that bypasses AD entirely. By exploiting the local linearity of these architectures, our method achieves a consistent ~2x speedup across a wide range of standard deep learning models while significantly reducing memory consumption.
Introduction
The Jacobian matrix, which captures the sensitivity of model outputs to input perturbations, is a critical tool for understanding and improving deep neural networks. While the full Jacobian is typically too large to compute or store, many applications only require the Jacobian-vector product (JVP). Traditionally, JVPs are computed using forward-mode AD or the "double-backpropagation" trick. Despite their ubiquity, these methods are often computationally expensive and can lead to out-of-memory (OOM) errors, particularly in recurrent architectures.
Methodology: Deep Network Cloning
The proposed method leverages the geometric properties of Continuous Piecewise Affine (CPA) networks. In such networks, the mapping from input to output is locally linear; within a specific region of the input space, the network functions as a simple affine transformation: $f(x) = A_\omega x + b_\omega$, where $A_\omega$ is the Jacobian.
As illustrated in Figure 1, the "cloning" algorithm operates in two primary phases:
- Observation Pass: The input $x$ is passed through the network to determine and "freeze" the states of all piecewise linear activations (e.g., identifying which ReLU units are active).
- Cloned Pass: A new vector is passed through a "cloned" version of the network where the nonlinearities are forced to maintain the states recorded in the first pass.
By combining these passes—for instance, by calculating the difference between cloned outputs—the JVP ($A_\omega u$) can be extracted directly. This approach eliminates the need for tracking computational graphs or derivative rules, making it highly efficient and easy to implement in environments without full AD support.
Experimental Results
The efficiency of the cloning method was evaluated across 13 diverse architectures, including RNNs, UNets, and various iterations of VGG, ResNet, and EfficientNet. Benchmarks were conducted on multiple hardware platforms, including NVIDIA Quadro RTX 8000 and TITAN X (Pascal) GPUs.
As shown in the table above (Figure 6), the cloning method consistently outperformed all AD-based alternatives. Key findings include:
- Speed: The method provides an average speedup factor of approximately 1.7x to 2.1x compared to the fastest alternative AD methods (such as
jvpor double-VJP). - Memory Efficiency: In Recurrent Neural Networks (RNNs), where AD often fails due to the memory requirements of tracking long-term dependencies, the cloning method remained stable and successfully computed JVPs without OOM errors.
- Scalability: The performance gains were consistent across different input resolutions ($100 \times 100$ and $400 \times 400$), demonstrating the method's viability for high-resolution computer vision tasks.
Discussion and Limitations
While highly effective for CPA networks, the cloning method is currently limited to piecewise affine nonlinearities. Architectures utilizing smooth activations like Sigmoid, Tanh, or Swish would require linear approximations to benefit from this technique. Additionally, while the method excels at forward-mode JVP computation, it does not replace the Vector-Jacobian Products (VJPs) used in standard backpropagation for weight updates.
Conclusion
Deep Network Cloning offers a simple yet powerful alternative to automatic differentiation for computing JVPs in modern neural networks. By bypassing the overhead of AD, it enables faster model analysis and more efficient implementation of robustness-enhancing regularization techniques. Future work may involve integrating this fast JVP directly into training loops to develop models that are inherently more resilient to adversarial noise.
Summary updated