Differentiable Approximation Bridges For Training Networks Containing Non-Differentiable Functions

Jason Ramapuram, Russ Webb

Abstract

Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow simple non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable approximation bridge (DAB) neural network which approximates the non-differentiable forward function and provides gradient updates during backpropagation. We present strong empirical results (performing over 600 experiments) in four different domains: unsupervised (image) representation learning, variational (image) density estimation, image classification, and sequence sorting to demonstrate that our proposed method improves state of the art performance. We demonstrate that training with DAB aided discrete non-differentiable functions improves image reconstruction quality and posterior linear separability by 10% against the Gumbel-Softmax relaxed estimator as well as providing a 9% improvement in the test variational lower bound in comparison to the state of the art RELAX discrete estimator. We also observe an accuracy improvement of 77% in neural sequence sorting and a 25% improvement against the straight-through estimator in an image classification setting. The DAB network is not used for inference and expands the class of functions that are usable in neural networks.

Introduction

Standard neural network training relies on backpropagation, which requires every function in the network to be differentiable (or at least sub-differentiable, like ReLU). This constraint excludes many powerful, stable algorithms used in traditional computer science, such as sorting, top-k selection, and discrete clustering.

Existing workarounds have significant limitations. Straight-Through Estimators (STE) often fail when functions change data dimensions. Score Function Estimators (REINFORCE) suffer from high variance, and Relaxed Estimators like Gumbel-Softmax are difficult to generalize to complex algorithms like Merge-Sort. This paper introduces Differentiable Approximation Bridges (DAB) to overcome these hurdles.

Methodology: Differentiable Approximation Bridges (DAB)

The core innovation is the Differentiable Approximation Bridge (DAB). Instead of mathematically deriving a "soft" version of a hard function, a secondary neural network ($l_\phi$) is trained to learn an approximation of the non-differentiable function ($l_{hard}$).

The DAB architecture: The forward pass uses the hard function $l_{hard}$, while backpropagation is routed through the differentiable bridge $l_\phi$.

The training process follows these steps:

  1. Forward Pass: The network uses the actual "hard" non-differentiable function ($l_{hard}$) to compute the output.
  2. The Bridge: Simultaneously, the DAB network ($l_\phi$) takes the same input and predicts the output of the hard function.
  3. The Loss: A regularizer (DAB Loss) minimizes the distance (typically L2) between the DAB's output and the hard function's output.
  4. Backward Pass: During backpropagation, the gradient is routed through the smooth DAB network instead of the non-differentiable hard function, providing a meaningful training signal to earlier layers.
  5. Inference: Post-training, the DAB network is discarded, resulting in zero extra computational cost at runtime.

Experimental Setup and Results

The authors validated DAB across four distinct domains: Neural Sequence Sorting, Unsupervised Representation Learning (VAEs), Variational Density Estimation, and Image Classification.

Integration of DAB into different architectures: (Left) A standard dense network with a hard function. (Right) An LSTM-based architecture for sequence sorting.

Key results include:

  • Sorting: DAB achieved a 77% improvement in accuracy for neural sequence sorting compared to Pointer Networks.
  • Density Estimation: On Binarized MNIST, DAB provided a 9% improvement in the test variational lower bound (ELBO) compared to the RELAX estimator.
  • Classification: When using a sort operation as a final activation in ResNet18, DAB outperformed the Straight-Through Estimator by 25%.
  • Latent Space Quality: In VAEs, DAB-aided discrete functions resulted in latent spaces that were 10% more linearly separable than those created with Gumbel-Softmax.

Discussion and Conclusion

DAB is dimension-agnostic and generalizes well to complex algorithms. It allows preceding layers to learn how to "prepare" data for specific algorithms, such as binning values for a sorting operator. While DAB models may take longer to converge and introduce a regularization hyperparameter ($\gamma$), they significantly expand the class of functions usable in deep learning. Future work will focus on improving convergence speed through learning rate schedulers and exploring even more complex algorithmic structures within neural architectures.

Summary updated

Abstract

Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow simple non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable approximation bridge (DAB) neural network which approximates the non-differentiable forward function and provides gradient updates during backpropagation. We present strong empirical results (performing over 600 experiments) in four different domains: unsupervised (image) representation learning, variational (image) density estimation, image classification, and sequence sorting to demonstrate that our proposed method improves state of the art performance. We demonstrate that training with DAB aided discrete non-differentiable functions improves image reconstruction quality and posterior linear separability by 10% against the Gumbel-Softmax relaxed estimator [37, 26] as well as providing a 9% improvement in the test variational lower bound in comparison to the state of the art RELAX [16] discrete estimator. We also observe an accuracy improvement of 77% in neural sequence sorting and a 25% improvement against the straight-through estimator [5] in an image classification setting. The DAB network is not used for inference and expands the class of functions that are usable in neural networks.
BibTeX
@inproceedings{ramapuram2020differentiable,
  author = {Jason Ramapuram and Russ Webb},
  title = {{Differentiable Approximation Bridges For Training Networks Containing Non-Differentiable Functions}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=0pT20aXygHC},
}
Figure