Improving Discrete Latent Representations With Differentiable Approximation Bridges

Jason Ramapuram, Russ Webb

Abstract

Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable approximation bridge (DAB) neural network which approximates the non-differentiable forward function and provides gradient updates during backpropagation. We present empirical results from more than 600 experiments in four different domains: unsupervised (image) representation learning, variational (image) density estimation, image classification, and sequence sorting to demonstrate that our proposed method improves state of the art performance. We demonstrate that training with DAB aided discrete non-differentiable functions improves image reconstruction quality and posterior linear separability by 10% against the Gumbel-Softmax relaxed estimator as well as providing a 9% improvement in the test variational lower bound in comparison to the state of the art RELAX discrete estimator. We also observe an accuracy improvement of 77% in neural sequence sorting and a 25% improvement against the straight-through estimator in an image classification setting. The DAB network is not used for inference and expands the class of functions that are usable in neural networks.

Introduction

Deep learning's success is fundamentally tied to backpropagation, which necessitates that every layer in a network be differentiable. This requirement excludes a vast array of powerful, non-differentiable algorithms such as sorting, k-means clustering, or discrete logic gates. While existing workarounds like the Straight-Through Estimator (STE) or relaxed estimators (e.g., Gumbel-Softmax) exist, they often suffer from significant drawbacks, including gradient mismatch or limitations to specific probability distributions. This paper introduces Differentiable Approximation Bridges (DAB), a general-purpose framework to integrate arbitrary non-differentiable "hard" functions into neural architectures.

Methodology: The Differentiable Approximation Bridge

The core innovation of DAB is the introduction of a small, auxiliary neural network ($l_\phi$) that acts as a differentiable "bridge" for a non-differentiable function ($l_{hard}$).

The DAB computational graph showing the forward pass through the hard function and the backward pass through the bridge network.

Forward and Backward Passes

During the forward pass, the model utilizes the actual "hard" function (e.g., a sorting algorithm or a thresholding operation). This ensures that the network benefits from the exact discrete logic intended by the designer. During the backward pass, since gradients cannot flow through the non-differentiable $l_{hard}$, they are instead routed through the DAB network $l_\phi$. Because $l_\phi$ is a standard, smooth neural network, it provides a meaningful training signal to the preceding layers.

Joint Training and Inference

The DAB is trained concurrently with the main model. A regularization term (typically L2 loss) encourages the DAB's output to approximate the output of the hard function. Crucially, the DAB is only used during training. For inference, the bridge is discarded, and the model uses the hard function directly, resulting in zero additional computational overhead at deployment.

Experimental Setup

The effectiveness of DAB was evaluated across four distinct domains:

  1. Neural Sequence Sorting: Testing combinatorial logic on synthetic sequences of lengths 5, 10, and 15.
  2. Unsupervised Representation Learning: Using VAEs on FashionMNIST, CIFAR10, and high-resolution ImageNet (512x512).
  3. Generative Modeling: Density estimation on Binarized MNIST and Binarized Omniglot.
  4. Image Classification: Modified ResNet18 architectures on CIFAR10.
DAB integration in standard architectures: a feed-forward dense network (left) and a recurrent LSTM network (right).

Key Results and Insights

The empirical results demonstrate that DAB consistently outperforms existing discrete estimators:

  • Combinatorial Tasks: In sequence sorting, DAB achieved 99.3% accuracy for length-5 sequences. On more challenging length-15 sequences, where standard models often fail (4-10% accuracy), DAB maintained approximately 80% accuracy.
  • Improved Latent Spaces: Compared to Gumbel-Softmax, DAB improved image reconstruction and increased the linear separability of the latent space by 10% in VAE settings.
  • State-of-the-Art Density Estimation: DAB surpassed RELAX and REBAR estimators on binarized datasets, achieving superior Variational Lower Bounds (ELBO).
  • Classification Performance: In dense model settings, DAB provided a 25% accuracy boost over the Straight-Through Estimator.

A key insight from this work is that backpropagation does not require mathematically exact gradients to be effective; rather, it requires a "meaningful training signal" that correctly directs weight updates. Furthermore, unlike STE, DAB can handle operations that change data dimensions, such as top-k selection.

Conclusion and Future Work

Differentiable Approximation Bridges offer a robust and flexible method for incorporating discrete logic into deep learning. While DAB currently requires more epochs to converge compared to some alternatives and introduces a regularization hyperparameter ($\gamma$), its performance gains are substantial. Future research may focus on optimizing convergence through specialized learning rate schedulers and applying DAB to even more complex modules like physics engines or discrete optimizers.

Summary updated

Abstract

Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable approximation bridge (DAB) neural network which approximates the non-differentiable forward function and provides gradient updates during backpropagation. We present empirical results from more than 600 experiments in four different domains: unsupervised (image) representation learning, variational (image) density estimation, image classification, and sequence sorting to demonstrate that our proposed method improves state of the art performance. We demonstrate that training with DAB aided discrete non-differentiable functions improves image reconstruction quality and posterior linear separability by 10% against the Gumbel-Softmax relaxed estimator [26], [37] as well as providing a 9% improvement in the test variational lower bound in comparison to the state of the art RELAX [16] discrete estimator. We also observe an accuracy improvement of 77% in neural sequence sorting and a 25% improvement against the straight-through estimator [5] in an image classification setting. The DAB network is not used for inference and expands the class of functions that are usable in neural networks. Our code is available at https://github.com/apple/ml-dab.
BibTeX
@inproceedings{ramapuram2020improving,
  author = {Jason Ramapuram and Russ Webb},
  title = {{Improving Discrete Latent Representations With Differentiable Approximation Bridges}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=0pT20aXygHC},
}
Figure