Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression

Guohao Shen, Yuling Jiao, Yuanyuan Lin, Jian Huang

Abstract

We study the properties of differentiable neural networks activated by rectified power unit (RePU) functions. We show that the partial derivatives of RePU neural networks can be represented by RePUs mixed-activated networks and derive upper bounds for the complexity of the function class of derivatives of RePUs networks. We establish error bounds for simultaneously approximating $C^s$ smooth functions and their derivatives using RePU-activated deep neural networks. Furthermore, we derive improved approximation error bounds when data has an approximate low-dimensional support, demonstrating the ability of RePU networks to mitigate the curse of dimensionality. To illustrate the usefulness of our results, we consider a deep score matching estimator (DSME) and propose a penalized deep isotonic regression (PDIR) using RePU networks. We establish non-asymptotic excess risk bounds for DSME and PDIR under the assumption that the target functions belong to a class of $C^s$ smooth functions. We also show that PDIR has a robustness property in the sense it is consistent with vanishing penalty parameters even when the monotonicity assumption is not satisfied. Furthermore, if the data distribution is supported on an approximate low-dimensional manifold, we show that DSME and PDIR can mitigate the curse of dimensionality.

Introduction

In modern statistical learning, many tasks require the estimation of not just a target function, but also its derivatives. For instance, score matching in generative modeling relies on the partial derivatives of the score function, while isotonic regression often enforces monotonicity by penalizing negative derivatives.

Standard Rectified Linear Unit (ReLU) networks, despite their popularity, are ill-suited for these tasks. Because ReLU is piecewise linear, its derivative is a step function and its second derivative vanishes almost everywhere. While smooth alternatives like Sigmoid exist, they often suffer from vanishing gradients during training. This work proposes Rectified Power Units (RePUs) as a superior alternative that is both differentiable and non-saturating.

Comparison of activation functions across key properties for gradient-based estimation.

The RePU Framework

The authors focus on the RePU activation function defined as $\sigma_p(x) = \max\{0, x\}^p$ for $p \ge 2$. By utilizing $\sigma_p$, the resulting network becomes $(p-1)$ times continuously differentiable, allowing it to learn the shape and slope of target functions simultaneously.

Key Methodological Contributions

  • Mixed-RePU Networks: The authors prove that partial derivatives of a RePU network can be exactly represented by a "Mixed-RePU" network using a combination of $\sigma_p, \sigma_{p-1}, \dots, \sigma_1$ activations.
  • Exact Polynomial Representation: Unlike ReLU networks, which only approximate polynomials, RePU networks can represent multivariate polynomials exactly with efficient architectures. This allows for tighter approximation bounds via Taylor expansion.
  • Manifold Adaptation: The research demonstrates that RePU networks can adapt to the intrinsic manifold dimension $d_{\mathcal{M}}$ of data, effectively bypassing the curse of dimensionality when $d_{\mathcal{M}} \ll d$.
Comparison of approximation results for various neural network architectures, highlighting the advantages of the proposed RePU framework.

Statistical Applications

Deep Score Matching Estimator (DSME)

The authors establish non-asymptotic excess risk bounds for DSME, providing a theoretical foundation for using RePU networks in energy-based models and diffusion priors. The differentiability of RePUs ensures that the score function's gradients are well-behaved during training.

Penalized Deep Isotonic Regression (PDIR)

The proposed PDIR estimator achieves the minimax optimal convergence rate of $O(n^{-2s/(d+2s)})$. A key finding is its robustness to misspecification: PDIR remains consistent even if the true function is not perfectly monotonic. If the penalty parameter is tuned correctly, the model converges to the best non-monotonic fit, avoiding the failures of traditional "hard-constrained" methods.

Conclusion

This work establishes RePU-activated neural networks as a powerful tool for high-dimensional statistics. By providing simultaneous approximation of functions and their derivatives, RePU networks offer a theoretically sound and empirically effective alternative to ReLU for gradient-sensitive tasks. Future work may explore the practical training stability of RePU in large-scale generative models and its extension to other smooth activations like GELU or Swish.

Summary updated

Abstract

We study the properties of differentiable neural networks activated by rectified power unit (RePU) functions. We show that the partial derivatives of RePU neural networks can be represented by RePUs mixed-activated networks and derive upper bounds for the complexity of the function class of derivatives of RePUs networks. We establish error bounds for simultaneously approximating $C^s$ smooth functions and their derivatives using RePU-activated deep neural networks. Furthermore, we derive improved approximation error bounds when data has an approximate low-dimensional support, demonstrating the ability of RePU networks to mitigate the curse of dimensionality. To illustrate the usefulness of our results, we consider a deep score matching estimator (DSME) and propose a penalized deep isotonic regression (PDIR) using RePU networks. We establish non-asymptotic excess risk bounds for DSME and PDIR under the assumption that the target functions belong to a class of $C^s$ smooth functions. We also show that PDIR has a robustness property in the sense it is consistent with vanishing penalty parameters even when the monotonicity assumption is not satisfied. Furthermore, if the data distribution is supported on an approximate low-dimensional manifold, we show that DSME and PDIR can mitigate the curse of dimensionality.
BibTeX
@misc{shen2023differentiable,
  author = {Guohao Shen and Yuling Jiao and Yuanyuan Lin and Jian Huang},
  title = {{Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression}},
  year = {2023},
  eprint = {2305.00608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.00608},
}
Figure