Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures

Michael Unser, Alexis Goujon, Stanislas Ducotterd

Abstract

In modern deep learning, activation functions are typically fixed as simple, predefined shapes like ReLU or Sigmoid. This paper introduces a rigorous variational framework for learning "freeform" activation functions that are mathematically guaranteed to be adaptive linear splines. By incorporating second-order total variation ($TV^{(2)}$) regularization and explicit slope constraints, the proposed method ensures critical properties such as Lipschitz stability and monotonicity. Experimental results demonstrate that these learned nonlinearities lead to faster convergence and superior performance in image reconstruction tasks compared to standard fixed-activation architectures.

Introduction

While traditional neural network training focuses on optimizing connection weights, the choice of activation functions—the "brains" of individual neurons—remains largely static. In fields such as medical imaging and signal processing, where inverse problems (e.g., MRI reconstruction or deblurring) require high stability, fixed activations can be limiting. A small input perturbation in an unstable network can lead to significant artifacts or "hallucinations."

Existing methods for learning activation functions often lack theoretical guarantees or suffer from computational inefficiency. This research addresses these challenges by providing a framework that allows neurons to learn their own shapes while remaining within the bounds required by classical signal processing theory.

Methods

The researchers employ a variational approach to define the optimal activation function. Rather than simply minimizing training loss, they introduce a regularization term based on the second-order total variation ($TV^{(2)}$) of the function.

The Representer Theorem and Splines

A key contribution of the paper is the proof of a Representer Theorem, which establishes that the optimal solution for this variational problem is always a nonuniform linear spline. This allows a complex, continuous function to be described by a finite set of "knots" or breakpoints. To implement this efficiently, the authors use a nonuniform B-spline basis. Unlike standard ReLUs, B-splines offer "local support," meaning parameter updates only affect a specific region of the function, leading to more stable optimization.

Learned nonuniform linear splines $f_1$ and $f_2$ illustrating the adaptive knot placement and piecewise linear nature of the optimal solutions.

Constraints for Stability

The framework allows for explicit bounds on the derivative ($s_{min} \le f'(x) \le s_{max}$). These slope constraints enable the enforcement of 1-Lipschitz stability, which is essential for "Plug-and-Play" algorithms, or strict monotonicity for invertible networks. The $TV^{(2)}$ regularizer is particularly effective because it does not penalize linear functions, naturally pushing learned activations toward simple, piecewise-linear shapes.

Experimental Results

The framework was evaluated using synthetic 1D data and the BSD500 (Berkeley Segmentation Dataset) for image denoising.

  • Convergence Speed: In 1D function fitting, the B-spline model reached minimum loss orders of magnitude faster than standard Multi-Layer Perceptrons (MLPs) or fixed-grid ReLUs due to its local support properties.
  • Denoising Performance: On the BSD500 dataset, relaxing strict convexity to "weak convexity" ($\rho=1$) resulted in a +0.5 dB improvement in PSNR across various noise levels. The learned potentials remained competitive with state-of-the-art classical denoisers like BM3D while providing the theoretical stability needed for iterative reconstruction in CT or MRI scans.

Discussion and Limitations

While the framework offers significant advantages in stability and convergence, some challenges remain. The authors noted that training "synthesis" filterbanks is more complex and slower than "analysis" filterbanks. Additionally, the current work focuses on 1D pointwise nonlinearities; extending these mathematical guarantees to higher-dimensional activations remains a subject for future research. The team has released the DeepSplines toolbox to facilitate community adoption of controlled nonlinearity learning.

Summary updated

BibTeX
@inproceedings{unser2025controlled,
  author = {Michael Unser and Alexis Goujon and Stanislas Ducotterd},
  title = {{Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures}},
  booktitle = {Applied and Computational Harmonic Analysis},
  year = {2025},
  doi = {10.1016/j.acha.2025.101764},
}
Figure