In safety-critical applications—from autonomous driving to high-energy physics—we need neural networks that are not just accurate, but provably stable. A common way to guarantee stability is to enforce Lipschitz continuity, ensuring that a small change in input cannot lead to an arbitrarily large change in output.
However, building deep 1-Lipschitz networks is notoriously difficult. When we strictly constrain the weights of a network to limit its Lipschitz constant, standard activation functions like ReLU tend to "kill" gradients, leading to the vanishing gradient problem and a massive loss in expressivity.
A growing body of research suggests a solution: replacing threshold-based activations with gradient norm preserving activations. Instead of zeroing out values, these functions reorder or reflect them, maintaining the flow of information and enabling the training of deep, certifiably robust models.
The Problem with Vanishing Expressivity
To make a network 1-Lipschitz, we typically constrain the norm of each layer's weight matrix. If every layer has a Lipschitz constant of 1, the entire composition is also 1-Lipschitz. But as Ben Barrett, Alexander Camuto, and colleagues note in Certifiably Robust Variational Autoencoders, standard activations like ReLU are "contractive"—they can reduce the norm of the gradient to zero.
In a standard network, large weights can compensate for this loss. In a 1-Lipschitz network, the weights are capped. If the activation function also shrinks the gradient, the signal quickly vanishes as it moves through the layers. This leaves the model unable to learn complex patterns, effectively trading away all its "intelligence" for a guarantee of safety.
Sorting as a Nonlinearity
The breakthrough intuition is to use activations that are isometries: functions that preserve the $l_2$ norm of the gradient. The most prominent examples are sorting-based activations like GroupSort.
Instead of applying a threshold to each neuron independently, GroupSort partitions neurons into groups and sorts them by value. As Ugo Tanielian and Gérard Biau demonstrate in Approximating Lipschitz continuous functions with GroupSort neural networks, sorting is a nonlinear operation that is 1-Lipschitz and, crucially, gradient-norm-preserving. Because sorting only moves values around rather than squashing them, the gradient magnitude remains exactly 1 (almost everywhere).
This allows researchers like Ouail Kitouni and Niklas Nolte to build Robust and Provably Monotonic Networks. By maintaining gradient flow, they can train deep architectures that are expressive enough to handle real-time data selection at the Large Hadron Collider while remaining mathematically guaranteed to be stable.
The Geometry of Householder Reflections
While sorting is intuitive, it is actually a specific case of a broader geometric principle. Sahil Singla and Soheil Feizi argue in Householder Activations for Provable Robustness against Adversarial Attacks that any continuous, piecewise linear activation that preserves the gradient norm must transition its Jacobian through a Householder transformation at its boundaries.
A Householder transformation is essentially a reflection across a plane. By designing activations as learnable reflections, we can create nonlinearities that are more flexible than simple sorting but still satisfy the strict requirements of 1-Lipschitz continuity. This framework unifies various activations—like GroupSort and FullSort—into a single family of reflection-based transformations that maximize the "robustness budget" of the network.
Beyond Robustness: Monotonicity and Optimization
The benefits of these activations extend beyond adversarial defense. In many scientific domains, we require monotonicity: if an input variable (like energy) increases, the output must also increase.
Standard networks struggle with this, but as shown in Expressive Monotonic Neural Networks, combining 1-Lipschitz layers with residual connections and sorting activations creates a model that is provably monotonic by design.
Furthermore, these activations improve the optimization landscape itself. Meisam Razaviyayn and Maher Nouiehed show in Learning Deep Models: Critical Points and Local Openness that strictly monotone, norm-preserving mappings help maintain "local openness." This geometric property ensures that the mapping from parameters to outputs is well-behaved, which can theoretically guarantee that every local minimum is a global minimum under certain architectural conditions.
Summary
By shifting from "squashing" activations (ReLU, Sigmoid) to "rearranging" activations (GroupSort, Householder), we solve the fundamental tension between Lipschitz constraints and model expressivity. This allows for a new class of deep models that are:
- Certifiably Robust: Guaranteed to resist adversarial perturbations.
- Trainable: Free from the vanishing gradient issues that plague constrained networks.
- Physically Sensible: Capable of enforcing hard constraints like monotonicity for scientific discovery.