Deep learning excels at unconstrained function approximation. However, when neural networks are deployed inside optimization loops, physical simulations, or geometric tasks, this unconstrained flexibility becomes a liability. Standard architectures can easily violate physical laws or produce non-convex landscapes that make downstream optimization computationally intractable.
To bridge the gap between the expressive power of deep learning and the rigorous guarantees of convex analysis, researchers use Input Convex Neural Networks (ICNNs). By constraining the network's architecture so that the output is provably convex with respect to its inputs, ICNNs serve as a powerful structural prior for modeling physical potentials, solving optimal transport problems, and enabling efficient verification.
How to Bake Convexity into an Architecture
How do we force a neural network $f(x)$ to be a convex function of its input $x$? The core intuition relies on two fundamental rules from calculus and convex analysis:
- The sum of convex functions is convex.
- The composition $g(h(x))$ is convex if $g$ is a convex, non-decreasing function, and $h$ is a convex function.
In a standard multi-layer perceptron, layer outputs are computed via $z_{l+1} = \sigma(W_l z_l + b_l)$. If any weight in $W_l$ is negative, the composition rule breaks because the linear transformation is no longer non-decreasing.
As formalized by Ashok Vardhan Makkuva et al. in Optimal transport mapping via input convex neural networks, an ICNN maintains strict convexity by enforcing two architectural constraints:
- All layer-to-layer weight matrices $W_l$ (for $l \ge 1$) must be non-negative.
- The activation functions $\sigma$ must be both convex and non-decreasing (such as the Rectified Linear Unit, or ReLU).
To prevent the non-negative weight constraint from severely restricting the network's capacity, ICNNs typically include skip connections directly from the input $x$ to every hidden layer. Because these input-to-layer weights can be negative, the network can still learn complex, non-linear functions while mathematically guaranteeing that the final output remains convex with respect to $x$.
Alternative formulations exist to achieve similar properties. For instance, Carles Domingo-Enrich et al. introduce Input Convex Maxout Networks in Learning with Stochastic Orders, leveraging the fact that the maximum of affine functions is inherently convex. This approach achieves strong parametric statistical rates in high dimensions without relying on weight-clipping heuristics.
The Geometry of Optimal Transport and Brenier Maps
The most significant application of ICNNs is in solving high-dimensional Optimal Transport (OT) problems. Under a quadratic cost function, Brenier’s Theorem states that the unique optimal transport map $T(x)$ that pushes one probability distribution to another is the gradient of a strictly convex potential function $\psi(x)$:
$$T(x) = \nabla \psi(x)$$
Standard neural networks cannot easily parameterize valid transport maps because ensuring that an arbitrary vector field is the gradient of a convex potential is incredibly difficult. As noted by Mohammad Al-Jarrah and Niyizhen Jin in Nonlinear Filtering with Brenier Optimal Transport Maps, unconstrained architectures like residual networks fail to satisfy these theoretical assumptions.
By parameterizing the scalar potential $\psi(x)$ as an ICNN, its gradient $\nabla \psi(x)$ is guaranteed to represent a mathematically valid, monotonic transport map. Alexander Korotin leverages this property in Wasserstein-2 Generative Networks to build generative models that bypass unstable minimax (adversarial) optimization entirely, optimizing the convex potential directly.
[Input x] ---> [ ICNN Potential ψ(x) ] ---> [ Gradient ∇ψ(x) ] ---> [ Valid Transport Map T(x) ]
This structural guarantee allows deep learning to scale to complex, high-dimensional distribution problems that were previously computationally intractable:
- Flow Matching: Nikita Kornilov et al. use ICNNs in Optimal Flow Matching: Learning Straight Trajectories in Just One Step to parameterize vector fields as gradients of convex functions, recovering perfectly straight transport trajectories in a single step.
- Population Dynamics: Modeling time-evolving biological populations via Jordan-Kinderlehrer-Otto (JKO) flows traditionally requires expensive grid discretizations. Charlotte Bunne et al. (Proximal Optimal Transport Modeling of Population Dynamics) and Petr Mokrov (Large-Scale Wasserstein Gradient Flows) use ICNNs to parameterize the potential steps, transforming infinite-dimensional fluid dynamics into stable, discrete optimization tasks.
Partial Convexity for Real-Time Optimization
In many engineering and decision-making problems, a function does not need to be convex across all variables—only with respect to the specific variables being optimized. This requirement motivates Partially Input Convex Neural Networks (PICNNs).
Consider the vehicle-to-grid optimization problem studied by Arghya Mallick et al. in User-centric Vehicle-to-Grid Optimization with an Input Convex Neural Network-based Battery Degradation Model. A neural network must predict battery degradation based on historical context (non-convex) and immediate charge/discharge control inputs. By enforcing input convexity only on the control inputs, the model captures highly non-linear physical dynamics while ensuring that the downstream real-time optimization problem remains strictly convex and solvable in milliseconds.
A similar strategy is deployed by Jiaojiao Fan and Yongxin Chen in Scalable Computations of Wasserstein Barycenter via Input Convex Neural Networks. Their architecture maintains strict convexity with respect to data coordinates while remaining non-convex with respect to auxiliary parameters like distribution weights, allowing a single network to compute geometric averages across arbitrary distribution combinations.
Bypassing Hardness and Enabling Verification
Verifying the properties of standard deep networks—such as finding their exact Lipschitz constants or proving that an output will never violate a safety threshold—is generally W[1]-hard or NP-hard as the input dimension scales (Vincent Froese et al. in Parameterized Hardness of Zonotope Containment and Neural Network Verification). The difficulty stems from the presence of countless local minima and non-convex boundaries.
Restricting the architecture to be input convex fundamentally alters this complexity landscape. Because any local minimum of a convex function is guaranteed to be a global minimum, optimization-based verification tasks become polynomial-time solvable.
This tractability enables strict, provable guarantees in critical systems:
- Power Grid Safety: Steven H. Low et al. demonstrate in Fast and Reliable N - k Contingency Screening with Input-Convex Neural Networks that ICNNs can screen power grid failures with zero false negatives, verifying that a predicted feasible operating region is entirely contained within the true safe operating zone.
- Medical Image Segmentation: In Implicit Representations for Constrained Image Segmentation, Jan Philipp Schneider et al. constrain coordinate-based neural networks to be input convex. This ensures that the resulting segmentation boundaries are guaranteed to form a convex set, preventing non-physical, disjoint artifacts even when the underlying image data is heavily occluded or noisy.
- Inverse Problems: In physical reconstruction tasks, Subhadip Mukherjee et al. show in Learning convex regularizers satisfying the variational source condition for inverse problems that using an ICNN as a data-driven regularizer ensures the entire reconstruction optimization remains well-posed, providing classical convergence guarantees alongside deep learning performance.
By embedding the mathematical laws of convexity directly into the network weights and activations, ICNNs transform unstable, unconstrained black boxes into reliable, mathematically consistent operators.
Key Foundational Work
Optimal Control Via Neural Networks: A Convex Approach (2019) — contributed 2 concept elements to this direction.
Wasserstein-2 Generative Networks (2019) — contributed 1 concept elements to this direction.
Geometrical Insights for Implicit Generative Modeling (2017) — contributed 1 concept elements to this direction.