In classical engineering, we often rely on convex optimization because it is "safe." If a problem is convex, we know a global minimum exists, and we have efficient algorithms to find it. In contrast, deep learning is famously "unsafe"—standard neural networks are non-convex black boxes that offer no guarantees when embedded inside a control loop or a physical simulation.
Input Convex Neural Networks (ICNNs) bridge this gap. By enforcing specific architectural constraints, researchers have developed a class of models that are as expressive as deep networks but as mathematically well-behaved as convex functions. This allows us to replace slow, iterative solvers with ultra-fast neural surrogates without losing the stability and convergence guarantees required for critical infrastructure, medical imaging, and robotics.
The Geometry of Constraint
The core intuition behind an ICNN is simple: if you want the output of a network to be convex with respect to its inputs, you must ensure that every operation within the network preserves convexity.
As established by Yize Chen, Yuanyuan Shi, and Baosen Zhang in Optimal Control Via Neural Networks: A Convex Approach, this is achieved through two primary rules:
- Non-negative weights: All weights connecting the input-dependent layers must be non-negative.
- Convex, non-decreasing activations: Functions like ReLU or Softplus are used because they preserve the convexity of the preceding layers.
When these rules are followed, the entire network becomes a composite convex function. This isn't just a mathematical curiosity; it changes how the model behaves during inference. If an ICNN is used to model the cost of a power grid or the energy of a molecule, a simple gradient descent on the input is guaranteed to find the global optimum.
Accelerating the "Inner Loop" of Science
Many scientific problems involve an "inner loop" where an optimization problem must be solved repeatedly. For example, in fluid dynamics, the Boltzmann equation requires an "entropy closure" step—a complex optimization that ensures physical laws like the second law of thermodynamics are satisfied.
Steffen Schotthöfer and Martin Frank demonstrated in Structure Preserving Neural Networks: A Case Study in the Entropy Closure of the Boltzmann Equation that an ICNN can act as a surrogate for this closure step. Because the network is strictly convex, its gradients automatically yield valid physical multipliers. This architectural "inductive bias" allows the model to achieve 1000x speedups while remaining strictly hyperbolic and entropy-dissipative—properties that unconstrained networks frequently violate.
Similarly, in power systems, Andrew Rosemberg and Mathieu Tanneau show in Learning Optimal Power Flow Value Functions with Input-Convex Neural Networks that ICNNs can learn the "value function" of grid demand. Because the learned proxy is convex, it can be dropped into larger grid-level optimization problems without turning a manageable linear program into a nightmare of non-convexity.
Provable Stability in Inverse Problems
In medical imaging (CT or MRI), we often reconstruct images by minimizing a combination of data fit and a "regularizer" (a prior on what a clean image looks like). Standard deep learning denoisers often produce "hallucinations" because they lack a clear underlying energy landscape.
Researchers like Subhadip Mukherjee and Carola-Bibiane Schönlieb have pioneered the use of ICNNs as learned convex regularizers. In Learned reconstruction with convergence guarantees, they show that if the regularizer is a convex neural network, the reconstruction process inherits the global convergence guarantees of classical variational analysis.
This theme extends to Optimal Transport, where ICNNs are used to represent "Brenier potentials." In 2-Wasserstein Approximation via Restricted Convex Potentials, Amirhossein Taghvaei and Amin Jalali use the convexity of ICNNs to directly approximate the transport map between distributions. Because the network is convex, its gradient is guaranteed to be a valid, monotonic map, providing a bias-free way to train GANs or solve high-dimensional transport problems.
Beyond Pure Convexity: Partial and Invex Structures
The field is now moving toward more flexible constraints. Real-world systems are rarely convex in every variable. Dominik K. Klein and Oliver Weeger introduced Partially Input Convex Neural Networks (PICNNs) in Parametrized polyconvex hyperelasticity with physics-augmented neural networks. These networks are convex with respect to mechanical strain (ensuring physical stability) but remain non-convex with respect to parameters like temperature or material age, allowing the model to learn complex, non-linear environmental dependencies.
This evolution suggests a new paradigm for "Physics-Informed" AI: rather than just penalizing physical violations in the loss function, we bake the fundamental mathematical structure—convexity, monotonicity, or symmetry—directly into the layers of the network. As Felix Bünning demonstrates in Input Convex Neural Networks for Building MPC, this structural integrity is what finally makes machine learning "competitive" and "reliable" enough for real-world deployment in complex control systems.