TL;DR
This paper extends the "Input Convex Neural Network" (ICNN) framework to broader classes of functions—semiconvex, quasiconvex, and convex transformable—by embedding structural constraints directly into the network architecture. The authors establish sharp approximation rates for these models and demonstrate that architectural guarantees provide far more reliable global structural validity than standard "soft" shape penalties.
What Problem Does This Paper Address?
In fields like optimal control, economics, and physics-informed machine learning, we often know that a target function must be convex or quasiconvex. While standard Multi-Layer Perceptrons (MLPs) are universal approximators, they don't naturally respect these geometric properties.
The common workaround is to use shape penalties (adding a term to the loss function that penalizes non-convexity). However, penalties are "soft": they are only enforced at specific training points and can easily be violated elsewhere, especially out-of-distribution. Input Convex Neural Networks (ICNNs) solved this for convexity by using non-negative weights, but many real-world problems involve functions that are only "mostly" convex (semiconvex) or have convex sublevel sets (quasiconvex). This paper fills the gap by providing a rigorous theoretical and architectural framework for these more general classes.
Methods
The authors propose a hierarchy of architectures based on the idea of recoverable convex latent structure. The core insight is that many non-convex functions can be represented as transformations of a hidden convex function.
- ICNN (Convex): The baseline, using non-negative hidden-to-hidden weights.
- IsCNN (Semiconvex): An ICNN plus a fixed quadratic correction term ($-\frac{\rho}{2}\|x\|^2$). This "lifts" the semiconvex function back into the convex domain.
- IctNN (Convex Transformable): A composition of a monotone-increasing neural network and an ICNN ($f = \phi \circ g$).
- IqCNN (General Quasiconvex): For functions that don't have a single latent convex representation, the authors use a "max-of-monotone-responses" to affine projections.
The paper also introduces a Transfer Principle, proving that these results—originally derived for ReLU networks—hold for smooth activations like Softplus and ELU, provided they are "ReLU-transferable."
Datasets
The authors evaluated their models on several high-dimensional PDE-based approximation problems where the ground truth belongs to a specific function class:
- Monge-Ampère Dirichlet Problem: A classic test for convex approximation.
- Hamilton-Jacobi Equation: Used to test semiconvexity via the Hopf-Lax formula.
- Brascamp-Lieb Profiles: Synthetic functions used to evaluate robust quasiconvexity.
- High-Dimensional Synthetic Targets: Specifically designed to test convex transformability in up to 32 dimensions.
Key Insights and Ideas
- The Latent Structure Rule: The paper shows that if a function has a single recoverable convex latent representation (like semiconvex or transformable functions), it can be approximated with the same efficiency as a purely convex function ($W^{1,p}$ norm). If it doesn't (general quasiconvex), the approximation regime changes to a weaker $L^\infty$ guarantee.
- Architectural vs. Soft Constraints: The experiments highlight a "stability-accuracy tradeoff." While a heavily tuned MLP with shape penalties might occasionally reach a lower error on training data, it frequently violates the required geometry at test time. The proposed models satisfy the constraints globally by construction, regardless of the parameter values.
- Gradient-Level Control: Unlike many papers that focus only on function values, this work establishes rates in the Sobolev $W^{1,p}$ norm. This means the networks don't just get the "shape" right; they get the "slopes" (gradients) right, which is critical for optimization and physics applications.
Key Results
- Sharp Rates: The authors prove a minimax approximation rate of $O(\epsilon^{-d})$ for convex, semiconvex, and transformable classes, where $d$ is the dimension and $\epsilon$ is the error.
- Monge-Ampère (Convex): ICNN was the only method that maintained a non-negative Hessian across all tested dimensions (2, 3, and 8) while achieving the lowest Sobolev error.
- Hamilton-Jacobi (Semiconvex): While unregularized MLPs had lower raw error, they violated semiconvexity on up to 20% of test samples. The IsCNN maintained zero violations.
- High-Dimensional Scaling: In 32-dimensional convex-transformable tests, the IctNN significantly outperformed regularized MLPs in-domain and maintained structural validity out-of-distribution.
Limitations and Future Work
- The Curse of Dimensionality: The complexity $O(\epsilon^{-d})$ shows that as dimensions increase, the number of parameters needed grows exponentially. This is a fundamental limit of approximation theory, but it suggests that further structural assumptions (like sparsity or low-rank structure) might be needed for extremely high-dimensional problems.
- IqCNN Complexity: The general quasiconvex model (IqCNN) has a slightly worse complexity $O(\epsilon^{-(d+1)})$ and only provides $L^\infty$ (value-level) guarantees rather than $W^{1,p}$ (gradient-level) guarantees. Bridging this gap remains an open theoretical challenge.
- Training Dynamics: While the architectures guarantee the existence of a good, structure-preserving solution, the paper does not deeply explore whether non-convex optimization (SGD/Adam) always finds the best such solution efficiently.
Summary updated