The Curious Case of Convex Networks

Sarath Sivaprasad, Naresh Manwani, Vineet Gandhi

Abstract: In this paper, we investigate a constrained formulation of neural networks where the output is a convex function of the input. We show that the convexity constraints can be enforced on both fully connected and convolutional layers, making them applicable to most architectures. The convexity constraints include restricting the weights (for all but the first layer) to be non-negative and using a non-decreasing convex activation function. Albeit simple, these constraints have profound implications on the generalization abilities of the network. We draw three valuable insights: (a) Input Output Convex Neural Networks (IOC-NNs) self regularize and reduce the problem of overfitting; (b) Although heavily constrained, they outperform the base multi layer perceptrons and achieve similar performance as compared to base convolutional architectures and (c) IOC-NNs show robustness to noise in train labels. We demonstrate the efficacy of the proposed idea using thorough experiments and ablation studies on standard image classification datasets with three different neural network architectures.

Introduction

Deep Neural Networks (DNNs) are remarkably effective but suffer from a significant conceptual hurdle: they are often "too good" at memorization. As famously demonstrated by Zhang et al. (2017), standard DNNs can perfectly fit training data even when labels are completely randomized. This brute-force memorization makes it difficult to explain their generalization capabilities and necessitates heavy explicit regularization like dropout or weight decay.

This research explores a fundamental question: Can we restrict the hypothesis space of a neural network so that it is physically incapable of memorizing noise, yet remains expressive enough to learn complex real-world patterns? The answer lies in Input-Output Convex Neural Networks (IOC-NNs).

Methodology: Enforcing Convexity

To ensure the network's output is a convex function of its input, three primary architectural constraints are implemented:

  1. Weight Constraints: All weights for all layers except the first must be non-negative.
  2. Activation Functions: The network must use non-decreasing, convex activation functions. While ReLU is a common choice, this work highlights the superiority of the Exponential Linear Unit (ELU). ELU allows negative values to flow through, enabling the network to perform identity mappings—a task that previously hindered "fully convex" architectures.
  3. Training via Exponentiation: To maintain non-negativity during backpropagation, the authors use an exponentiation strategy on negative weights after each update, rather than simple clipping.

For classification tasks, where decision boundaries are inherently non-convex, the authors represent each class as a separate convex function and take the argmax over them. This allows the system to learn complex decision boundaries while maintaining the mathematical benefits of convex components.

Experimental Insights

The proposed architecture was evaluated on standard benchmarks including MNIST, Fashion-MNIST, STL-10, SVHN, and the duplicate-free CIFAIR-10/100 datasets. The results reveal several unique properties of IOC-NNs.

Self-Regularization and Generalization

By forcing convexity, the network's complexity (VC-dimension) is effectively reduced. The model is mathematically discouraged from fitting high-frequency noise, leading to a "simpler" hypothesis. In several instances, IOC-NNs exhibited a negative generalization gap, where test accuracy actually exceeded training accuracy. This suggests the model captures the underlying distribution while ignoring training-specific outliers.

Robustness to Label Noise

A standout feature of IOC-NNs is their immunity to random label noise. In experiments where labels were 100% randomized, standard MLPs reached 100% training accuracy (pure memorization), whereas IOC-MLPs remained at ~10% (random chance). This proves the architecture is structurally incapable of memorizing such noise.

Comparison of standard and IOC networks across true label, random label, and noisy label experiments.

As shown in the figure above, when 50% label noise is introduced (panels c and d), the IOC-AllConv model maintains a stable gap between training and test performance, converging toward the true hypothesis. In contrast, the standard AllConv model eventually begins to overfit the noisy labels, leading to a divergence in loss and a degradation of test accuracy.

Key Results

  • MLP Performance: IOC-MLPs outperformed standard MLPs on five out of six datasets. On CIFAR-10, the generalization gap was slashed from 34.16% in the standard model to just 3.38% in the IOC version.
  • Convolutional Parity: For architectures like AllConv and DenseNet, IOC versions achieved test performance comparable to unconstrained models but with significantly lower generalization error. On CIFAR-100, IOC-AllConv recorded a generalization error of only 1.99%, compared to 28.4% for the base model.
  • Calibration: IOC-NNs produce more reliable confidence scores. Reliability diagrams indicate they are less prone to the overconfidence typical of standard deep networks.

Conclusion and Future Directions

IOC-NNs represent a promising shift toward architectures that are "regularized by design." While they may exhibit lower training accuracy due to their restricted hypothesis space, their superior generalization and robustness to noise make them highly attractive for real-world applications where data may be noisy or limited.

Future work includes extending these principles to Recurrent Neural Networks (RNNs) and exploring the interpretability of convex partitions, which may correspond to human-understandable features in the input space.

Summary updated

Abstract

In this paper, we investigate a constrained formulation of neural networks where the output is a convex function of the input. We show that the convexity constraints can be enforced on both fully connected and convolutional layers, making them applicable to most architectures. The convexity constraints include restricting the weights (for all but the first layer) to be non-negative and using a non-decreasing convex activation function. Albeit simple, these constraints have profound implications on the generalization abilities of the network. We draw three valuable insights: (a) Input Output Convex Neural Networks (IOC-NNs) self regularize and reduce the problem of overfitting; (b) Although heavily constrained, they outperform the base multi layer perceptrons and achieve similar performance as compared to base convolutional architectures and (c) IOC-NNs show robustness to noise in train labels. We demonstrate the efficacy of the proposed idea using thorough experiments and ablation studies on standard image classification datasets with three different neural network architectures.
BibTeX
@inproceedings{sivaprasad2021curious,
  author = {Sarath Sivaprasad and Naresh Manwani and Vineet Gandhi},
  title = {{The Curious Case of Convex Networks}},
  booktitle = {ECML/PKDD},
  year = {2021},
  doi = {10.1007/978-3-030-86486-6\_45},
}
Figure