The Curious Case of Convex Neural Networks

Sarath Sivaprasad, Ankur Singh, Naresh Manwani, Vineet Gandhi

TL;DR

This paper introduces Input Output Convex Neural Networks (IOC-NNs) by adding convexity constraints to standard neural networks. These constraints significantly improve generalization, reduce overfitting, and enhance robustness to noisy labels, offering a compelling alternative to traditional deep learning models.

What Problem Does This Paper Address?

Deep neural networks are known for their remarkable ability to fit complex data, often to the point of memorizing training data, even when labels are randomized. This phenomenon, termed "overfitting," makes it challenging to understand their generalization capabilities. While traditional regularization techniques exist, they don't always fully address this issue. The paper questions whether a more fundamental constraint on the network's structure could lead to better generalization and a deeper understanding of how neural networks learn. Specifically, they investigate the impact of enforcing convexity on the relationship between a network's input and its output.

Methods

The core idea is to transform standard feed-forward neural networks into Input Output Convex Neural Networks (IOC-NNs). This is achieved by imposing two key constraints:

  • Non-negative weights: For all layers except the first, weights are restricted to be non-negative.
  • Non-decreasing convex activation function: The activation function used in the hidden layers must be both non-decreasing and convex. The authors found that using activations like ELU (Exponential Linear Unit) is crucial, as it allows for negative values and facilitates identity mapping, which is often problematic with ReLU in constrained settings.

These constraints are applied to both fully connected (MLP) and convolutional architectures. The paper also proposes an optimization algorithm to efficiently train these IOC-NNs, primarily by modifying weight updates to enforce the non-negativity constraint.

Datasets

The paper evaluates IOC-NNs on the following datasets:

  • MNIST
  • FMNIST
  • STL-10
  • SVHN
  • CIFAR-10
  • CIFAR-100

They also use variants of CIFAR-10 and CIFAR-100 called "CIFAIR" datasets, which are created by removing duplicate images from the test sets to provide a more challenging evaluation of generalization.

Key Insights and Ideas

  • Convexity as a powerful self-regularizer: The paper demonstrates that imposing convexity constraints acts as a form of "self-regularization," significantly reducing the need for explicit regularization techniques and effectively combating overfitting.
  • Improved generalization is not just about complexity: Contrary to the intuition that more complex models might generalize better, IOC-NNs show that restricting the hypothesis space through convexity can lead to superior generalization.
  • Robustness to label noise: IOC-NNs exhibit remarkable resilience to noisy labels in the training data, resisting the tendency to memorize incorrect information, a common pitfall for standard neural networks.
  • Identity mapping is achievable: By using activations like ELU, IOC-NNs can overcome limitations that prevented previous convex network architectures from performing simple identity mappings, thus retaining architectural flexibility.
  • Decision boundaries can be complex even with convex outputs: While individual outputs of IOC-NNs are convex, the argmax operation over these convex outputs can still result in complex, non-convex decision boundaries, allowing them to tackle challenging classification tasks.

Key Results

  • Reduced generalization gap: IOC-NNs consistently achieve a significantly smaller generalization gap (difference between train and test accuracy) compared to their unconstrained counterparts across various datasets and architectures. For instance, on CIFAR-100, the generalization gap for AllConv drops from 28.4% to 1.99% with IOC constraints.
  • Competitive or improved test accuracy: While the primary focus is generalization, IOC-MLPs often achieve comparable or even higher test accuracies than standard MLPs. For convolutional architectures, they maintain performance while gaining significant generalization benefits.
  • Robustness to random labels: When trained with up to 100% randomized labels, standard neural networks achieve near 100% training accuracy but perform poorly on test data. In contrast, IOC-NNs show a large negative generalization gap, indicating they resist learning noise and retain performance related to the true label percentage.
  • Improved calibration: IOC constraints lead to better confidence calibration in neural networks, meaning the predicted probabilities more accurately reflect the likelihood of correctness.

Limitations and Future Work

  • Capacity limitations: While ELU helps, the paper acknowledges that previous efforts with convex networks faced capacity limitations. Future work aims to explore proofs for universal approximation for IOC-NNs to fully understand their representational power.
  • Ensemble improvements: While boosted ensembles of IOC-NNs show promise, the paper suggests that further improvements in model selection (gating architecture) could enhance performance.
  • Recurrent architectures: The authors plan to investigate the application of IOC-NNs to recurrent neural networks (RNNs).
  • Interpretability: The paper suggests exploring the interpretability aspects of IOC-NNs and how convexity constraints influence generalization bounds. The MoE approach with IOC-NNs shows potential for partitioning input space in an interpretable way.

Summary updated

Abstract

In this paper, we investigate a constrained formulation of neural networks where the output is a convex function of the input. We show that the convexity constraints can be enforced on both fully connected and convolutional layers, making them applicable to most architectures. The convexity constraints include restricting the weights (for all but the first layer) to be non-negative and using a non-decreasing convex activation function. Albeit simple, these constraints have profound implications on the generalization abilities of the network. We draw three valuable insights: (a) Input Output Convex Neural Networks (IOC-NNs) self regularize and reduce the problem of overfitting; (b) Although heavily constrained, they outperform the base multi layer perceptrons and achieve similar performance as compared to base convolutional architectures and (c) IOC-NNs show robustness to noise in train labels. We demonstrate the efficacy of the proposed idea using thorough experiments and ablation studies on standard image classification datasets with three different neural network architectures.
BibTeX
@inproceedings{sivaprasad2021curious,
  author = {Sarath Sivaprasad and Ankur Singh and Naresh Manwani and Vineet Gandhi},
  title = {{The Curious Case of Convex Neural Networks}},
  booktitle = {ECML/PKDD},
  year = {2021},
  doi = {10.1007/978-3-030-86486-6\_45},
}
Figure