CoopSubNet: Cooperating Subnetwork for Data-Driven Regularization of Deep Networks under Limited Training Budgets

Riddhish Bhalodia, Shireen Y. Elhabian, Ladislav Kavan, Ross T. WhitakerarXiv 2019 · arXiv

Abstract

Deep neural networks often struggle to generalize when training data is scarce, leading to overfitting. Standard regularization techniques like Dropout or weight decay are "data-blind," applying generic constraints regardless of the underlying data distribution. We present CoopSubNet, a novel data-driven regularization framework that attaches a cooperating auto-encoder subnetwork to a primary model's feature space. This mechanism forces learned features to adhere to a low-dimensional manifold, significantly improving performance in low-budget training scenarios across classification and regression tasks.

Introduction

The success of deep learning typically hinges on the availability of massive labeled datasets. However, in domains such as medical imaging or specialized physical simulations, data acquisition is often constrained by cost or physical limitations. While traditional regularization methods mitigate overfitting by constraining model weights, they fail to leverage the intrinsic structure of the data itself. CoopSubNet addresses this by introducing a "soft" manifold constraint that guides the feature extraction process based on population statistics.

Methodology

The CoopSubNet framework utilizes a dual-network architecture consisting of a Primary Network and a Cooperating Subnetwork (CoopSubNet).

The CoopSubNet training and testing architecture. During training, the subnetwork regularizes the feature extractor. At test time, the subnetwork is discarded.

Architecture

  1. Primary Network: Comprises a feature extraction subnetwork ($f_\Theta$) and an output subnetwork ($g_\Phi$). It performs the target task, such as classification or regression.
  2. CoopSubNet: An auto-encoder ($e_\Psi, d_\Gamma$) attached to the intermediate feature vector $f_i$. It attempts to reconstruct $f_i$ through a low-dimensional bottleneck $z_i$.

Cooperation Mechanism

The primary network is encouraged to produce features that are easily reconstructible by the CoopSubNet. This "cooperation" ensures that the features lie on a low-dimensional manifold, effectively regularizing the model in a data-driven manner. To prevent the network from collapsing feature magnitudes to zero, a Relative Reconstruction Loss ($||f - \hat{f}||^2 / ||f||^2$) is employed. Additionally, a Burn-in Phase allows the primary network to establish initial feature representations before the manifold constraint is enforced.

Experiments and Datasets

The efficacy of CoopSubNet was evaluated on three distinct tasks representing different data modalities and objectives:

  • Reduced MNIST: A classification task using only 0.5% and 1% of the original training set to simulate extreme data scarcity.
  • IMM Facial Landmark Dataset: A regression task involving 240 images where the model predicts 58 facial coordinates.
  • Warwick Cell Nuclei Dataset: A segmentation task using histology images to estimate nuclei probability maps.

Results and Discussion

CoopSubNet consistently outperformed baseline models and standard regularization techniques across all benchmarks.

Qualitative results on the IMM Facial Landmark dataset. The model accurately predicts coordinates even with limited training samples.

Key Findings

  • Superior Generalization: On MNIST with 0.5% data, CoopSubNet achieved 93.7% accuracy, surpassing the baseline (90.2%) and Dropout (90.5%).
  • Regression Accuracy: In facial landmark detection, the method reduced prediction error from a baseline of 6.86 to 3.03.
  • Soft vs. Hard Constraints: Comparisons with "HardCon" (models with physical bottleneck layers) revealed that soft constraints via a subnetwork provide better flexibility, preventing the information loss associated with overly restrictive bottlenecks.

Conclusion

CoopSubNet provides a powerful, task-agnostic regularization strategy for deep learning under limited data budgets. By enforcing a manifold constraint on intermediate features, it compensates for missing labels with structural priors. While the method introduces some training-time overhead and requires hyperparameter tuning for the bottleneck size, it offers significant performance gains without increasing inference complexity.

Summary updated

Abstract

Deep networks are an integral part of the current machine learning paradigm. Their inherent ability to learn complex functional mappings between data and various target variables, while discovering hidden, task-driven features, makes them a powerful technology in a wide variety of applications. Nonetheless, the success of these networks typically relies on the availability of sufficient training data to optimize a large number of free parameters while avoiding overfitting, especially for networks with large capacity. In scenarios with limited training budgets, e.g., supervised tasks with limited labeled samples, several generic and/or task-specific regularization techniques, including data augmentation, have been applied to improve the generalization of deep networks.Typically such regularizations are introduced independently of that data or training scenario, and must therefore be tuned, tested, and modified to meet the needs of a particular network. In this paper, we propose a novel regularization framework that is driven by the population-level statistics of the feature space to be learned. The regularization is in the form of a \textbf{cooperating subnetwork}, which is an auto-encoder architecture attached to the feature space and trained in conjunction with the primary network. We introduce the architecture and training methodology and demonstrate the effectiveness of the proposed cooperative network-based regularization in a variety of tasks and architectures from the literature. Our code is freely available at \url{https://github.com/riddhishb/CoopSubNet
BibTeX
@misc{bhalodia2019coopsubnet,
  author = {Riddhish Bhalodia and Shireen Y. Elhabian and Ladislav Kavan and Ross T. Whitaker},
  title = {{CoopSubNet: Cooperating Subnetwork for Data-Driven Regularization of Deep Networks under Limited Training Budgets}},
  year = {2019},
  eprint = {1906.05441},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.05441},
}
Figure