Abstract
Deep neural networks often struggle to generalize when training data is scarce, leading to overfitting. Standard regularization techniques like Dropout or weight decay are "data-blind," applying generic constraints regardless of the underlying data distribution. We present CoopSubNet, a novel data-driven regularization framework that attaches a cooperating auto-encoder subnetwork to a primary model's feature space. This mechanism forces learned features to adhere to a low-dimensional manifold, significantly improving performance in low-budget training scenarios across classification and regression tasks.
Introduction
The success of deep learning typically hinges on the availability of massive labeled datasets. However, in domains such as medical imaging or specialized physical simulations, data acquisition is often constrained by cost or physical limitations. While traditional regularization methods mitigate overfitting by constraining model weights, they fail to leverage the intrinsic structure of the data itself. CoopSubNet addresses this by introducing a "soft" manifold constraint that guides the feature extraction process based on population statistics.
Methodology
The CoopSubNet framework utilizes a dual-network architecture consisting of a Primary Network and a Cooperating Subnetwork (CoopSubNet).
Architecture
- Primary Network: Comprises a feature extraction subnetwork ($f_\Theta$) and an output subnetwork ($g_\Phi$). It performs the target task, such as classification or regression.
- CoopSubNet: An auto-encoder ($e_\Psi, d_\Gamma$) attached to the intermediate feature vector $f_i$. It attempts to reconstruct $f_i$ through a low-dimensional bottleneck $z_i$.
Cooperation Mechanism
The primary network is encouraged to produce features that are easily reconstructible by the CoopSubNet. This "cooperation" ensures that the features lie on a low-dimensional manifold, effectively regularizing the model in a data-driven manner. To prevent the network from collapsing feature magnitudes to zero, a Relative Reconstruction Loss ($||f - \hat{f}||^2 / ||f||^2$) is employed. Additionally, a Burn-in Phase allows the primary network to establish initial feature representations before the manifold constraint is enforced.
Experiments and Datasets
The efficacy of CoopSubNet was evaluated on three distinct tasks representing different data modalities and objectives:
- Reduced MNIST: A classification task using only 0.5% and 1% of the original training set to simulate extreme data scarcity.
- IMM Facial Landmark Dataset: A regression task involving 240 images where the model predicts 58 facial coordinates.
- Warwick Cell Nuclei Dataset: A segmentation task using histology images to estimate nuclei probability maps.
Results and Discussion
CoopSubNet consistently outperformed baseline models and standard regularization techniques across all benchmarks.
Key Findings
- Superior Generalization: On MNIST with 0.5% data, CoopSubNet achieved 93.7% accuracy, surpassing the baseline (90.2%) and Dropout (90.5%).
- Regression Accuracy: In facial landmark detection, the method reduced prediction error from a baseline of 6.86 to 3.03.
- Soft vs. Hard Constraints: Comparisons with "HardCon" (models with physical bottleneck layers) revealed that soft constraints via a subnetwork provide better flexibility, preventing the information loss associated with overly restrictive bottlenecks.
Conclusion
CoopSubNet provides a powerful, task-agnostic regularization strategy for deep learning under limited data budgets. By enforcing a manifold constraint on intermediate features, it compensates for missing labels with structural priors. While the method introduces some training-time overhead and requires hyperparameter tuning for the bottleneck size, it offers significant performance gains without increasing inference complexity.
Summary updated