TL;DR
This paper introduces a novel approach to semi-supervised learning by using neuroevolutionary algorithms to evolve neural network architectures. They propose incorporating "neuron coverage metrics," inspired by software testing, into the fitness function to guide the search using unlabeled data, making the process less reliant on scarce labeled data and potentially improving generalization.
What Problem Does This Paper Address?
In many real-world machine learning applications, obtaining large amounts of labeled data for supervised training is expensive and time-consuming. Semi-supervised learning aims to leverage abundant unlabeled data to improve model performance when labeled data is limited. While neuroevolution (using evolutionary algorithms to design neural networks) is effective for supervised tasks, its application to semi-supervised problems, especially in automatically searching for optimal architectures (Neural Architecture Search or NAS), is less explored. This paper specifically tackles the challenge of how to effectively guide the neuroevolutionary search for neural network architectures when faced with limited labeled data and abundant unlabeled data.
Methods
The core of this paper is the integration of neuron coverage metrics into the fitness function of a neuroevolutionary algorithm.
- Neuroevolutionary Approach: They employ a Genetic Algorithm (GA) to evolve neural network architectures. The networks are represented by a list-based encoding that specifies aspects like the number of hidden layers, initialization functions, activation functions, dropout, and batch normalization.
- Neuron Coverage Metrics: Inspired by code coverage in software testing, these metrics quantify how well different components of a neural network (neurons, layers) are "exercised" by input data. The paper defines and uses five specific metrics:
- Neuron Coverage (NC): Proportion of neurons activated above a threshold.
- Top-K Neuron Coverage (TKNC): Proportion of neurons that were among the top-K most active in their layer for at least one input.
- k-Multi-section Neuron Coverage (KMN): How well the range of neuron activations is covered across predefined sections.
- Neuron Boundary Coverage (NBC): Measures coverage of "corner-case regions" where neuron activations fall outside their typical bounds.
- Strong Neuron Activation Coverage (SNAC): Similar to NBC but focuses on activations exceeding upper bounds.
- Fitness Function: The fitness of a candidate architecture is a weighted combination of:
- Classification Accuracy: Evaluated on labeled data (using balanced accuracy for imbalanced datasets).
- Neuron Coverage: Evaluated on unlabeled data using one of the proposed metrics. The weight
qcontrols the proportion of unlabeled data used.
Datasets
The study uses 20 publicly available binary classification datasets from the PMLB (Problem Matrix Benchmark Library) for their experiments. These datasets vary in size, number of features, and class imbalance.
Key Insights and Ideas
- Unlabeled Data as a "Test Set" for Architectures: The central idea is that unlabeled data can act as a proxy for testing the "behavioral coverage" of a neural network architecture, complementing the direct accuracy evaluation on labeled data.
- Neuron Coverage as a Regularizer/Guidance Signal: Neuron coverage metrics can help neuroevolution avoid getting stuck in local optima by encouraging exploration of architectures that exhibit diverse activation patterns across the unlabeled dataset.
- NC and TKNC are Robust: The paper finds that the Neuron Coverage (NC) and Top-K Neuron Coverage (TKNC) metrics are particularly stable and effective, showing less performance degradation as the amount of labeled data decreases.
- Potential for Improved Generalization: In some cases, using neuron coverage metrics led to classifiers that performed as well as, or even better than, those trained on fully labeled data, suggesting improved generalization.
Key Results
- Performance with Reduced Labels: For most datasets, the performance of the best evolved classifiers did not critically degrade as the proportion of unlabeled data (
q) increased, especially when using NC and TKNC. - Metric Comparison: NC and TKNC consistently performed well across various datasets and levels of unlabeled data. KMN, NBC, and SNAC showed more significant performance drops as
qincreased. - Comparison to Baselines: Neuroevolutionary approaches using NC and TKNC were competitive with, and often superior to, baseline methods based on uncertainty quantification (CERT) and retraining (RET), especially considering the computational cost of retraining.
- Faster Convergence: For some problems (e.g., chess dataset), neuroevolution with NC and TKNC showed faster convergence towards better performing architectures compared to other coverage metrics.
Limitations and Future Work
- Binary Classification Focus: The current work is limited to binary classification problems. The authors suggest extending the approach to multi-class and multi-label problems.
- Computational Cost: While generally less costly than retraining, neuroevolution can still be computationally intensive.
- Bias in Architecture Components: Future work could analyze if neuron coverage metrics introduce biases in the types of neural network components (e.g., activation functions) that are favored.
- Broader Applications: The authors propose exploring the use of neuron coverage metrics in other neuroevolutionary scenarios, such as regularization, diversification mechanisms, and adaptive instance selection for early verification of deployed models.
Summary updated