Abstract
This paper investigates the incremental training of a Neural Network (NN) with the input attributes introduced in order. A specially designed NN is used to evaluate the individual discrimination ability of each input attribute. Attributes are then sorted in descending, ascending, and random orders of their individual discrimination abilities and introduced into another NN being trained with an incremental training algorithm, ITID. To reduce the interference caused by irrelevant features and high-complexity tasks, only relevant features are involved and tasks are decomposed in the experiments. The experimental results of several benchmark problems show that descending order obtains the highest generalization accuracy among the three training orders for both classification and regression problems.
Introduction
In standard neural network training, all input attributes are typically fed into the model simultaneously. However, this can lead to significant "interference" in complex tasks where internal weights become strongly coupled, or where irrelevant features introduce noise. This research explores incremental training, specifically the ITID (Incremental neural network training with an increasing input dimension) algorithm. The central question is whether the sequence in which features are introduced—specifically their "discriminative power"—affects the final model performance.
Methodology
The ITID Architecture
The ITID algorithm constructs a network incrementally. As each new attribute is introduced, a corresponding sub-network is trained while the weights of previous sub-networks are frozen. This modular approach allows the model to build upon a stable foundation, mitigating the catastrophic interference often seen in large, monolithic networks.
Attribute Evaluation and Ordering
To determine the optimal sequence, the researchers implemented a multi-step pipeline:
- Discrimination Evaluation: A small three-layer NN evaluates each input attribute individually to determine its "individual discrimination ability."
- Ordering Strategies: Features are ranked and introduced in three ways:
- Descending: Most discriminative features first.
- Ascending: Least discriminative features first.
- Random: Original dataset order.
- Interference Reduction: Irrelevant features are removed using a contribution-based method, and complex multi-class problems are decomposed into binary sub-modules.
Experimental Setup
The study utilized four benchmarks from the PROBEN1 collection:
- Diabetes1: 2-class classification (8 attributes).
- Cancer1: 2-class classification (9 attributes).
- Glass1: Multi-class classification (9 attributes), decomposed into six sub-modules.
- Flare1: Regression for solar flare activity (10 attributes), decomposed into three modules.
Results and Discussion
The experimental results across all datasets consistently favored the Descending order.
Generalization Accuracy
In the Diabetes1 dataset, the Descending order achieved a classification error of 22.8%, significantly better than the 25.8% seen with the Ascending order. The "head start" provided by high-discrimination attributes was even more evident in the Cancer1 dataset, where the Descending order started with a 5.37% error rate (vs. 13.79% for Ascending) and finished with a superior 1.2% final error.
The "Strong Foundation" Hypothesis
The primary insight is that early attributes have a disproportionate influence on the network's trajectory. By starting with high-discrimination attributes, the network establishes an accurate baseline error that remains lower throughout the growth process. Interestingly, while the Ascending order was faster to train (fewer epochs), it resulted in sub-optimal convergence, suggesting an accuracy-speed tradeoff.
Conclusion
This study demonstrates that the order of feature introduction is a critical hyperparameter in incremental neural network training. Starting with the most informative features (Descending order) consistently yields the best generalization results for both classification and regression. Future work may explore how these principles apply to modern deep learning architectures and the role of top-level integration layers in further optimizing performance.
Summary updated