TL;DR
This paper tackles the notorious slowness of back-propagation (BP) neural networks by introducing a novel method for improving input data representation. By automatically constructing new, more informative features using decision trees, they significantly reduce the "difficulty" of the learning problem, leading to much faster convergence and fewer training epochs for BP.
What Problem Does This Paper Address?
Artificial neural networks, particularly back-propagation (BP), are powerful tools for various tasks like pattern recognition and association. However, a major hurdle in using BP is its extremely slow convergence speed, meaning it takes a very long time to learn from data. This slowness can be a significant bottleneck, especially in commercial applications where quick decision-making is crucial. The paper hypothesizes that the way data is represented (the features used) plays a critical role in how difficult a concept is for BP to learn. If the initial features don't cleanly separate different classes of data, the network struggles, leading to slow learning.
Methods
The core of this paper's contribution lies in its approach to feature construction. They leverage a system called CITRE (developed by Matheus) which uses decision trees to automatically generate new, more informative features from existing ones. The process works iteratively:
- Decision Tree Generation: A decision tree is built from the current set of features.
- Feature Construction: CITRE identifies "fringe" nodes in the tree (parent and grandparent of positive leaves) and creates new features by conjoining them. These new features aim to capture more complex interactions between the original attributes.
- Feature Set Augmentation: The newly constructed features are added to the original set.
- Iteration: A new decision tree is built using this augmented feature set, and the process repeats until no new, useful features can be generated.
The paper also introduces a metric called dispersion (Δ) to quantify the "difficulty" of a concept given a feature set. A higher dispersion means the concept is harder to learn. The goal of feature construction here is to reduce this dispersion.
Datasets
The authors created three synthetic disjunctive boolean concepts (y1, y2, y3) to serve as their "datasets." These concepts were generated based on specific logical operations of binary attributes. They then used these generated data points as input to the CITRE feature construction module and subsequently to the back-propagation algorithm. No standard, publicly available datasets were used.
Key Insights and Ideas
- Feature Representation is Key: The paper strongly suggests that the "difficulty" of a learning problem for BP is heavily influenced by the quality of the input features. Poorly chosen features lead to high concept dispersion and slow learning.
- Automated Feature Engineering: CITRE provides an automated way to perform feature engineering, which is typically a manual and time-consuming process. This automation is crucial for practical applications.
- Dispersion as a Quality Metric: The concept of dispersion (Δ) offers a quantifiable way to assess feature set quality and predict learning difficulty. Lower dispersion correlates with easier learning.
- Feature Construction Reduces Complexity: By creating new features that capture interactions, the complexity of the concept in the feature space is reduced, making it easier for BP to find a solution.
- 100% Accuracy Achieved: Remarkably, even with a reduced set of features, the BP network achieved 100% classification accuracy on the test data.
Key Results
- Significant Speedup: The most striking result is the dramatic reduction in BP convergence time and the number of epochs required. For the final feature sets generated by CITRE, the number of epochs was reduced to about half compared to the initial feature sets. The convergence time dropped by an order of magnitude in some cases.
- Reduced Feature Set Size: The feature construction process often led to a reduction in the number of effective attributes needed to define the concepts, which in turn reduced the size of the neural network (fewer input and hidden units).
- Improved Performance: The paper demonstrates a clear trend: as the dispersion (Δ) of the concept decreases through feature construction, the performance of the BP algorithm (in terms of speed) increases considerably.
Limitations and Future Work
- Synthetic Data: The experiments were conducted on synthetic data. The authors acknowledge that performance with noisy or real-world data remains to be investigated.
- Focus on Disjunctive Concepts: The study focused on disjunctive boolean concepts. It's unclear how well this approach would generalize to more complex, non-disjunctive concept types.
- Serial Processing: The paper notes that the time taken per epoch is influenced by serial processing. Future work could explore how this approach performs on parallel computing architectures (like the Connection Machine) where the impact of network size might be different.
- Integration with Other Methods: The authors suggest that combining their feature construction approach with other methods for increasing BP convergence speed (like second-order gradient methods) could lead to even greater improvements.
Summary updated