Introduction
In deep learning, there is a persistent trend toward increasing model depth and complexity. However, for classic, smaller datasets like the Iris flower dataset, "over-engineering" can lead to diminishing returns, computational inefficiency, or failure to converge. This study investigates the structural "sweet spot" for feed-forward neural networks, aiming to identify the optimal number of layers, neurons, and activation functions for high-precision classification.
Methodology
The research utilizes a standard Feed-Forward Neural Network trained via the Backpropagation algorithm. To maintain a controlled environment, the learning rate ($\eta$) was kept constant at 0.02. The experimental design systematically varied the network's depth (from 0 to 5 hidden layers) and width (5 to 12 neurons per layer).
To evaluate efficiency, the author introduced a metric $Q$ (Total Updates), defined as the product of the total number of weights ($q$) and the number of iterations ($r$) required for convergence. The study used the standard Iris dataset, consisting of 150 instances across three classes, split 70% for training and 30% for testing.
Activation Functions and Performance
A critical component of the study was the comparison of four distinct activation functions: Logistic (Sigmoid), Hyperbolic Tangent (Tanh), Arctangent, and a Modified Hyperbolic Tangent proposed by Yann LeCun.
The Hyperbolic Tangent (Tanh) emerged as the most effective "uniform" function, consistently reaching target precision with fewer iterations across various architectures. In contrast, the Logistic function required significantly more effort; for instance, a 1-layer setup with 10 neurons took 10,811 iterations to converge, while a similar Tanh-based network reached the goal in only 259 iterations.
Key Insights
- The Linear Limit: Networks with zero hidden layers failed to classify the Iris dataset. While one class is linearly separable, the other two are not, necessitating at least one hidden layer to resolve non-linear boundaries.
- The "Too Big" Penalty: Adding more than two hidden layers often resulted in failure. For a small dataset of 150 samples, a 5-layer network possesses too many weights to adjust effectively, often failing to converge within the 20,000-iteration limit.
- Learning Rate Sensitivity: The "Modified Tangent" performed poorly in these specific tests. This is likely because its steeper derivative required a more finely tuned learning rate than the fixed $\eta = 0.02$ used in the study, highlighting that architecture and hyperparameters are deeply intertwined.
Results and Conclusion
For the Iris dataset, the optimal architecture consists of 1 or 2 hidden layers. The most efficient configuration found was a single hidden layer with 5 neurons using the Tanh activation function, totaling only 15,540 weight updates.
While these findings provide a clear guideline for small-scale pattern recognition, the study highlights that architecture and hyperparameters must be balanced. Future work should explore how these optimal size rules scale with increasing data complexity and investigate dynamic learning rate adjustments to better accommodate diverse activation functions.
Summary updated