A Comparative Study of the Practical Characteristics of Neural Network and Conventional Pattern Classifiers

NeurIPS 1990

TL;DR

This paper dives deep into the real-world performance of seven different pattern classifiers, including neural networks and traditional methods, across artificial and speech recognition tasks. It reveals that while error rates can be surprisingly similar, the practical trade-offs in training time, memory, and computational cost can vary dramatically, suggesting that "best" classifier choice hinges on these practicalities, not just accuracy.

What Problem Does This Paper Address?

In the realm of pattern recognition, there's often a strong focus on achieving the lowest classification error rate, especially with the rise of neural networks. However, many studies tend to concentrate on specific architectures (like backpropagation) or use artificial datasets, potentially overlooking crucial practical considerations. This paper aims to bridge that gap by comparing a diverse set of classifiers – both neural network-based and conventional – on real-world tasks, evaluating not just their accuracy but also their efficiency in terms of training time, classification speed, and memory footprint. The goal is to provide a more holistic understanding of their "practical characteristics" and guide practitioners in selecting the most suitable classifier for a given application.

Methods

The study implements and evaluates seven distinct pattern classifiers:

  • Neural Networks:
    • Radial Basis Function (RBF) network
    • High-order polynomial GMDH network
  • Conventional Classifiers:
    • Gaussian mixture model
    • Linear decision tree
    • K-nearest neighbor (KNN)
    • KD-tree
    • Condensed K-nearest neighbor (CKNN)

These classifiers were implemented on a serial computer and tuned using cross-validation and training data to find optimal global parameters (e.g., basis function width for RBF, polynomial order for GMDH, number of neighbors for KNN variants). The complexity of some classifiers, like the linear tree and GMDH, was carefully matched to the training data to avoid overfitting or underfitting, using techniques like pruning and information theoretic measures. Two novel variants were also introduced: a multi-scale RBF classifier and an improved CKNN classifier designed to reduce memory requirements and data order dependency.

Datasets

The classifiers were evaluated on a combination of:

  • Artificial Tasks: Two tasks with two input dimensions, designed to require either a single convex or two disjoint decision regions. These are the same artificial tasks used in a previous study (Lee and Lippmann, 1989) for continuity.
  • Speech Recognition Tasks:
    • Digit Recognition: 7 digits, 22 cepstral inputs, 16 talkers, with 70 training and 112 testing patterns per talker.
  • Vowel Recognition: 10 vowels, 2 formant frequency inputs, 67 talkers, with 338 training and 333 testing patterns.

Key Insights and Ideas

  • Error Rate Equivalence: When classifiers are sufficiently powerful, well-tuned, and have enough training data, their classification error rates can be remarkably similar. This challenges the notion that one classifier is universally "better" based solely on accuracy.
  • The Practicality Gap: The real differentiator lies in practical characteristics. Training time, classification speed, and memory usage can vary by orders of magnitude between classifiers, even if their error rates are comparable.
  • Trade-offs are Key: There's a clear inverse relationship: classifiers that are fast to train and classify often consume more memory, and vice-versa. Understanding these trade-offs is crucial for real-world deployment.
  • Generalization Matters: Some classifiers, like RBF and Gaussian mixture, generalize well even in high-dimensional spaces with limited data, while others, like linear trees and GMDH, struggle in such scenarios.
  • Tuning and Complexity are Critical: The performance of many classifiers is highly sensitive to the careful selection and tuning of global parameters and matching classifier complexity to the data's complexity.

Key Results

  • On the bullseye, disjoint, and vowel problems, statistical analysis showed no significant difference in error rates between the tested classifiers at a 0.01 significance level.
  • For the more challenging digit recognition task (limited training data, high dimensionality), RBF and Gaussian mixture classifiers achieved very low error rates, demonstrating good generalization. GMDH and linear tree classifiers performed poorly.
  • Training Time vs. Memory:
    • KNN and KD-tree classifiers train quickly but require substantial memory.
  • Gaussian mixture and linear tree classifiers use little memory but take longer to train.
  • RBF and CKNN fall in the intermediate range for both.
  • GMDH was the slowest to train, often taking 10-100 times longer than RBF or KD-tree.
  • Classification Time vs. Memory:
    • Linear tree classifiers were extremely fast and memory-efficient.
  • GMDH was the slowest and most memory-intensive.
  • Generally, classification time scaled with memory usage.
  • The improved CKNN classifier showed significant speedups over conventional KNN on the digit problem due to a reduced number of stored patterns.

Limitations and Future Work

  • Limited Classifier Set: While seven classifiers were evaluated, the authors acknowledge that many other alternative classifiers exist and could be explored.
  • Serial Implementation: All classifiers were implemented on a serial computer. Performance on parallel architectures might differ.
  • Specific Tasks: The study focused on specific artificial and speech recognition tasks. Performance might vary on different types of data or problems.

The authors suggest that researchers should invest time in understanding the wide array of available classifiers and their practical trade-offs, guiding their selection based on memory, implementation ease, computational resources, and time constraints, rather than solely on marginal differences in error rates. Future work could involve exploring genetic algorithms for further memory reduction in KNN classifiers, as hinted at in the references.

Summary updated

BibTeX
@inproceedings{1990comparative,
  title = {{A Comparative Study of the Practical Characteristics of Neural Network and Conventional Pattern Classifiers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
}

Loading related content...

Figure