On the Use of Projection Pursuit Constraints for Training Neural Networks

Nathan IntratorNeurIPS 1992

Introduction

In high-dimensional data environments, machine learning models frequently encounter the "curse of dimensionality." As the number of input features expands, data becomes increasingly sparse, making parameter estimation highly susceptible to overfitting. Standard back-propagation, which strictly minimizes the error between predictions and true labels (supervised learning), often converges on "noise-based" solutions in these high-dimensional spaces that fail to generalize. To address this, Nathan Intrator introduces a hybrid training methodology that integrates supervised learning with unsupervised "exploratory projection pursuit." By embedding structural priors into the network, the model is guided toward extracting more meaningful, robust features.

Methodology: Hybridizing Supervised and Unsupervised Learning

The core innovation of this paper is the fusion of two classical statistical techniques within a neural network architecture:

  1. Projection Pursuit Regression (PPR): A supervised technique that approximates functions by analyzing low-dimensional projections of the data.
  2. Exploratory Projection Pursuit (EPP): An unsupervised method that searches for "interesting" data projections—specifically, those deviating significantly from a normal (Gaussian) distribution.

Rather than relying solely on output error minimization, the proposed framework introduces a cost/complexity penalty term to the weight update equation of the hidden layer. This penalty is governed by a "projection index." Specifically, the architecture employs a variant of the BCM (Bienenstock, Cooper, and Munro) neuron model. Biologically inspired, the BCM model functions as an unsupervised feature extractor designed to identify multi-modality (clustering) within the data.

By integrating this BCM-based penalty into the back-propagation cost function, the network is constrained to discover hidden representations that satisfy two criteria: they must be effective for the downstream classification task, and they must capture inherent, clustered structures present in the raw input.

Key Insights

  • Normality is Uninformative: In high-dimensional spaces, the majority of random data projections resemble a featureless Gaussian "cloud." The discriminative information necessary for classification typically resides in projections that are distinctly non-Gaussian, such as those exhibiting clear clusters.
  • Unsupervised Regularization: The BCM penalty serves as a potent regularizer. It effectively instructs the network to ensure that the features selected to minimize label error also correspond to distinct clusters in the underlying input space.
  • Decoupling Features from Labels: This hybrid approach partially separates the search for optimal data projections from the search for the regression function. This decoupling makes the optimization process significantly more resilient to the noise inherent in high-dimensional datasets.

Empirical Validation

The hybrid method was rigorously evaluated on two high-dimensional, real-world classification tasks:

  • Speech Recognition (TIMIT Speech Database): The task involved classifying voiceless stop consonants. Following pre-processing with a cochlear model, the input space consisted of 5,040 dimensions (84 channels × 60 time slices). The hybrid model achieved an average error rate of 18.8%, significantly outperforming a standard back-propagation network's error rate of 20.9% ($P < .0013$).
  • Face Recognition (MIT Media Lab Face Database): This dataset comprised gray-level images of 16 individuals under varying conditions, resulting in a 2,294-dimensional input space (37 × 62 pixels) after alignment. An ensemble of hybrid networks achieved a remarkably low error rate of 0.62%. In contrast, a comparable ensemble of standard back-propagation networks yielded double the error at 1.2%. Furthermore, single standard networks exhibited high instability, with error rates fluctuating between 2.5% and 6%, underscoring the stabilizing effect of the projection pursuit constraints.

Limitations and Future Directions

While highly effective, the approach presents certain limitations:

  • Index Selection: The efficacy of the method heavily depends on the choice of the projection index. While a multi-modality index is optimal for classification tasks, alternative indices (such as those based on the Karhunen-Loève expansion) may be required for tasks like image compression.
  • Computational Complexity: Although the BCM variant is relatively efficient, applying complex penalty terms to every hidden unit could introduce substantial computational overhead, particularly if scaled to modern, large-scale architectures.
  • Scaling to Deep Networks: The empirical validation in this paper is restricted to shallow networks with a single hidden layer. The dynamics of how these projection pursuit constraints interact across multiple layers in deep neural networks remain an open question for future research.

Summary updated

BibTeX
@inproceedings{intrator1992use,
  author = {Nathan Intrator},
  title = {{On the Use of Projection Pursuit Constraints for Training Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1992},
}
Figure