Pixelhop++: A Small Successive-Subspace-Learning-Based (Ssl-Based) Model For Image Classification

Yueru Chen, Mozhdeh Rouhsedaghat, Suya You, Raghuveer Rao, C.-c. Jay Kuo

TL;DR

This paper introduces PixelHop++, an improved image classification model that significantly reduces model size while maintaining high accuracy. By decoupling spatial and spectral components and introducing a novel feature selection method, PixelHop++ offers fine-grained control over model complexity, making it suitable for resource-constrained environments.

What Problem Does This Paper Address?

The increasing demand for machine learning models in mobile and edge computing applications necessitates the development of smaller, more efficient models. While techniques like compression and acceleration exist for large networks, designing inherently small, interpretable models from scratch is a crucial research area. Existing interpretable models like PixelHop, while effective, can still be computationally intensive. This paper aims to address the model size challenge of interpretable, successive subspace learning (SSL)-based models by proposing PixelHop++, a more compact and flexible architecture.

Methods

PixelHop++ builds upon the Successive Subspace Learning (SSL) principle, similar to its predecessor PixelHop, but introduces several key modifications to reduce model size and improve efficiency:

  • Channel-wise (c/w) Saab Transform: The traditional Saab transform, used for dimension reduction, is computationally expensive. PixelHop++ replaces this with a channel-wise Saab transform. This is motivated by the observation that spectral components of the Saab transform are weakly correlated. By decoupling these components, the transform can be applied independently to each spectral channel, significantly reducing filter size and computational cost.
  • Tree-Decomposed Feature Representation: A novel hierarchical feature representation is introduced. Features are extracted in stages (hops), and at each stage, a tree structure is formed. Leaf nodes in this tree represent 1D features derived from response maps. This structure allows for a more organized and potentially more discriminative feature set.
  • Cross-Entropy-Guided Feature Selection: To further reduce model size and improve classification performance, features from the leaf nodes are ranked based on their cross-entropy values. Features with lower cross-entropy are considered more discriminative. A subset of the top-ranked features is then selected for classification, allowing for fine-grained control over the number of features used.
  • Hyperparameter Control: Two key hyperparameters, the energy threshold (T) for determining leaf vs. intermediate nodes and the number of selected features (Ns), allow for flexible control over the model's size and complexity.

Datasets

The proposed PixelHop++ method was evaluated on three standard image classification datasets:

  • MNIST: A dataset of handwritten digits (28x28 grayscale images).
  • Fashion MNIST: A dataset of clothing items (28x28 grayscale images).
  • CIFAR-10: A dataset of 10 object classes (32x32 color images).

For MNIST and Fashion MNIST, images were zero-padded to a size of 32x32 to match the input size of the PixelHop++ architecture.

Key Insights and Ideas

  • Spectral Decoupling is Key: The weak correlation between spectral components of the Saab transform is a crucial insight that enables the significant model size reduction achieved by the channel-wise Saab transform.
  • Hierarchical Features for Efficiency: The tree-decomposed feature representation naturally lends itself to hierarchical processing and allows for the identification of features at different levels of abstraction.
  • Cross-Entropy as a Discriminative Measure: Using cross-entropy to rank features provides an effective way to select the most informative features for classification, directly contributing to model compactness.
  • Fine-Grained Model Size Control: The combination of the energy threshold (T) and the number of selected features (Ns) offers unprecedented flexibility in tuning the model's size, allowing for a direct trade-off between model complexity and classification performance.

Key Results

  • PixelHop++ demonstrates the ability to significantly reduce model size. For example, a "small" PixelHop++ model requires approximately 2x, 6x, and 6x fewer parameters than LeNet-5 for MNIST, Fashion MNIST, and CIFAR-10, respectively.
  • Despite the reduction in model size, PixelHop++ maintains competitive classification accuracy. While LeNet-5 achieved slightly higher accuracy on MNIST and CIFAR-10, the "large" PixelHop++ model outperformed LeNet-5 on Fashion MNIST. The "small" PixelHop++ model achieved accuracy close to its larger counterpart with substantially fewer parameters.
  • The paper shows that by adjusting the energy threshold (T), model size can be reduced by up to 4x for MNIST with only a minor drop in test accuracy (0.51%). Similarly, reducing the number of selected features (Ns) also leads to smaller models with marginal accuracy degradation.

Limitations and Future Work

  • Hyperparameter Sensitivity: While T and Ns offer flexibility, their optimal values might require careful tuning for different datasets and tasks. The paper demonstrates their impact but doesn't provide a universal method for setting them.
  • Dataset Specificity: The experiments were conducted on relatively small datasets. Further evaluation on larger and more complex datasets would be beneficial to assess the generalizability of PixelHop++.
  • Future Work: The authors suggest exploring the impact of different kernel sizes for the c/w Saab transform and investigating more advanced feature selection techniques. They also imply that the interpretability of the tree-decomposed features could be further leveraged.

Summary updated

Abstract

The successive subspace learning (SSL) principle was developed and used to design an interpretable learning model, known as the PixelHop method,for image classification in our prior work. Here, we propose an improved PixelHop method and call it PixelHop++. First, to make the PixelHop model size smaller, we decouple a joint spatial-spectral input tensor to multiple spatial tensors (one for each spectral component) under the spatial-spectral separability assumption and perform the Saab transform in a channel-wise manner, called the channel-wise (c/w) Saab transform.Second, by performing this operation from one hop to another successively, we construct a channel-decomposed feature tree whose leaf nodes contain features of one dimension (1D). Third, these 1D features are ranked according to their cross-entropy values, which allows us to select a subset of discriminant features for image classification. In PixelHop++, one can control the learning model size of fine-granularity,offering a flexible tradeoff between the model size and the classification performance. We demonstrate the flexibility of PixelHop++ on MNIST, Fashion MNIST, and CIFAR-10 three datasets.
BibTeX
@inproceedings{chen2020pixelhop,
  author = {Yueru Chen and Mozhdeh Rouhsedaghat and Suya You and Raghuveer Rao and C.-C. Jay Kuo},
  title = {{Pixelhop++: A Small Successive-Subspace-Learning-Based (Ssl-Based) Model For Image Classification}},
  booktitle = {ICIP},
  year = {2020},
  doi = {10.1109/icip40778.2020.9191012},
}
Figure