Abstract
This paper presents a "heuristics-free" approach to unsupervised feature learning by integrating Independent Component Analysis (ICA) with nonparametric Bayesian priors. Utilizing the Indian Buffet Process (IBP), the proposed SC-IBP-ICA model automatically determines the optimal number of features for video data, bypassing the need for extensive cross-validation. Experimental results on action recognition benchmarks demonstrate that this approach achieves competitive accuracy with significantly reduced manual tuning and improved generalization.
Introduction
In the field of unsupervised feature learning, Independent Component Analysis (ICA) is a powerful tool for learning receptive fields that mimic the human visual cortex. However, traditional deep learning architectures suffer from a "manual labor" bottleneck: researchers must pre-define the number of latent features (layer width) before training. Finding the optimal configuration typically involves running dozens of cross-validation experiments, which is both time-consuming and computationally expensive. This dependency prevents models from being truly autonomous when encountering unfamiliar data types.
Methods
The authors introduce IBP-ICA and its deep, convolutional counterpart, SC-IBP-ICA, to automate architecture discovery.
- The Indian Buffet Process (IBP): The core innovation is using the IBP as a nonparametric prior. This allows the model to "pick" the necessary number of features from an infinite pool based on the complexity of the data, allowing the architecture to grow or shrink automatically.
- Spike-and-Slab Prior: To maintain sparsity, a spike-and-slab distribution is used to effectively "turn off" unnecessary features for specific inputs.
- Hybrid Variational Inference: To ensure scalability, the authors developed a hybrid algorithm combining fast variational inference with local Metropolis-Hastings steps, allowing the model to propose adding or removing features on the fly.
- Stacked Convolutional Architecture: The process is applied to video patches and stacked greedily. This creates a hierarchy where lower layers learn simple edges and motion, while higher layers capture complex action patterns.
Experimental Setup
The model was evaluated on three standard action recognition benchmarks:
- Hollywood2: Human actions extracted from movies.
- YouTube: Realistic, "in the wild" action sequences.
- KTH: Human actions in controlled environments.
Key Results and Insights
Automatic Complexity Discovery
The model successfully learns the required feature count without human intervention. For example, on the KTH dataset, the model automatically settled on approximately 293 features for the first layer. This Bayesian approach is largely self-correcting, avoiding the need for careful tuning of learning rates or dropout.
Performance and Efficiency
On the YouTube dataset, the two-layer SC-IBP-ICA achieved 75.4% accuracy, performing nearly as well as a manually tuned ISA model (75.8%) but without the exhaustive search for hyperparameters.
The automatic feature discovery was only 0.4% to 0.7% less accurate than an exhaustive 40-model cross-validation search, while saving 100% of the researcher's manual tuning time. Once trained, the model processes video efficiently at 0.12 to 0.38 seconds per frame.
Superior Generalization
In transfer learning tests, SC-IBP-ICA outperformed traditional Independent Subspace Analysis (ISA), suggesting that the Bayesian prior helps the model learn more robust and universal features.
Limitations and Future Work
Currently, the model is trained "greedily" layer-by-layer. The authors suggest that moving toward fully end-to-end joint inference across all layers could further improve performance. Additionally, while feature extraction is fast, the initial Bayesian inference during training is mathematically complex. Future work may extend this nonparametric approach to other deep learning building blocks, potentially leading to fully "self-architecting" neural networks.
Summary updated