TL;DR
This paper introduces Principal Feature Analysis (PFA), a novel method for selecting a subset of original features that retain essential information, outperforming traditional Principal Component Analysis (PCA) in certain applications. PFA offers a computationally efficient way to reduce dimensionality while preserving the physical meaning of the selected features, making it valuable for tasks like face tracking and image retrieval.
What Problem Does This Paper Address?
In many pattern recognition and classification tasks, reducing the dimensionality of data is crucial for efficient processing and analysis. While methods like Principal Component Analysis (PCA) are popular for this, they project data onto a new space using all original features, which can be computationally expensive and obscure the original feature meanings. The paper addresses the need for a feature selection method that identifies a subset of the original features that captures most of the essential information, similar to PCA's optimality criteria, but without requiring all features. This is particularly important in applications where acquiring or processing all features is costly (e.g., sensor costs) or where interpretability of the selected features is desired.
Methods
The paper proposes Principal Feature Analysis (PFA), a method that leverages the structure of principal components to select an optimal subset of original features. Unlike previous PCA-based methods that might be computationally expensive or select redundant features, PFA uses the eigenvectors (principal components) to infer relationships between original features.
The core idea is to analyze the weight vectors of each original feature across all principal components. Features with similar weight vectors are considered highly correlated. PFA then clusters these weight vectors and selects one representative feature from each cluster. This approach aims to capture the "spread" of data in the lower-dimensional space (similar to PCA's goal) and minimize reconstruction error, while also ensuring that the selected features are not redundant.
The algorithm involves:
- Computing the covariance/correlation matrix of the feature set.
- Calculating principal components and eigenvalues (standard PCA).
- Choosing the subspace dimension (q) based on desired retained variability.
- Clustering the absolute values of the feature weight vectors (derived from the principal components) using K-Means.
- Selecting the feature closest to the mean of each cluster as a principal feature.
Datasets
The paper evaluates PFA on two applications:
- Facial Motion Feature Points: Data captured from video sequences of human facial motion with markers on the face. The dataset consists of 40 facial points, represented by 80 features (horizontal and vertical directions).
- Content-Based Image Retrieval (Corel Dataset): A large dataset of images (17695 images, including "airplanes" and "American eagles" subsets). Features used include 9 color moments and 20 wavelet moments.
No new benchmarks were introduced; existing datasets were used for evaluation.
Key Insights and Ideas
- PFA offers a balance between PCA's optimality and feature interpretability: While PCA finds an optimal projection, it loses the original feature meaning. PFA aims to select a subset of original features that collectively achieve similar optimality properties as PCA, but the selected features retain their original physical meaning.
- Exploiting feature correlations through principal components: The key insight is that the structure of principal component coefficients reveals correlations between original features. By clustering these coefficients, PFA can identify groups of related features and select a representative from each.
- Reducing redundancy is crucial for performance: The experiments suggest that using too many features, especially noisy ones, can degrade performance. PFA's ability to select a parsimonious yet informative subset is a significant advantage.
- PFA can be computationally efficient: Compared to exhaustive subset search methods, PFA's complexity is comparable to PCA, making it practical for larger feature sets.
Key Results
- Facial Motion: PFA successfully identified a subset of facial points that effectively capture non-rigid motion. For example, it prioritized lip motion features for the lower face and eyebrow features for the upper face, aligning with physical models of facial movement. This reduced the number of points needed for tracking compared to using all points.
- Content-Based Image Retrieval: PFA demonstrated comparable or superior performance to using the full feature set and PCA. In one experiment, PFA selected 7 color and 15 wavelet features, achieving results comparable to using all 9 color and 20 wavelet features. In another, PFA's selected subset of 7 features ranked in the top 5% of all possible subset combinations for wavelet features.
Limitations and Future Work
- Choice of 'p' (number of clusters): The paper notes that choosing the number of clusters 'p' (which determines the number of selected features) is usually set to be slightly larger than 'q' (the desired subspace dimension), often 1-5 additional dimensions. The optimal strategy for selecting 'p' is not explicitly detailed.
- Comparison to Optimal Subsets: While PFA features are shown to be ranked highly (top 5%) among all possible subsets, the paper acknowledges that finding the absolute optimal subset according to the criteria in [4] is computationally infeasible for large feature sets. PFA provides a practical approximation.
- Future Work: The paper implies that PFA's effectiveness in reducing computational cost and retaining feature meaning makes it a promising direction for various pattern recognition and classification tasks. Further exploration of its application in other domains is suggested.
Summary updated