TL;DR
This paper introduces Feature Weighted Non-negative Matrix Factorization (FNMF), a novel approach that addresses the limitation of standard NMF by adaptively learning the importance of features. FNMF uses multiple feature weighting components to capture data diversity and achieves state-of-the-art performance in representation learning and clustering tasks.
What Problem Does This Paper Address?
Non-negative Matrix Factorization (NMF) is a powerful technique for data representation and clustering, decomposing a data matrix into a linear combination of basis vectors. This allows for learning interpretable, low-dimensional representations. However, a key limitation of traditional NMF is its assumption that all features are equally important. In real-world scenarios, features often have varying degrees of relevance for different tasks or even different samples within the same dataset. Some methods attempt to address this by projecting data into a subspace, but this can distort original feature attributes and fail to account for sample-specific feature importance. This paper aims to overcome these limitations by developing an NMF variant that explicitly learns and utilizes feature weights.
Methods
The core innovation of this paper is the Feature Weighted Non-negative Matrix Factorization (FNMF). FNMF introduces a mechanism to learn feature weights adaptively. The model is formulated to simultaneously learn:
- Feature Weights ($\Theta$): These capture the importance of each feature.
- Multiple Feature Weighting Components ($P$): To account for the diversity of samples, each sample can be associated with different sets of feature weights. This allows for capturing how feature importance can vary across different data points.
- Basis Vectors ($U$) and Coefficient Matrix ($V$): These are the standard NMF components, but they are learned using the weighted features.
The objective function incorporates terms for data reconstruction, feature weighting diversity, and a graph regularization term to preserve local data structure. An efficient iterative optimization algorithm is proposed to solve the FNMF problem, decomposing it into sub-problems for updating each variable. The convergence of this algorithm is theoretically proven.
Datasets
The paper evaluates FNMF on a variety of datasets:
- Synthetic Dataset: A custom dataset with 900 samples, 7 dimensions, and 3 classes was created to specifically test feature weighting. The first two dimensions exhibit clear Gaussian distributions, while the last five are noisy.
- Real-World Datasets: The following standard datasets were used for evaluation:
- YALE (face recognition)
- ORL (face recognition)
- BA (Binary Alphadigits)
- USPS (handwritten digits)
- Semeion (handwritten digits)
- CNAE-9 (business document descriptions)
- Glass (chemical composition)
- Mfeat (handwritten numerals)
Key Insights and Ideas
- Feature Importance is Crucial: The paper strongly emphasizes that features are not created equal, and their varying importance is a critical factor for effective data representation and clustering.
- Sample-Specific Feature Weights: The introduction of multiple feature weighting components ($P$) is a key insight, recognizing that feature importance can be context-dependent on the sample itself, not just a global property.
- Synergy of Feature Weighting and NMF: FNMF demonstrates that integrating feature weighting directly into the NMF framework, rather than as a pre-processing step, leads to more robust and informative representations.
- Handling Noisy Features: The experimental results, particularly on the noised Yale dataset, highlight FNMF's ability to effectively mitigate the impact of irrelevant or noisy features.
Key Results
- State-of-the-Art Performance: FNMF consistently achieved the best performance in terms of Clustering Accuracy (ACC) and Normalized Mutual Information (NMI) across most of the evaluated real-world datasets, outperforming established NMF variants.
- Robustness to Noise: On the synthetic dataset with added noisy dimensions and the noised Yale dataset, FNMF showed superior performance compared to other methods, demonstrating its ability to filter out irrelevant information.
- Efficiency: The proposed optimization algorithm converges quickly, typically within twenty iterations on the tested datasets.
- Necessity of Multiple Components: Experiments showed that using multiple feature weighting components (m=3) improved performance over a single component, indicating the importance of capturing feature diversity. However, too many components could lead to performance degradation.
Limitations and Future Work
- Computational Cost: While efficient, the paper notes that FNMF has a slightly higher computational cost than basic NMF due to the incorporation of multiple feature weighting components and graph regularization.
- Parameter Sensitivity: The paper includes figures showing parameter sensitivity (Figures 7 and 8), indicating that while performance is generally stable, extreme parameter values can degrade results.
- Future Work: The authors propose two main directions:
- Developing a deep model of FNMF to handle larger-scale datasets.
- Extending FNMF to a semi-supervised learning scheme.
Summary updated