Image Denoising with a Unified Schattern-p Norm and ℓq Norm Regularization

Jing Wang, Meng Wang, Xuegang Hu, Shuicheng Yan

Abstract

Online feature selection with dynamic features has become an active research area in recent years. However, in some real-world applications such as image analysis and email spam filtering, features may arrive by groups. Existing online feature selection methods evaluate features individually, while existing group feature selection methods cannot handle online processing. Motivated by this, we formulate the online group feature selection problem, and propose a novel selection approach for this problem. Our proposed approach consists of two stages: online intra-group selection and online inter-group selection. In the intra-group selection, we use spectral analysis to select discriminative features in each group when it arrives. In the inter-group selection, we use Lasso to select a globally optimal subset of features. This 2-stage procedure continues until there are no more features to come or some predefined stopping conditions are met. Extensive experiments conducted on benchmark and real-world data sets demonstrate that our proposed approach outperforms other state-of-the-art online feature selection methods.

Introduction

Feature selection is a standard technique used to reduce the dimensionality of data by discarding irrelevant or redundant features. Traditionally, this is performed "offline," requiring the algorithm to access the entire dataset and all its features simultaneously. However, in dynamic environments like spam filtering or streaming image analysis, features arrive continuously over time. This necessitates online feature selection, where the model must decide whether to keep or discard a feature the moment it arrives.

A significant gap in existing research is that online feature selection methods (e.g., Grafting, Alpha-investing, or OSFS) evaluate features individually. They ignore the fact that in many real-world scenarios, features arrive in logical groups. For example, in computer vision, an image might be represented by a group of shape descriptors followed by a group of color descriptors. While offline methods like Group Lasso handle grouped features, they cannot process streaming data. This paper bridges this gap by proposing the first framework for Online Group Feature Selection (OGFS).

The OGFS Algorithm

The OGFS algorithm processes streaming groups of features in two distinct stages:

  1. Online Intra-Group Selection (Local Filtering): When a new group of features arrives, OGFS uses spectral analysis to evaluate the discriminative power of each feature within that specific group. It relies on maximizing the distance between different classes while minimizing the distance between data points of the same class. A statistical t-test ensures the feature's discriminative power is mathematically significant compared to already selected features.
  2. Online Inter-Group Selection (Global Filtering): Once the best features from the new group are isolated, they are pooled with previously selected features. The algorithm then applies Lasso (a sparse linear regression model using an $\ell_1$ penalty). Lasso acts as a global referee, forcing the coefficients of redundant or less useful features to zero, effectively dropping them and keeping only the globally optimal subset.

Experimental Evaluation

The authors evaluated OGFS on a mix of eight benchmark datasets and three real-world image datasets (Soccer, Flower-17, and 15 Scenes).

Benchmark Results

OGFS outperformed the Alpha-investing baseline on 7 out of 8 benchmark datasets. A notable result was seen on the Leukemia dataset, where OGFS achieved a perfect 1.0 (100%) accuracy, compared to Alpha-investing's 0.65.

Performance comparison on benchmark datasets showing accuracy and dimensionality across various methods.

Real-World Image Datasets

On the massive "15 Scenes" image dataset, which contains over 21,000 features, OGFS successfully processed the stream and achieved the highest accuracy among online methods (0.54) while compressing the data to just 369 features. Competing methods like Fast-OSFS were unable to complete the task due to memory constraints.

Performance comparison on real-world image datasets (Soccer, Flower-17, 15 Scenes).

Key Insights and Conclusion

OGFS demonstrates that acknowledging group structures allows models to better balance relevance and redundancy. The two-stage filtering process is computationally efficient; by aggressively filtering features within a new group first, the global selection step remains tractable. Furthermore, the use of Lasso allows for the dynamic re-evaluation of the entire feature pool, overcoming the "first-come, first-served" bias of older online methods.

While OGFS currently assumes known group structures and is limited to supervised learning, its superior performance in accuracy and feature compactness marks a significant step forward in handling dynamic, grouped feature streams.

Summary updated

Abstract

Online feature selection with dynamic features has become an active research area in recent years. However, in some real-world applications such as image analysis and email spam filtering, features may arrive by groups. Existing online feature selection methods evaluate features individually, while existing group feature selection methods cannot handle online processing. Motivated by this, we formulate the online group feature selection problem, and propose a novel selection approach for this problem. Our proposed approach consists of two stages: online intra-group selection and online inter-group selection. In the intra-group selection, we use spectral analysis to select discriminative features in each group when it arrives. In the inter-group selection, we use Lasso to select a globally optimal subset of features. This 2-stage procedure continues until there are no more features to come or some predefined stopping conditions are met. Extensive experiments conducted on benchmark and real-world data sets demonstrate that our proposed approach outperforms other state-of-the-art online feature selection methods.
BibTeX
@misc{wang2014image,
  author = {Jing Wang and Meng Wang and Xuegang Hu and Shuicheng YAN},
  title = {{Image Denoising with a Unified Schattern-p Norm and ℓq Norm Regularization}},
  year = {2014},
  eprint = {1404.4774},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1404.4774},
}
Figure