Feature space learning model

Renchu Guan, Xu Wang, Maurizio Marchese, Mary Qu Yang, Yanchun Liang, Chen Yang

Abstract

This paper introduces the Feature Space Learning (FSL) model, a semi-supervised framework that adaptively evolves its feature space during the clustering process. By leveraging linguistic principles like Zipf’s law and word burstiness, FSL achieves performance gains similar to deep learning—discovering compact, descriptive representations—without the prohibitive training costs or lack of interpretability. Experimental results on Reuters-21578 and 20 Newsgroups datasets show that FSL-based algorithms significantly outperform traditional unsupervised and semi-supervised methods, achieving up to a 73.6% improvement in F-measure while reducing the feature space footprint by up to 41.3%.

Introduction

Text data analysis often faces the dual challenges of high dimensionality and the "Matthew Effect" of error propagation in semi-supervised learning. Traditional parametric models often lack the complexity to capture nuanced patterns, while deep learning models, though powerful, are computationally expensive and difficult to interpret. The Feature Space Learning (FSL) model seeks a middle ground by "learning" an optimized feature space from unlabeled data through an iterative, statistically-grounded process.

Methodology

The FSL model integrates clustering and feature space updating into a unified iterative loop. The process begins with initial seeds and proceeds through similarity computing and clustering.

Key components include:

  • Trust Sample Selection: The model identifies unlabeled samples that are classified with high confidence to serve as guides for the next iteration, mitigating the risk of semantic shift.
  • Rich-Information Features (RIFs): A statistical "firewall" based on Zipf’s law (frequency) and word burstiness (topicality) ensures that only relevant features are used to span the evolving feature space.
  • Adaptive Upgrading: The feature space is dynamically updated, adding new relevant features and adjusting weights to distill high-dimensional inputs into compact, topical representations.
Flowchart of the FSL process

As the model iterates, the feature space transitions through various states (State 0 to State 4), refining cluster centers and sample distributions as illustrated in the conceptual diagram below.

Conceptual evolution of the feature space

Experimental Results

The FSL framework was evaluated using the Reuters-21578 and 20 Newsgroups (20NG) benchmarks. Key findings include:

  • Performance Gains: FSSK-means achieved a 73.6% higher F-measure than standard k-means on the 20NG dataset.
  • Robustness: FSSAP outperformed Incremental Affinity Propagation (IAP) by 14.3% in F-measure, demonstrating the effectiveness of the RIF "firewall" in preventing noise accumulation.
  • Efficiency: The model successfully compressed the feature space by 13.4–41.3% (e.g., from 8,277 to under 3,500 features) while simultaneously improving clustering accuracy.

Discussion and Conclusion

FSL provides a unique combination of performance and interpretability. Unlike the abstract latent spaces of neural networks, FSL produces feature spaces that directly reflect the subject matter of the clusters, allowing for clear visualization. While the iterative nature of the model introduces higher computational complexity than basic clustering algorithms, the resulting accuracy and dimensionality reduction offer a compelling trade-off. Future research will explore optimization strategies and the application of FSL to non-textual data types.

Summary updated

Abstract

With the massive volume and rapid increasing of data, feature space study is of great importance. To avoid the complex training processes in deep learning models which project original feature space into low-dimensional ones, we propose a novel feature space learning (FSL) model. The main contributions in our approach are: (1) FSL can not only select useful features but also adaptively update feature values and span new feature spaces; (2) four FSL algorithms are proposed with the feature space updating procedure; (3) FSL can provide a better data understanding and learn descriptive and compact feature spaces without the tough training for deep architectures. Experimental results on benchmark data sets demonstrate that FSL-based algorithms performed better than the classical unsupervised, semi-supervised learning and even incremental semi-supervised algorithms. In addition, we show a visualization of the learned feature space results. With the carefully designed learning strategy, FSL dynamically disentangles explanatory factors, depresses the noise accumulation and semantic shift, and constructs easy-to-understand feature spaces.
BibTeX
@article{guan2019feature,
  author = {Renchu Guan and Xu Wang and Maurizio Marchese and Mary Qu Yang and Yanchun Liang and Chen Yang},
  title = {{Feature space learning model}},
  journal = {J. Ambient Intell. Humaniz. Comput.},
  year = {2019},
  doi = {10.1007/s12652-018-0805-4},
}
Figure