Semisupervised Feature Analysis by Mining Correlations Among Multiple Tasks

Xiaojun Chang, Yi YangIEEE TNNLS 2017 · OpenReview

TL;DR

This paper introduces a novel semi-supervised feature selection method that leverages correlations between features and across multiple related tasks. By jointly learning from labeled and unlabeled data and exploiting shared task structures, their approach significantly improves performance, especially in low-data regimes, outperforming existing methods on various multimedia tasks.

What Problem Does This Paper Address?

In many real-world applications, especially in computer vision and multimedia, datasets are characterized by high-dimensional feature spaces. Not all features are equally informative; many can be noisy or redundant, hindering the performance of downstream tasks like classification or annotation. Feature selection aims to address this by identifying and retaining only the most relevant features.

Traditional feature selection methods often suffer from several limitations:

  1. Supervised Nature: Many methods rely heavily on labeled data, which is expensive and time-consuming to obtain in large quantities.
  2. Individual Feature Evaluation: They tend to evaluate features one by one, neglecting the crucial correlations that exist between features.
  3. Task Isolation: Feature selection is often performed independently for each task, failing to exploit the valuable shared information that can exist between related tasks.

This paper tackles these issues by proposing a framework that is semi-supervised (uses both labeled and unlabeled data), considers feature interdependencies, and leverages knowledge from multiple related tasks simultaneously.

Methods

The core of the proposed method, named Semi-supervised Feature selection by Mining Correlations among multiple tasks (SFMC), is a unified optimization framework. Here's a breakdown of its key components:

  • Multi-Task Learning: The algorithm assumes that multiple related tasks share underlying common structures. It aims to learn feature selection functions for each task jointly, allowing knowledge transfer between them. This is achieved by incorporating a trace norm regularization term on the combined feature selection matrices, encouraging shared components.
  • Semi-Supervised Learning: To overcome the reliance on labeled data, SFMC incorporates manifold learning. This technique utilizes both labeled and unlabeled data to learn the underlying data structure, effectively leveraging the information present in the larger unlabeled set. A Laplacian matrix is constructed to capture this manifold structure.
  • Joint Feature Selection: Instead of selecting features individually, SFMC considers correlations between features. This is achieved through an $l_{2,1}$-norm regularization on the feature selection matrices for each task, promoting sparsity across features for each task.
  • Optimization: The objective function, combining loss, $l_{2,1}$-norm, trace norm, and manifold regularization, is non-smooth and challenging to solve directly. The authors propose an iterative optimization algorithm that efficiently converges to a solution.

The mathematical formulation involves minimizing a loss function (least squares is used) subject to these regularization terms, aiming to find optimal feature selection matrices ($W_l$) for each task ($l$) and a predicted label matrix ($F_l$).

Datasets

The paper evaluates SFMC on four diverse multimedia datasets:

  • Columbia Consumer Video dataset (CCV): Used for video classification.
  • NUS-WIDE dataset: Used for image annotation.
  • HMDB dataset: Used for human motion recognition.
  • HumanEva dataset: Used for 3D motion data analysis.

These datasets cover a range of modalities and tasks, demonstrating the generalizability of the proposed approach.

Key Insights and Ideas

  • Synergy of Multi-Task and Semi-Supervised Learning: The paper convincingly demonstrates that combining multi-task learning with semi-supervised learning is more powerful than either approach alone, especially when labeled data is scarce. The shared task structure amplifies the benefits of using unlabeled data.
  • Importance of Feature Interdependencies: By explicitly modeling correlations between features (via $l_{2,1}$-norm) and across tasks (via trace norm), SFMC achieves superior performance compared to methods that treat features or tasks in isolation.
  • Robustness in Low-Data Regimes: A recurring theme is that SFMC's advantages are most pronounced when the percentage of labeled training data is low (e.g., 1% or 5%). This highlights its practical value in scenarios where data annotation is a bottleneck.
  • Manifold Learning for Structure Discovery: The integration of manifold learning effectively captures the underlying data manifold, providing a richer representation that aids feature selection, particularly with unlabeled data.

Key Results

The experimental results consistently show that SFMC outperforms several state-of-the-art feature selection algorithms across all tested applications (video classification, image annotation, human motion recognition, and 3D motion data analysis).

  • Significant Gains: SFMC consistently achieves higher Mean Average Precision (MAP) scores. For instance, in video classification with 5% labeled data, SFMC outperforms the second-best algorithm by 6.6%.
  • Low-Data Advantage: The performance gap between SFMC and other methods widens significantly as the amount of labeled data decreases.
  • Parameter Sensitivity: While the method has parameters (α, β, γ), the experiments suggest that its performance is relatively robust, and reasonable tuning can yield excellent results. The parameter γ, controlling shared information across tasks, is shown to be beneficial.

Limitations and Future Work

  • Computational Complexity: While the authors propose an iterative algorithm with fast convergence, the optimization problem itself can still be computationally intensive, especially with a very large number of tasks or features.
  • Assumption of Shared Structure: The effectiveness of the multi-task component relies on the assumption that the tasks are indeed related and share common structures. If tasks are unrelated, the performance might not improve or could even degrade.
  • Future Work: The paper implicitly suggests future work could explore more sophisticated manifold learning techniques, investigate different loss functions, or extend the framework to handle even larger-scale, more complex multimedia data. The authors also mention studying the influence of the number of selected features, which is a common avenue for further analysis.

Summary updated

Abstract

In this paper, we propose a novel semisupervised feature selection framework by mining correlations among multiple tasks and apply it to different multimedia applications. Instead of independently computing the importance of features for each task, our algorithm leverages shared knowledge from multiple related tasks, thus improving the performance of feature selection. Note that the proposed algorithm is built upon an assumption that different tasks share some common structures. The proposed algorithm selects features in a batch mode, by which the correlations between various features are taken into consideration. Besides, considering the fact that labeling a large amount of training data in real world is both time-consuming and tedious, we adopt manifold learning, which exploits both labeled and unlabeled training data for a feature space analysis. Since the objective function is nonsmooth and difficult to solve, we propose an iteractive algorithm with fast convergence. Extensive experiments on different applications demonstrate that our algorithm outperforms the other state-of-the-art feature selection algorithms.
BibTeX
@article{chang2017semisupervised,
  author = {Xiaojun Chang and Yi Yang},
  title = {{Semisupervised Feature Analysis by Mining Correlations Among Multiple Tasks}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2017},
  url = {https://openreview.net/forum?id=gWdzEPLVey},
}
Figure