Transfer Learning Using Feature Selection

Paramveer S. Dhillon, Dean P. Foster, Lyle H. UngararXiv 2009 · arXiv

Abstract

In high-dimensional datasets where the number of features significantly exceeds the number of observations ($p \gg n$), traditional feature selection methods often suffer from overfitting or poor interpretability. This paper proposes a novel framework for feature selection using the Minimum Description Length (MDL) principle. By treating feature selection as an information-encoding problem, we introduce three methods—Multiple Inclusion Criterion (MIC), Three Part Coding (TPC), and Transfer-TPC—that facilitate transfer learning across related tasks and structured feature spaces. Our results across genomics and natural language processing tasks demonstrate that these methods achieve superior sparsity and predictive accuracy compared to standard regularization techniques.

Introduction

The challenge of "small $n$, large $p$" is ubiquitous in modern data science, particularly in fields like genomics and Natural Language Processing (NLP). While traditional transfer learning often focuses on mapping different tasks into a shared latent space, this research explores a different dimension: transferring knowledge about feature relevance.

The core premise is that related tasks often share the same underlying predictive features. By "borrowing strength" from related models or utilizing known structural relationships between features (such as biological pathways), we can improve the robustness of feature selection. This paper frames this problem through the lens of Information Theory, specifically the Minimum Description Length (MDL) principle, to create a principled approach to penalized regression.

Methodology: MDL-Based Feature Selection

The proposed methods utilize a penalized regression framework: $$\min_{\beta} ||Y - \hat{Y}||_2 + \lambda||\beta||_0$$ Unlike the common $\ell_1$ (Lasso) penalty, which is chosen for its convex properties, this work employs an $\ell_0$ penalty. The penalty $\lambda$ is not an arbitrary hyperparameter but is derived from the bits required to encode the model according to the MDL principle. The "best" model is defined as the one that minimizes the combined description length of the model and the data.

1. Multiple Inclusion Criterion (MIC)

MIC is designed for simultaneous transfer across multiple related tasks. It utilizes a coding scheme where the cost of including a feature in a subsequent task is reduced if that feature has already been selected for a previous task. This "discount" reflects the lower informational cost of identifying a feature that is already known to be relevant in a related context.

2. Three Part Coding (TPC)

TPC addresses structured feature spaces where features are grouped into classes (e.g., genes within the same biological pathway). The coding scheme accounts for the index of the feature class, the index of the feature within that class, and the coefficient value. Once a class is "activated" by the inclusion of one feature, the marginal cost of including additional features from the same class decreases, encouraging the model to select clusters of related features.

3. Transfer-TPC

Transfer-TPC extends these concepts to sequential transfer learning. It uses the frequency of feature inclusion in "source" tasks to build an informative prior. This prior effectively "smooths" the probability of feature relevance, reducing the coding cost for features that have proven useful in the past when training on a new "target" task.

Experimental Evaluation

The efficacy of MIC, TPC, and Transfer-TPC was evaluated across several domains:

  • Biological Data: The models were tested on yeast growth rates (6,715 features across 20 conditions) and breast cancer prognosis. In Gene Set Enrichment Analysis (GSEA), the methods utilized chromosomal positions and curated pathways as feature classes.
  • Natural Language Processing: The methods were applied to Word Sense Disambiguation (WSD) for 172 ambiguous verbs. Features included both "syntactic" (grammatical patterns) and "semantic" (topic-based) information.
  • Synthetic Benchmarks: Controlled experiments were conducted to test the models under varying degrees of task overlap (Partial, Full, and Independent sharing).

Key Insights and Results

The "Cheap" Feature Effect

A primary insight of this work is that transfer learning can be viewed as an informational discount. By sharing the "address" or index of a feature across tasks, the model requires fewer bits to justify its inclusion. This effectively lowers the threshold for feature selection in related tasks, allowing the model to capture subtle signals that would be ignored by independent models.

Sparsity and Interpretability

On the Yeast growth dataset, MIC demonstrated a significant advantage in model parsimony. While maintaining or exceeding the accuracy of Bayesian Lasso (BBLasso), MIC selected an average of only 4 features compared to BBLasso’s 63. This high degree of sparsity is critical for biological interpretability, as it highlights a small, manageable set of candidate genes for further study.

Performance in NLP

In WSD tasks, TPC outperformed standard Support Vector Machines (SVMs) and Lasso in 7 out of 10 randomly sampled verbs. Across the entire 172-verb dataset, the improvement was statistically significant at a 5% level. Transfer-TPC showed its greatest utility in "low-resource" scenarios; when a target verb had very few observations, transferring knowledge from related, high-frequency verbs improved accuracy by 4.6% to 9.2%.

Conclusion and Future Work

This research demonstrates that MDL-based feature selection provides a powerful framework for transfer learning in high-dimensional settings. By replacing arbitrary penalties with information-theoretic costs, MIC and TPC produce models that are both accurate and exceptionally sparse.

However, challenges remain. The greedy stepwise search used to solve the $\ell_0$ optimization is computationally intensive ($O(h^2 mm^*)$). Future work may explore more efficient optimization paths, such as homotopy-based algorithms. Additionally, the risk of "negative transfer" requires more robust methods for determining task similarity to ensure that unrelated source tasks do not degrade performance.

Summary updated

Abstract

We present three related ways of using Transfer Learning to improve feature selection. The three methods address different problems, and hence share different kinds of information between tasks or feature classes, but all three are based on the information theoretic Minimum Description Length (MDL) principle and share the same underlying Bayesian interpretation. The first method, MIC, applies when predictive models are to be built simultaneously for multiple tasks (``simultaneous transfer'') that share the same set of features. MIC allows each feature to be added to none, some, or all of the task models and is most beneficial for selecting a small set of predictive features from a large pool of features, as is common in genomic and biological datasets. Our second method, TPC (Three Part Coding), uses a similar methodology for the case when the features can be divided into feature classes. Our third method, Transfer-TPC, addresses the ``sequential transfer'' problem in which the task to which we want to transfer knowledge may not be known in advance and may have different amounts of data than the other tasks. Transfer-TPC is most beneficial when we want to transfer knowledge between tasks which have unequal amounts of labeled data, for example the data for disambiguating the senses of different verbs. We demonstrate the effectiveness of these approaches with experimental results on real world data pertaining to genomics and to Word Sense Disambiguation (WSD).
BibTeX
@misc{dhillon2009transfer,
  author = {Paramveer S. Dhillon and Dean P. Foster and Lyle H. Ungar},
  title = {{Transfer Learning Using Feature Selection}},
  year = {2009},
  eprint = {0905.4022},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/0905.4022},
}
Figure