Adaptive Feature Ranking for Unsupervised Transfer Learning

Son N. Tran, Artur d'Avila GarcezICLR 2014 · arXiv · OpenReview

Introduction

Transfer learning is the machine learning equivalent of a human applying their knowledge of riding a bicycle to learn how to ride a motorcycle. Models trained on a "source" dataset attempt to apply that learned knowledge to a "target" dataset. However, two major hurdles exist: label scarcity (most transfer learning requires expensive labeled data in the source domain) and feature selection (not all source features are useful; some might be noise or domain-specific quirks that confuse the model on a new task).

This paper introduces an unsupervised transfer learning approach that ranks features learned by a Restricted Boltzmann Machine (RBM). By identifying and transferring only the most informative features from a source task to a target task, the researchers achieved significant performance boosts on image recognition datasets without requiring any labels in the source domain.

Methodology: Ranking and Transferring Features

The researchers focused on Restricted Boltzmann Machines (RBMs), a type of generative neural network, tackling the "Self-Taught Learning" challenge: enabling a model to autonomously decide which features are high-quality enough to transfer.

1. Feature Ranking

They developed a mathematical ranking function that calculates a score for every hidden unit (feature) in the RBM. This score is based on the average absolute strength of the weights connecting that feature to the input. High-scoring features are those that minimize information loss if the network were simplified.

Feature Ranking Mechanism

2. Selection and Transfer

After training an RBM on a source dataset (like general handwritten digits), the features are ranked, and the top performers are selected. These "elite" features are then transferred to a new RBM designed for the target task.

Transfer Learning Architecture

3. Adaptive Learning

Unlike standard methods where transferred features act as static inputs, the authors propose an Adaptive Feature Learning algorithm. The transferred features actively guide the learning of new features in the target domain, helping the model fill in task-specific gaps.

Key Insights

  • Weight Strength Equals Semantic Meaning: A striking finding is that stronger weights correlate with smarter features. Low-scoring features often encode incorrect or noisy rules, while high-scoring ones consistently capture meaningful data patterns.
  • Visualizing the Noise: When visualizing the filter bases, high-ranking features appear as clear, recognizable strokes (e.g., parts of numbers). In contrast, low-ranking features resemble blurry, gray noise.
  • The Pruning Threshold: The researchers found they could prune away nearly 50% of the low-scoring features with almost zero impact on accuracy. Conversely, removing just 10% of the high-scoring features caused the model's performance to plummet, proving the ranking function successfully identifies the most critical network components.

Experimental Results

The method was evaluated using three major image datasets: MNIST (70,000 handwritten digits, used as the primary source), ICDAR (Optical Character Recognition for digits), and TiCC (a diverse collection of digits and letters from various writers). This setup allowed for testing "cross-domain" transfer, such as learning from digits to recognize letters.

  • Superior Accuracy: The proposed Adaptive Self-Taught Learning (ASTL) method outperformed standard RBMs and traditional Self-Taught Learning in three out of four major test cases.
  • MNIST to TiCC Transfer: When transferring knowledge from MNIST digits to TiCC letters, the ASTL method achieved an accuracy of 77.56%, significantly higher than a standard RBM (75.20%) or basic transfer methods.
  • Guidance Matters: The results confirm that using transferred knowledge to guide the learning of new features is more effective than simply using them as static inputs.

Limitations and Future Work

While highly effective, the current research focuses on RBMs, which are relatively shallow compared to modern deep learning architectures like Deep Belief Networks or multi-layer Convolutional Neural Networks. Extending this ranking logic to deeper models is a natural next step.

Additionally, the current ranking function is "general"—it selects features that are optimal for the source domain. Future work could explore "target-specific" ranking, where the model identifies source features specifically relevant to the unique requirements of the target task. Finally, the method excels when domains are moderately related; if domains are too similar, the adaptive advantage diminishes because the source features already encompass most of the target's needs.

Summary updated

Abstract

Transfer Learning is concerned with the application of knowledge gained from solving a problem to a different but related problem domain. In this paper, we propose a method and efficient algorithm for ranking and selecting representations from a Restricted Boltzmann Machine trained on a source domain to be transferred onto a target domain. Experiments carried out using the MNIST, ICDAR and TiCC image datasets show that the proposed adaptive feature ranking and transfer learning method offers statistically significant improvements on the training of RBMs. Our method is general in that the knowledge chosen by the ranking function does not depend on its relation to any specific target domain, and it works with unsupervised learning and knowledge-based transfer.
BibTeX
@inproceedings{tran2014adaptive,
  author = {Son N. Tran and Artur d'Avila Garcez},
  title = {{Adaptive Feature Ranking for Unsupervised Transfer Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2014},
  url = {https://openreview.net/forum?id=00Rp6XTNJq0GY},
}
Figure