Incorporating Prior Knowledge on Features into Learning

Eyal Krupka, Naftali TishbyAISTATS 2007

Authors: Eyal Krupka and Naftali Tishby

Abstract

Traditional supervised learning often treats features as anonymous numerical values, ignoring the inherent relationships and geometries that define them. This paper introduces a framework for incorporating prior knowledge about features through "meta-features"—auxiliary information such as spatial coordinates or categorical properties. By modeling feature weights as a smooth function of these meta-features via a Gaussian process, the authors provide a principled method for regularization. Experimental results on the MNIST dataset demonstrate that this approach significantly improves classification accuracy while maintaining high computational efficiency compared to standard kernel-based methods.

Introduction

In standard supervised learning paradigms, features are typically represented as an anonymous vector of numbers. An algorithm processing a digital image, for instance, does not inherently "know" that pixel #45 is adjacent to pixel #46, nor does a text classifier naturally understand the linguistic relationship between two specific words. While researchers often attempt to inject "prior knowledge" to bridge this gap—most commonly through data augmentation techniques like rotating or scaling training images—these methods are frequently task-specific and computationally expensive.

Krupka and Tishby identify a fundamental gap in machine learning: the lack of a general, principled mechanism to inform a model that specific features are related and should be treated similarly. They propose a solution that leverages the underlying geometry of the feature space itself.

The Meta-Feature Framework

The core innovation of this work is the introduction of meta-features. A meta-feature is a vector of properties assigned to every primary feature in a dataset. For image data, meta-features might be the $(x, y)$ spatial coordinates of pixels. In a recommendation engine, the meta-features for a "movie" feature could include its genre, director, or release year.

Conceptual framework for incorporating meta-features into the learning process.

The authors propose a three-step methodology to integrate this knowledge:

  1. Define Meta-Features: Assign a descriptive vector to every feature.
  2. Gaussian Process Prior: Instead of assuming feature weights in a linear classifier are independent, they assume the weights are a smooth function of the meta-features. A covariance matrix $C$ is defined such that the similarity between two weights depends on the distance between their corresponding meta-features.
  3. The Transformation Trick: The authors demonstrate that this complex prior can be implemented by transforming the input data. By multiplying the input vector by the square root of the covariance matrix ($C^{1/2}$), the problem can be solved using a standard, efficient linear Support Vector Machine (SVM).

Experimental Evaluation

The framework was primarily evaluated using the MNIST dataset of handwritten digits. The researchers used a subset of 5,000 training instances and expanded the feature set to include "monomials" (products of pixel intensities). The meta-features for these monomials included their position, orientation, and geometric shape descriptors.

Visualization of the learned weight vector and its classification boundary for MNIST digits.

The results showed a clear advantage for the meta-feature approach:

  • Standard 4th-degree Polynomial SVM: 4.2% error
  • Linear SVM on a subset of monomials: 2.9% error
  • Proposed Meta-Feature Framework: 1.8% error

Key Insights and Efficiency

By forcing the mapping from meta-features to weights to be smooth, the model effectively regularizes against noise in individual features. This ensures the model is more "physical" or "logical," mirroring biological systems like the human cortex, where neurons processing similar features are often physically clustered.

Crucially, this method offers significant advantages over the traditional "kernel trick." Standard non-linear SVMs require storing a large number of support vectors, leading to high memory and computational costs during inference. Because the proposed method results in a single, consolidated weight vector $w$, it is significantly faster and requires approximately 80% less memory during deployment than its kernel-based counterparts.

Conclusion and Future Work

While the framework is powerful, its success depends on the engineering of relevant meta-features. Furthermore, calculating and inverting the covariance matrix $C$ for massive feature sets remains a computational challenge, though the authors suggest Fourier analysis as a potential optimization for stationary kernels. Future research directions include applying this framework to gene-expression data, where biological pathways could serve as meta-features, and further exploring the parallels between this model and biological neural architectures.

Summary updated

BibTeX
@article{krupka2007incorporating,
  author = {Eyal Krupka and Naftali Tishby},
  title = {{Incorporating Prior Knowledge on Features into Learning}},
  journal = {AISTATS},
  year = {2007},
}
Figure