HyperFast: Instant Classification for Tabular Data

David Bonet, Daniel Mas Montserrat, Xavier Giró-i-Nieto, Alexander Ioannidis

TL;DR

This paper introduces HyperFast, a novel meta-trained hypernetwork that can instantly generate a fully trained neural network for tabular data classification in a single forward pass. This eliminates the need for traditional, time-consuming model training and hyperparameter tuning, offering competitive performance with significantly reduced computational cost.

What Problem Does This Paper Address?

The paper tackles the significant computational burden and time investment associated with training deep learning models for tabular data. While traditional methods like gradient boosting are often preferred due to their efficiency, they can be outperformed by neural networks. However, neural networks for tabular data typically require extensive, task-specific hyperparameter tuning, making them impractical for rapid deployment or scenarios with limited computational resources. Existing "instant" classification models like TabPFN are limited to very small datasets. HyperFast aims to bridge this gap by providing a neural network solution that is both fast and adaptable to a wide range of tabular datasets without requiring per-task training.

Methods

HyperFast employs a meta-learning approach using a hypernetwork. The core idea is to train a "meta-model" (the hypernetwork) that learns to predict the weights of a smaller, task-specific "main network." This process is done offline during a meta-training phase on a diverse set of tabular datasets.

Here's a breakdown of the key components:

  • Hypernetwork: This is the meta-model that takes a small "support set" (a few labeled examples) from an unseen dataset as input. It then outputs the weights for a main neural network.
  • Main Network: This is a smaller neural network whose weights are generated by the hypernetwork. Once its weights are predicted, it can perform classification inference on new data points from the same task in a single forward pass.
  • Initial Transformation Layers: To handle varying dataset dimensionalities and ensure permutation invariance, HyperFast uses Random Features (RF) and Principal Component Analysis (PCA) to project input data into a fixed-size representation.
  • Layer-by-Layer Weight Generation: The hypernetwork generates weights for the main network in a modular, layer-by-layer fashion.
  • Meta-Training: The hypernetwork is trained end-to-end by minimizing the classification loss of the generated main networks on a diverse set of meta-training datasets. This allows it to learn generalizable patterns for adapting to new tasks.
  • Instant Inference: During meta-testing, the trained hypernetwork takes a support set and directly outputs the weights for the main network, which is then used for classification without any further training.

Datasets

The authors evaluated HyperFast on a variety of tabular datasets, including:

  • OpenML Curated Classification Benchmarking Suite (OpenML-CC18): A large collection of 70 diverse tabular datasets.
  • Genomic Datasets:
    • HapMap3: Single-nucleotide polymorphism (SNP) data for human subpopulations.
  • Dogs: Dog DNA sequences for breed clades.
  • UK Biobank: SNP data for phenotype prediction (e.g., hair color, skin tanning).
  • High-Dimensional Biomedical Datasets:
    • hapmap-100k: A large-scale HapMap3 dataset.
  • diabetes-31k: A UK Biobank diabetes prediction dataset.

The datasets were split into meta-training, meta-validation, and meta-testing sets to ensure robust evaluation of the model's generalization capabilities. A "mini-test" subset was also created for comparison with models like TabPFN that are limited to smaller datasets.

Key Insights and Ideas

  • Meta-learning for Instant Tabular Classification: The core innovation is using a hypernetwork to "learn to learn" how to generate classification models, enabling instant inference without per-task training.
  • Decoupling Model Generation from Inference: HyperFast separates the computationally intensive meta-training of the hypernetwork from the fast inference performed by the generated main network.
  • Importance of RF and PCA: The initial Random Features and PCA transformation layers are crucial for handling diverse dataset dimensionalities and creating a consistent input representation for the hypernetwork, significantly improving performance.
  • Adaptability to Various Dataset Sizes and Structures: HyperFast demonstrates robustness across datasets with different numbers of samples, features, and categories, overcoming limitations of previous instant models.
  • Ensembling and Fine-tuning for Enhanced Performance: While a single forward pass is fast, ensembling multiple generated models or fine-tuning them on the full dataset can further boost performance with manageable increases in runtime.

Key Results

  • Speed: HyperFast achieves classification in a single forward pass, making it orders of magnitude faster than traditional training methods. For example, it can perform inference in seconds, while AutoML systems might take hours.
  • Performance: HyperFast achieves competitive balanced accuracy, often on par with or exceeding state-of-the-art gradient boosting methods and AutoML systems, especially within shorter runtime budgets.
  • Adaptability: It shows strong performance across a wide range of tabular datasets, including those with high dimensionality and imbalanced classes, without requiring hyperparameter tuning for each new task.
  • Comparison to TabPFN: HyperFast significantly outperforms TabPFN on larger datasets and is competitive on smaller ones, while also handling more classes.
  • High-Dimensional Data: HyperFast demonstrates robustness and achieves the best performance on high-dimensional biomedical datasets, outperforming linear models and gradient-boosted trees.

Limitations and Future Work

  • Number of Samples: For very large datasets, a single support set might not fully represent the entire data distribution, potentially limiting the optimal results of the generated network. However, ensembling and fine-tuning can mitigate this.
  • Number of Features: While RF and PCA help manage feature dimensionality, very high-dimensional datasets might still pose challenges. Feature bagging and training larger HyperFast versions are suggested solutions.
  • Number of Categories: The current HyperFast version supports up to 100 classes. Accommodating more categories would linearly increase the complexity of initial hypernetwork layers, which is computationally manageable.
  • Future Work: The authors suggest extending HyperFast to regression tasks, developing multi-GPU meta-training for very large datasets and modalities (like images), and investigating how dataset distribution differences affect generalization. They also plan to explore more complex architectures and multi-hypernetwork settings.

Summary updated

Abstract

Training deep learning models and performing hyperparameter tuning can be computationally demanding and time-consuming. Meanwhile, traditional machine learning methods like gradient-boosting algorithms remain the preferred choice for most tabular data applications, while neural network alternatives require extensive hyperparameter tuning or work only in toy datasets under limited settings. In this paper, we introduce HyperFast, a meta-trained hypernetwork designed for instant classification of tabular data in a single forward pass. HyperFast generates a task-specific neural network tailored to an unseen dataset that can be directly used for classification inference, removing the need for training a model. We report extensive experiments with OpenML and genomic data, comparing HyperFast to competing tabular data neural networks, traditional ML methods, AutoML systems, and boosting machines. HyperFast shows highly competitive results, while being significantly faster. Additionally, our approach demonstrates robust adaptability across a variety of classification tasks with little to no fine-tuning, positioning HyperFast as a strong solution for numerous applications and rapid model deployment. HyperFast introduces a promising paradigm for fast classification, with the potential to substantially decrease the computational burden of deep learning. Our code, which offers a scikit-learn-like interface, along with the trained HyperFast model, can be found at https://github.com/AI-sandbox/HyperFast.
BibTeX
@inproceedings{bonet2024hyperfast,
  author = {David Bonet and Daniel Mas Montserrat and Xavier Giró-i-Nieto and Alexander Ioannidis},
  title = {{HyperFast: Instant Classification for Tabular Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i10.28988},
}
Figure