SuperTML: Two-Dimensional Word Embedding for the Precognition on Structured Tabular Data

Baohua Sun, Lin Yang, Wenhan Zhang, Michael Lin, Patrick Dong, Charles Young, Jason DongWorkshops @ CVPR 2019 · DOI

Abstract

SuperTML introduces a novel methodology for processing structured tabular data by transforming it into two-dimensional (2D) images, which are then analyzed using high-performance Convolutional Neural Networks (CNNs). By mapping feature values into 2D word embeddings, the approach leverages pre-trained computer vision models via transfer learning. Experimental results on benchmarks including the Higgs Boson Kaggle challenge and the Adult dataset demonstrate that SuperTML achieves state-of-the-art performance, often surpassing traditional gradient boosting methods like XGBoost.

Introduction

In the current machine learning landscape, a sharp divide exists between unstructured data (images, audio, text) and structured tabular data. While Deep Learning—specifically CNNs and Transformers—has revolutionized unstructured data processing, Gradient Boosting Trees (like XGBoost) and Random Forests still dominate tabular data tasks.

Previous attempts to apply deep learning to tabular data typically relied on one-dimensional (1D) embeddings. However, these approaches often fail to fully exploit the sophisticated feature-extraction capabilities of modern CNN architectures. Furthermore, tabular data often requires tedious pre-processing, such as one-hot encoding for categorical variables or complex imputation for missing values. SuperTML aims to bridge this gap by treating tabular classification as an image recognition problem, allowing models to "see" the relationships between features.

Methodology

The core of the research is the SuperTML (Tabular Machine Learning) method, which adapts the "Super Characters" method used in text classification to the tabular domain. The process follows two main stages:

1. 2D Embedding (Image Generation)

For every row in a dataset, the features are projected onto a 2D plane, essentially creating a "picture" of the data.

  • Categorical and Missing Values: Instead of encoding them numerically, the actual text (e.g., "blue" or "missing") is written onto the image.
  • Numerical Values: The digits themselves are drawn into the image.
  • SuperTML_VF (Variable Font): A variant where the font size of a feature is determined by its importance (calculated via traditional ML methods). More important features appear larger in the image.
  • SuperTML_EF (Equal Font): A simpler version where all features are given equal space and font size.
The SuperTML pipeline: converting tabular rows into 2D images for CNN processing.

2. Fine-Tuning

These generated images are fed into state-of-the-art CNNs (such as ResNet, SE-Net, or PolyNet) that have been pre-trained on ImageNet. Through transfer learning, the models learn to recognize patterns in the shapes and locations of the data values to make classifications.

Experimental Setup

The authors evaluated SuperTML on four diverse benchmarks to test scalability and robustness:

  • Iris and Wine Datasets: Small, classic datasets used to test if the method overfits on limited data.
  • Adult Dataset: A larger dataset (48,842 instances) involving both integer and categorical data to predict income levels.
  • Higgs Boson Machine Learning Challenge: A massive Kaggle dataset (250,000 training instances) used to classify quantum events.
Example of a 2D embedded image from the Adult dataset, showing how text and numerical features are spatially arranged.

Key Insights and Results

CNNs are exceptionally proficient at identifying spatial relationships. By placing tabular features in consistent locations on a 2D grid, the CNN can learn complex interactions between features similarly to how it learns the relationship between facial features in an image. This method allows tabular data tasks to benefit from the massive research investment in computer vision.

SuperTML demonstrated impressive performance across the board:

  • Higgs Boson Challenge: SuperTML_EF achieved an AMS score of 3.979, surpassing the previous state-of-the-art result of 3.806.
  • Adult Dataset: It achieved 87.64% accuracy, outperforming both XGBoost (87.32%) and traditional Gradient Boosting (86.20%).
  • Small Datasets: Despite the "data-hungry" reputation of CNNs, SuperTML matched or slightly beat XGBoost on the Iris and Wine datasets, proving that transfer learning helps mitigate the risks of small sample sizes.

Conclusion and Future Work

SuperTML provides a powerful alternative for tabular data by utilizing computer vision advancements. It eliminates the need for manual pre-processing like one-hot encoding, as the model handles raw categorical strings and missing values naturally.

However, the method faces challenges regarding computational overhead; converting every row into a high-resolution image for a massive CNN is more resource-intensive than running a decision tree. Additionally, CNNs remain "black boxes," making interpretability more difficult than with tree-based models. Future research will explore low-power CNN accelerators for edge deployment and refine feature importance calculations to further optimize the 2D layout of the data.

Summary updated

BibTeX
@inproceedings{sun2019supertml,
  author = {Baohua Sun and Lin Yang and Wenhan Zhang and Michael Lin and Patrick Dong and Charles Young and Jason Dong},
  title = {{SuperTML: Two-Dimensional Word Embedding for the Precognition on Structured Tabular Data}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00360},
}
Figure