Introduction
While deep learning has achieved remarkable success in domains with inherent spatial or sequential structures like computer vision and natural language processing, tabular data remains a challenging frontier. Tabular datasets are often "structureless"—columns can be permuted without loss of information—and typically consist of mixed-variable types (continuous and categorical). In real-world scenarios, obtaining high-quality labels is expensive, necessitating robust semi-supervised learning (SSL) frameworks.
GFTab is a novel framework designed to address these challenges by treating data corruptions as points on a geometric manifold and leveraging tree-based embeddings to capture hierarchical relationships. This approach allows for effective representation learning even when labeled data is scarce or noisy.
Methodology
The GFTab architecture integrates three core innovations to bridge the gap between classical machine learning and deep representation learning.
1. Variable-Specific Corruption
Unlike standard SSL methods that apply uniform noise, GFTab employs specialized corruption strategies for different variable types. Continuous variables are processed via a Variable Selection Network (VSN) and random shuffling. For categorical variables, the model introduces a "neighborhood-based" shift, moving categories to semantically "nearby" values rather than performing simple random replacement. This preserves underlying relational structures during the augmentation process.
2. Geodesic Flow Kernels (GFK)
At the heart of GFTab is the use of Geodesic Flow Kernels. The framework treats "soft" and "hard" corrupted versions of a data point as bases for subspaces on a Grassmann manifold. By computing the geodesic flow—the shortest path between these subspaces—the model learns features that are invariant to noise. This geometric approach effectively treats corrupted data as a "different domain," finding a smooth transition that emphasizes the true signal over augmentation noise.
3. Tree-based Embedding
To capitalize on the strengths of Gradient Boosted Decision Trees (GBDTs), GFTab uses them to generate embeddings from labeled data. These embeddings capture the hierarchical, rule-based relationships that GBDTs excel at identifying, which are then fed into a neural network (typically a Transformer-style architecture) for further refinement.
Experimental Results
The authors evaluated GFTab on a benchmark of 21 diverse tabular datasets from repositories like OpenML and the AutoML Benchmark (AMLB). The datasets were categorized into Panel A (categorical-dominant) and Panel B (continuous-dominant).
In a "low-label" setting (20% labeled data), GFTab demonstrated superior performance:
- Overall Lead: GFTab achieved the highest F1 score in nearly half of the 21 datasets (10 out of 21) and consistently ranked in the top three.
- Categorical Dominance: The model was particularly effective on categorical-heavy datasets, leading in 7 out of 14 cases in Panel A.
- Robustness to Noise: In experiments with 20% label noise, GFTab maintained its competitive edge, outperforming established models like CatBoost, XGBoost, and TabPFN.
Conclusion
GFTab represents a significant step forward for semi-supervised tabular data learning. By combining manifold-based geometric regularization with the hierarchical power of tree-based embeddings, it provides a robust solution for mixed-variable datasets. Future work may extend this framework to regression tasks and explore optimizations for high-dimensional data where manifold operations may incur higher computational costs.
Summary updated