TabCaps: A Capsule Neural Network for Tabular Data Classification with BoW Routing

Jintai Chen, KuanLun Liao, Yanwen Fang, Danny Chen, Jian WuICLR 2023 · OpenReview

Overview

Tabular data is the backbone of real-world AI applications—from medical records to financial statements—yet it remains a challenging frontier for deep learning. Unlike images or text, tabular features are heterogeneous (varying scales and types) and relation-agnostic (column order lacks inherent meaning). Most current neural networks attempt to force complex "feature-wise interactions," essentially trying to learn how every feature relates to every other feature. This approach is computationally expensive and prone to overfitting, which is why tree-based models like XGBoost still frequently outperform neural networks on tabular tasks.

TabCaps reimagines tabular data classification by moving away from traditional feature-wise interactions. Instead, it treats entire records as vectorial "capsules." By combining learnable multivariate Gaussian kernels with a novel, efficient "Bag-of-Words" (BoW) routing mechanism, TabCaps achieves state-of-the-art performance with a model size over 10x smaller than competing deep learning architectures.

The TabCaps Architecture

TabCaps Architecture and BoW Routing

The authors propose a Capsule Neural Network (CapsNet) specifically tailored for tabular data. The key innovations include:

  • Gaussian Primary Capsules: Instead of simple linear layers, TabCaps uses learnable multivariate Gaussian kernels to encode records. Each primary capsule represents a specific "profile" or data pattern. The output is a vector representing the marginal likelihood that a record fits that profile, capturing data-level semantics directly.
  • Sparse Projection (entmax): To transform primary capsules into "votes" for higher-level capsules, the model uses entmax. Unlike softmax, which assigns at least a tiny weight to everything, entmax can assign exactly zero, effectively filtering out irrelevant features and noise.
  • Bag-of-Words (BoW) Routing: Traditional capsule networks use iterative clustering (like Dynamic or EM routing) to pass information between layers, which is notoriously slow. TabCaps introduces BoW Routing, inspired by the Bag-of-Words model. It uses a learnable "word dictionary" to compute similarities between capsules in a single, straightforward pass, making it significantly faster and more stable.
  • Ensemble Senior Capsules: Rather than one capsule per class, the model uses multiple "senior capsules" per target class, mimicking the benefits of ensemble learning within a single network.

Key Insights and Results

The model was evaluated on 8 diverse real-world tabular datasets (including medical, physical, and general domains) chosen specifically for having few categorical features to isolate the architectural performance.

  • Efficiency is a Feature: By avoiding iterative routing and massive attention matrices, TabCaps achieves its results with only 0.2M parameters, compared to the 9.3M required by models like FT-Transformer. Furthermore, BoW Routing proved to be roughly 25% faster than standard dynamic routing while providing better accuracy.
  • Training Stability: A striking finding is that TabCaps does not require the "early stopping" trick common in tabular deep learning. While models like TabNet or FT-Transformer often see their validation loss collapse after a few epochs, TabCaps remains remarkably stable during training.
  • Performance and Generalization: TabCaps achieved the best or second-best performance on 7 out of 8 datasets, frequently outperforming heavyweights like XGBoost and CatBoost. In "Extreme Generalization" tests—where training and testing data have very different distributions—TabCaps showed superior robustness.
  • Missing Value Imputation: The authors designed a custom decoder for TabCaps that significantly outperformed standard auto-encoders at reconstructing missing data.

Limitations and Future Work

While highly effective, the current iteration of TabCaps focuses entirely on classification tasks; extending it to regression is a key next step. Additionally, while the model handles categorical data, further testing on high-cardinality categorical datasets is necessary to prove its universality. Finally, automating the hyperparameter tuning for the number of "words" in the BoW dictionary and the number of capsules could make the model even more user-friendly.

Summary updated

Abstract

Records in a table are represented by a collection of heterogeneous scalar features. Previous work often made predictions for records in a paradigm that processed each feature as an operating unit, which requires to well cope with the heterogeneity. In this paper, we propose to encapsulate all feature values of a record into vectorial features and process them collectively rather than have to deal with individual ones, which directly captures the representations at the data level and benefits robust performances. Specifically, we adopt the concept of "capsules" to organize features into vectorial features, and devise a novel capsule neural network called "TabCaps" to process the vectorial features for classification. In TabCaps, a record is encoded into several vectorial features by some optimizable multivariate Gaussian kernels in the primary capsule layer, where each vectorial feature represents a specific "profile" of the input record and is transformed into senior capsule layer under the guidance of a new straightforward routing algorithm. The design of routing algorithm is motivated by the Bag-of-Words (BoW) model, which performs capsule feature grouping straightforwardly and efficiently, in lieu of the computationally complex clustering of previous routing algorithms. Comprehensive experiments show that TabCaps achieves competitive and robust performances in tabular data classification tasks.
BibTeX
@inproceedings{chen2023tabcaps,
  author = {Jintai Chen and KuanLun Liao and Yanwen Fang and Danny Chen and Jian Wu},
  title = {{TabCaps: A Capsule Neural Network for Tabular Data Classification with BoW Routing}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=OgbtSLESnI},
}
Figure