Revisiting Multimodal Transformers for Tabular Data with Text Fields

Thomas BonnierACL 2024 · DOI

Abstract

Multimodal datasets combining structured tabular data with free-form text are prevalent in real-world applications such as healthcare and finance. Traditional approaches often struggle with complex numerical distributions and lack reliable uncertainty quantification. This paper introduces the Tabular-Text Transformer (TTT), a novel dual-stream architecture designed to address these gaps. By employing a "distance-to-quantile" numerical encoding and an "overall attention" mechanism, TTT not only outperforms standard baselines across diverse datasets but also leverages internal stream disagreement to provide interpretable uncertainty signals.

Introduction

In many high-stakes domains, data is inherently multimodal, consisting of both structured tables (e.g., patient vitals, financial metrics) and unstructured text (e.g., clinical notes, news reports). Standard modeling approaches typically fall into two traps: suboptimal encoding of numerical features, which often fails to handle outliers or complex distributions, and a lack of transparency regarding prediction uncertainty. When a model makes a high-stakes prediction, it is crucial to know if it is "guessing" and which specific features are driving that uncertainty. The Tabular-Text Transformer (TTT) tackles these challenges by rethinking how Transformers attend to different data types and using the disagreement between data streams to flag unreliable predictions.

Methods

The TTT architecture introduces three primary innovations to improve multimodal learning:

  • Distance-to-Quantile Embedding: Instead of standard linear mapping, numerical features are embedded based on their distance to a set of learned quantiles. This allows the model to better represent the order of values and handle extreme outliers more gracefully.
  • Overall Attention: Traditional multimodal Transformers often compute self-attention and cross-attention separately. TTT’s "Overall Attention" module considers the entire context—both tabular and text elements—simultaneously at every layer, building richer cross-modal representations.
  • Uncertainty via Stream Disagreement: TTT utilizes two parallel streams (text and tabular). If these streams predict different classes, the model flags the prediction as uncertain. To explain this hesitation, the paper adapts Shapley values to a bimodal context, identifying which specific features (e.g., a rare word vs. an unusual numerical value) are driving the conflict.
The TTT architecture featuring dual streams for text and tabular data, integrated via Overall Attention and distance-to-quantile embeddings.

Results and Key Insights

TTT was evaluated on eight diverse classification datasets, including Airbnb, Wine, and Jigsaw. It consistently outperformed major baselines such as EarlyConcat, LateFuse, and MulT. For example, on the Airbnb dataset, TTT achieved an accuracy of 0.383, compared to the next best baseline at 0.363.

Beyond raw performance, the model's ability to quantify uncertainty proved highly effective. The "stream disagreement" method was found to be a reliable proxy for uncertainty; when streams disagreed, model accuracy dropped significantly. The explainability component was demonstrated on the Wine dataset, where the model correctly identified that a prediction was uncertain because the feature "Country: Brazil" was rare and the word "abrasive" in the text conflicted with typical profiles for that variety.

Uncertainty quantification and explainability results, showing JSD distributions and feature contributions via Shapley values.

Furthermore, a lightweight version of the model, TTT-SRP (using Sparse Random Projections), achieved results comparable to much larger pretrained models like AllTextBERT, despite having significantly fewer parameters (8.2M vs. 66.4M). This suggests that architectural efficiency can be as impactful as raw scale.

Conclusion and Future Work

The Tabular-Text Transformer offers a robust framework for processing multimodal tabular data with integrated uncertainty quantification. While the results are promising across standard benchmarks, future work will focus on validating the model on critical real-world datasets in medicine and finance. Additionally, the authors plan to explore specialized pretraining strategies and handle cases with highly unbalanced sequence lengths between modalities.

Summary updated

BibTeX
@inproceedings{bonnier2024revisiting,
  author = {Thomas Bonnier},
  title = {{Revisiting Multimodal Transformers for Tabular Data with Text Fields}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.87},
}
Figure