A Package for Learning on Tabular and Text Data with Transformers

Ken Gu, Akshay BudhkarNAACL 2021

TL;DR

This paper introduces Multimodal-Toolkit, a Python library that seamlessly integrates tabular data (numerical and categorical features) with Transformer models, enabling them to leverage both text and structured data for downstream tasks. This toolkit addresses the growing need to utilize rich, real-world datasets that often contain a mix of modalities, pushing the boundaries of what Transformer models can achieve.

What Problem Does This Paper Address?

Transformers have revolutionized natural language processing (NLP), achieving state-of-the-art results on a wide range of text-based tasks. However, many real-world datasets are not purely textual; they often include tabular data, such as customer demographics, product attributes, or sensor readings, which contain valuable signals. Existing Transformer architectures and libraries, while powerful for text, often struggle to directly incorporate and effectively utilize this tabular information. This creates a gap, preventing the full potential of Transformers from being realized on multimodal datasets. The authors aim to bridge this gap by providing a user-friendly tool that allows Transformers to learn from both text and tabular data simultaneously.

Methods

The core contribution of this paper is the Multimodal-Toolkit, a Python package designed to work seamlessly with the Hugging Face Transformers library. The toolkit's approach is to treat tabular data as an additional modality alongside text.

Key aspects of their method include:

  • Data Preprocessing: The toolkit includes modules for preprocessing tabular data. Numerical features are used directly, while categorical features are encoded. These processed tabular features, along with tokenized text, are then fed into a Transformer model.
  • Transformer Integration: The toolkit leverages the existing infrastructure of Hugging Face Transformers, including tokenization, model hub access, and training pipelines. This allows users to easily incorporate pre-trained Transformer models (like BERT, RoBERTa, XLNET) and fine-tune them on multimodal data.
  • Combining Module: A novel "combining module" is introduced. This module takes the output from the Transformer (representing text features) and the preprocessed tabular features (categorical and numerical) and fuses them into a single multimodal representation. Importantly, this combining module is designed to be model-agnostic and is applied after the Transformer has processed the text, allowing for easy integration without modifying the core Transformer architecture.
  • Feature Combination Methods: The paper explores various strategies for combining the text and tabular features within the combining module, including simple concatenation, MLPs applied to different modalities, attention mechanisms, and gating mechanisms.

Datasets

The authors evaluated their Multimodal-Toolkit on three real-world datasets:

  • Melbourne Airbnb Open Data (Airbnb): Used for a regression task (predicting listing price). Text features included listing name, summary, and host description.
  • Women's E-Commerce Clothing Reviews (Clothing): Used for a binary classification task (predicting if a review recommends a product). Text features included the review itself, and tabular features included rating, clothing category, etc.
  • PetFinder.my Adoption Prediction (PetFinder): Used for a multiclass classification task (predicting pet adoption speed). Text features included the listing description and pet name, while tabular features included pet attributes.

These datasets represent a mix of regression and classification problems, showcasing the toolkit's versatility.

Key Insights and Ideas

  • Tabular Data is Crucial: The most significant insight is that incorporating tabular data can substantially improve performance on tasks where text alone is insufficient. This highlights the importance of multimodal learning in practical applications.
  • Flexibility is Key: The toolkit's design, particularly the model-agnostic combining module applied post-Transformer, is a clever way to extend the power of existing Transformer libraries without requiring users to delve into complex architectural changes.
  • Dataset Dependency: The degree of performance improvement from tabular data varies significantly by dataset. For tasks where text is already a strong signal (like sentiment in reviews), the gains might be marginal. However, for tasks requiring a broader understanding of attributes (like pet adoption speed), tabular data becomes indispensable.
  • Scaling Challenges: The paper hints that simpler multimodal approaches (like treating all features as tokens) might not scale well when the number of tabular features or their complexity increases. This underscores the need for more sophisticated combining strategies.

Key Results

The paper demonstrates the effectiveness of Multimodal-Toolkit across the evaluated datasets:

  • Text-Only Baseline Underperforms: In all experiments, the text-only baseline was the worst performing model, clearly indicating that leveraging tabular data is beneficial.
  • Significant Improvements: For the Airbnb and PetFinder datasets, incorporating tabular features led to notable improvements in performance metrics (RMSE, MAE for regression; F1, AUPRC for classification). For instance, the "Gating" method on the PetFinder dataset achieved an F1macro score of 0.275, a substantial improvement over the text-only baseline's 0.088.
  • Best Performing Methods: While results varied, methods like "Gating" and "Weighted Sum" often showed strong performance, suggesting that more complex ways of combining modalities can be highly effective.
  • Clothing Dataset Nuance: For the Clothing Reviews dataset, the text-only baseline performed very well, suggesting the review text itself contained sufficient sentiment information. However, other methods still showed competitive results.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the performance gains from tabular data are dataset-dependent.
  • The "Unimodal" baseline (tokenizing all features) is noted as not scaling well with an increasing number of tabular features or when their text representation is not semantically rich.
  • Future Work:
    • The authors plan to include support for a wider range of Transformer models.
  • They aim to integrate the combining module at earlier layers within the Transformer architecture, potentially allowing for more intricate cross-modal interactions.
  • They encourage open-source contributions to further develop the toolkit.

Summary updated

BibTeX
@inproceedings{gu2021package,
  author = {Ken Gu and Akshay Budhkar},
  title = {{A Package for Learning on Tabular and Text Data with Transformers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure