Attention Augmented Convolutional Transformer for Tabular Time-series

Sharath M. Shankaranarayana, Davor Runje

Abstract

Tabular time-series data, such as financial transactions and sensor logs, remains a cornerstone of industrial data science. While Transformers have revolutionized fields like NLP and Computer Vision, their application to tabular data has been hindered by data heterogeneity and complex temporal dependencies. This paper introduces TabAConvBERT, a novel framework that combines the local pattern-matching capabilities of 1D convolutions with the global contextual awareness of Transformers. By implementing a specialized timestamp embedding block and a unique "row-masking" pretraining strategy, TabAConvBERT achieves state-of-the-art performance in fraud detection tasks without the need for manual feature engineering or data quantization.

Introduction

In the current landscape of machine learning, deep learning has largely replaced manual feature engineering in unstructured data domains. However, tabular time-series data—sequences of events where each entry contains a mix of categorical and continuous variables—often remains reliant on Gradient Boosted Trees and extensive human intervention. The challenge lies in the inherent messiness of dynamic tables: high dimensionality, mixed data types, and multi-scale temporal patterns.

TabAConvBERT aims to bridge this gap by bringing self-supervised learning and Transformer architectures to the tabular domain. It moves beyond simple adaptations of BERT by introducing architectural components specifically designed for the nuances of event-based sequences.

The Challenge of Dynamic Tabular Data

Most industrial data is "dynamic tabular data," characterized by three primary difficulties:

  1. Feature Engineering Bottlenecks: Traditionally, domain experts must spend weeks crafting features to capture temporal trends.
  2. Data Heterogeneity: Integrating floating-point numbers (e.g., transaction amounts) with categorical identifiers (e.g., merchant IDs) is difficult for standard neural networks.
  3. Temporal Complexity: Models must simultaneously capture periodic cycles (daily or weekly habits) and long-term behavioral shifts.

Existing solutions like TabNet or TabBERT often require data binning (quantization), which can lead to information loss, or they struggle to scale efficiently across long sequences.

Methodology: The TabAConvBERT Architecture

The core of the proposed framework is a hybrid architecture that leverages both convolutional and attention mechanisms. Unlike sequential models that might apply a CNN followed by a Transformer, TabAConvBERT utilizes Attention Augmented Convolutions (AAConv) in parallel.

Figure 3: The overall architecture of TabAConvBERT, showing the embedding network, the stacked AAConv layers, and the final dense output layers.

Key Architectural Innovations

  • Attention Augmented Convolutions (AAConv): 1D convolutions are highly effective at identifying local, translation-invariant patterns. By running them in parallel with Multi-Head Attention (MHA), which captures long-range dependencies, the model gains a comprehensive view of the sequence. The outputs of both are concatenated to form a rich representation.
  • Timestamp Embedding Block: Rather than treating a timestamp as a single scalar, the model decomposes it into discrete components (Year, Month, Day, Weekday, etc.) and normalized continuous values. These components are processed through embedding layers and shallow neural networks, then summed to create a high-dimensional temporal context.
  • Handling Continuous Features: To avoid the pitfalls of quantization, TabAConvBERT processes raw continuous features through shallow neural networks. This preserves the precision of numerical data, which is often critical in financial applications.

Pretraining with Masked Data Modeling

Borrowing from the success of BERT in NLP, the authors utilize Masked Data Modeling (MDM) for self-supervised pretraining. However, they introduce a critical modification for tabular data: Row Masking.

In standard tabular models, masking a single feature (like "Amount") might be too simple because other features in the same row (like "Merchant Category") are highly correlated. By masking an entire row, the model is forced to reconstruct the whole event based solely on the temporal context of preceding and succeeding events. This forces the model to learn the underlying "grammar" of the time-series.

Experimental Results

The model was evaluated on a massive transaction dataset comprising 24 million transactions from 20,000 users. Each transaction included 12 fields, ranging from merchant details to transaction amounts. The downstream task was binary classification for fraud detection.

Method Fraud F1 Score
Baseline MLP 0.74
Baseline LSTM 0.73
TabBERT LSTM 0.86
TabAConvBERT (Proposed) 0.896

TabAConvBERT significantly outperformed previous state-of-the-art models. Notably, the architectural improvements (AAConv and row-masking) provided a performance boost even when using the same data inputs as baseline models, proving the robustness of the hybrid approach.

Conclusion and Future Work

TabAConvBERT demonstrates that by treating tabular rows as contextual tokens and blending CNNs with Transformers, we can achieve high-performance automated machine learning for complex time-series. While the current model shows exceptional results in fraud detection, future research will focus on improving the scalability of the attention mechanism for even longer sequences and testing the framework's generalizability across other domains such as healthcare and telecommunications.

Summary updated

BibTeX
@inproceedings{shankaranarayana2021attention,
  author = {Sharath M. Shankaranarayana and Davor Runje},
  title = {{Attention Augmented Convolutional Transformer for Tabular Time-series}},
  booktitle = {Workshops @ ICDM},
  year = {2021},
  url = {https://openreview.net/forum?id=hkXxvHYZ3k},
}
Figure