Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification

Rachael DeVries, Casper Christensen, Marie Lisandra Zepeda Mendoza, Ole WintherarXiv 2025 · DOI · arXiv

Introduction

Electronic Health Records (EHRs) represent a critical resource for clinical research, yet they present significant modeling challenges. Unlike uniform time series data, EHRs are characterized by irregular observation intervals, extreme sparsity, and "informative missingness"—where the absence of a measurement itself carries clinical significance. Standard Transformer architectures often fail to capture these nuances, either by flattening the data into dense representations that obscure missingness or by attending to only a single dimension (time or sensor), thereby missing complex cross-dimensional relationships.

Methods: The Bi-Axial Transformer (BAT)

To address these challenges, we introduce the Bi-Axial Transformer (BAT). The core innovation of BAT is the use of Axial Attention, which decomposes the attention mechanism into two parallel tracks to process the 2D (Time x Sensors) structure of EHR data efficiently.

The BAT architecture employs parallel tracks (Track A and B) to attend to sensors and time points sequentially, capturing multi-dimensional relationships.

Key Architectural Components

  • Dual-Track Attention: Track A attends to sensors first, then time points, while Track B reverses this order. This allows the model to capture "intersectional" dependencies, such as how a specific physiological change at one time point relates to a different variable later.
  • Unique Sensor Embeddings: Rather than treating sensors as simple vector positions, BAT learns a distinct embedding for each clinical variable. This design facilitates transfer learning and the merging of datasets with overlapping sensor sets.
  • Continuous Temporal Encoding: To manage irregular time steps, BAT utilizes sinusoidal positional encodings based on actual observation times rather than sequence indices.
  • Leveraging Missingness: By maintaining a 2D structure and utilizing binary indicator masks, BAT treats the absence of data as a valuable feature, allowing the attention mechanism to learn from the patterns of clinical decision-making.

To ensure a fair benchmarking environment, the researchers also re-implemented several baseline models, including GRU-D and IP-Nets, in a modern PyTorch framework.

Datasets and Evaluation

The model was rigorously evaluated across four diverse datasets:

  1. PhysioNet Challenge 2012 (P12): Mortality prediction in ICU records.
  2. MIMIC-III: De-identified ICU database for mortality classification.
  3. PhysioNet Challenge 2019 (P19): Early detection of sepsis.
  4. Human Activity Recognition (HAR): Used as a control to test robustness against induced artificial sparsity (up to 99%).

Results

BAT demonstrates superior performance, particularly in complex clinical tasks like sepsis prediction. On the P19 dataset, BAT achieved state-of-the-art (SOTA) results, significantly outperforming established benchmarks.

Performance comparison across P12, MIMIC-III, and P19 datasets. BAT achieves SOTA results on the P19 sepsis detection task.

Beyond raw performance, BAT exhibited remarkable robustness. In tests on the HAR dataset, it maintained high accuracy even as data was randomly removed, outperforming standard Transformers until 99% of the data was missing. While recurrent models like GRU-D remained slightly more effective for specific mortality tasks on P12 and MIMIC-III, BAT established itself as the leading Transformer-based architecture for these applications.

Discussion and Future Work

A key insight from this work is that sparsity in EHR data is a feature, not a bug. "Inverted ablation" tests revealed that BAT could classify patients significantly better than random guessing using only the binary mask of performed tests, ignoring actual values. This confirms that the pattern of clinical observation is itself a powerful predictor.

Future research will explore more sophisticated sensor embeddings, potentially incorporating clinical knowledge graphs or Large Language Model (LLM) outputs. Additionally, scaling axial attention to handle multi-year longitudinal records remains a priority for extending BAT's utility to chronic disease management.

Summary updated

Abstract

Electronic Health Records (EHRs), the digital representation of a patient's medical history, are a valuable resource for epidemiological and clinical research. They are also becoming increasingly complex, with recent trends indicating larger datasets, longer time series, and multi-modal integrations. Transformers, which have rapidly gained popularity due to their success in natural language processing and other domains, are well-suited to address these challenges due to their ability to model long-range dependencies and process data in parallel. But their application to EHR classification remains limited by data representations, which can reduce performance or fail to capture informative missingness. In this paper, we present the Bi-Axial Transformer (BAT), which attends to both the clinical variable and time point axes of EHR data to learn richer data relationships and address the difficulties of data sparsity. BAT achieves state-of-the-art performance on sepsis prediction and is competitive to top methods for mortality classification. In comparison to other transformers, BAT demonstrates increased robustness to data missingness, and learns unique sensor embeddings which can be used in transfer learning. Baseline models, which were previously located across multiple repositories or utilized deprecated libraries, were re-implemented with PyTorch and made available for reproduction and future benchmarking.
BibTeX
@misc{devries2025bi,
  author = {Rachael DeVries and Casper Christensen and Marie Lisandra Zepeda Mendoza and Ole Winther},
  title = {{Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification}},
  year = {2025},
  eprint = {2508.12418},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.12418},
}
Figure