Pairwise Causality Guided Transformers for Event Sequences

Xiao Shou, Debarun Bhattacharjya, Tian Gao, Dharmashankar Subramanian, Oktie Hassanzadeh, Kristin BennettNeurIPS 2023 · OpenReview

Overview

Predicting the next event in a sequence—such as a patient's upcoming medical diagnosis or a sudden shift in the stock market—is a fundamental challenge in machine learning. While Transformer models excel at identifying patterns in sequential data, they often struggle with two major real-world issues:

  1. Confounding: Past events frequently influence both the current "treatment" (a specific event occurring now) and the "outcome" (what happens next). Standard models often mistake these historical correlations for direct causal relationships.
  2. Data Scarcity vs. Domain Knowledge: In many domains, data is limited, but experts possess valuable qualitative knowledge (e.g., "Insulin intake decreases blood glucose"). Traditional deep learning models lack a straightforward mechanism to incorporate this qualitative advice.

To address these challenges, researchers have developed PC-TES (Pairwise Causality Guided Transformers for Event Sequences). This novel framework bridges the gap between causal inference and sequence modeling by injecting qualitative domain knowledge directly into Transformer models, effectively handling time-varying confounding and improving prediction accuracy.

Methodology: Injecting Causal Knowledge

PC-TES modifies the training process of an autoregressive Transformer by integrating causal inference techniques. The key innovations of the framework include:

  • Causal Formulation: The authors adapt the potential outcome framework for event sequences. They define the Instance-Level Effect (ILE) as the difference in the probability of a future event $Y$ depending on whether event $Z$ just occurred, conditioned on the sequence history.
  • Inverse Probability Weighting (IPW): To mitigate confounding from historical events, PC-TES employs IPW. This technique reweights the training data so the model focuses on the true causal impact of an event rather than historical biases.
  • Incompatibility Loss: Rather than forcing the model to obey strict, hard-coded rules, the framework introduces a penalty term to the loss function. If the model's predicted probabilities contradict the provided causal knowledge (e.g., predicting that event $Z$ inhibits $Y$ when an expert states it amplifies it), the loss increases.
  • Theoretical Foundation: The paper provides a theoretical proof demonstrating that a Transformer's internal representations can act as "balancing scores." This allows the model to produce unbiased estimates of causal effects even within complex, multivariate sequences.

Datasets and Evaluation

The model was evaluated across a diverse mix of synthetic and real-world datasets, spanning healthcare, finance, and social sciences:

  • Synthetic Data (Synth 1-4): Sequences generated using Markov models with known ground-truth causal rules.
  • Diabetes: Medical records tracking meals, exercise, and insulin intake.
  • DeFi (Decentralized Finance): User trading events (e.g., borrows, liquidations, swaps) from the Aave platform.
  • LinkedIn: User-level employment history and career events.
  • BeigeBooks: Economic topics derived from Federal Reserve reports.
  • LLM-Generated Sequences: A unique dataset where a Large Language Model (Flan-T5) generated sequences of societal events based on Wikidata concepts.
Summary of the real-world and LLM-generated datasets used in the study.

Key Insights and Results

A major takeaway from the study is that Transformers are natural causal inference engines. The attention mechanism is mathematically well-suited to serve as a balancing score, meaning the Transformer can calculate propensity scores internally without needing a separate model. Furthermore, the research shows that qualitative knowledge acts as a powerful regularizer. Precise numerical effects are not required; simply knowing the direction of a causal effect (amplifies vs. inhibits) is sufficient to significantly boost performance.

Empirically, PC-TES consistently achieved higher log-likelihood (better prediction accuracy) than vanilla Transformers (TES) and other specialized baseline models.

  • Synthetic Performance: PC-TES successfully learned hidden causal rules, outperforming baselines across all four synthetic datasets.
  • Real-World Gains: In the DeFi dataset, PC-TES achieved a roughly 5% relative gain over a standard Transformer simply by injecting the rule: "borrowing increases the probability of liquidation."
  • Scaling Knowledge: On the BeigeBooks dataset, the model achieved its best results by incorporating eight different economic causal statements, demonstrating that the framework scales effectively to multiple pieces of domain knowledge.
Performance comparison on real-world datasets, showing PC-TES achieving the best log-likelihood scores.

Limitations and Future Directions

While powerful, the framework relies on the assumption of Strong Ignorability, meaning it assumes all important confounders are captured within the event history. If unobserved external factors exist, the causal estimates may be biased. Additionally, the model is sensitive to incorrect knowledge; providing flawed expert advice can lead the model astray.

Future work will explore extending this approach to Temporal Point Processes—where the exact continuous time between events is modeled—and developing methods to automatically discover causal pairs from text data, reducing the reliance on manual expert input.

Summary updated

Abstract

Although pairwise causal relations have been extensively studied in observational longitudinal analyses across many disciplines, incorporating knowledge of causal pairs into deep learning models for temporal event sequences remains largely unexplored. In this paper, we propose a novel approach for enhancing the performance of transformer-based models in multivariate event sequences by injecting pairwise qualitative causal knowledge such as `event Z amplifies future occurrences of event Y'. We establish a new framework for causal inference in temporal event sequences using a transformer architecture, providing a theoretical justification for our approach, and show how to obtain unbiased estimates of the proposed measure. Experimental results demonstrate that our approach outperforms several state-of-the-art models in terms of prediction accuracy by effectively leveraging knowledge about causal pairs. We also consider a unique application where we extract knowledge around sequences of societal events by generating them from a large language model, and demonstrate how a causal knowledge graph can help with event prediction in such sequences. Overall, our framework offers a practical means of improving the performance of transformer-based models in multivariate event sequences by explicitly exploiting pairwise causal information.
BibTeX
@inproceedings{shou2023pairwise,
  author = {Xiao Shou and Debarun Bhattacharjya and Tian Gao and Dharmashankar Subramanian and Oktie Hassanzadeh and Kristin Bennett},
  title = {{Pairwise Causality Guided Transformers for Event Sequences}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=5q8xovQF7r},
}
Figure