Overview
Predicting the next event in a sequence—such as a patient's upcoming medical diagnosis or a sudden shift in the stock market—is a fundamental challenge in machine learning. While Transformer models excel at identifying patterns in sequential data, they often struggle with two major real-world issues:
- Confounding: Past events frequently influence both the current "treatment" (a specific event occurring now) and the "outcome" (what happens next). Standard models often mistake these historical correlations for direct causal relationships.
- Data Scarcity vs. Domain Knowledge: In many domains, data is limited, but experts possess valuable qualitative knowledge (e.g., "Insulin intake decreases blood glucose"). Traditional deep learning models lack a straightforward mechanism to incorporate this qualitative advice.
To address these challenges, researchers have developed PC-TES (Pairwise Causality Guided Transformers for Event Sequences). This novel framework bridges the gap between causal inference and sequence modeling by injecting qualitative domain knowledge directly into Transformer models, effectively handling time-varying confounding and improving prediction accuracy.
Methodology: Injecting Causal Knowledge
PC-TES modifies the training process of an autoregressive Transformer by integrating causal inference techniques. The key innovations of the framework include:
- Causal Formulation: The authors adapt the potential outcome framework for event sequences. They define the Instance-Level Effect (ILE) as the difference in the probability of a future event $Y$ depending on whether event $Z$ just occurred, conditioned on the sequence history.
- Inverse Probability Weighting (IPW): To mitigate confounding from historical events, PC-TES employs IPW. This technique reweights the training data so the model focuses on the true causal impact of an event rather than historical biases.
- Incompatibility Loss: Rather than forcing the model to obey strict, hard-coded rules, the framework introduces a penalty term to the loss function. If the model's predicted probabilities contradict the provided causal knowledge (e.g., predicting that event $Z$ inhibits $Y$ when an expert states it amplifies it), the loss increases.
- Theoretical Foundation: The paper provides a theoretical proof demonstrating that a Transformer's internal representations can act as "balancing scores." This allows the model to produce unbiased estimates of causal effects even within complex, multivariate sequences.
Datasets and Evaluation
The model was evaluated across a diverse mix of synthetic and real-world datasets, spanning healthcare, finance, and social sciences:
- Synthetic Data (Synth 1-4): Sequences generated using Markov models with known ground-truth causal rules.
- Diabetes: Medical records tracking meals, exercise, and insulin intake.
- DeFi (Decentralized Finance): User trading events (e.g., borrows, liquidations, swaps) from the Aave platform.
- LinkedIn: User-level employment history and career events.
- BeigeBooks: Economic topics derived from Federal Reserve reports.
- LLM-Generated Sequences: A unique dataset where a Large Language Model (Flan-T5) generated sequences of societal events based on Wikidata concepts.
Key Insights and Results
A major takeaway from the study is that Transformers are natural causal inference engines. The attention mechanism is mathematically well-suited to serve as a balancing score, meaning the Transformer can calculate propensity scores internally without needing a separate model. Furthermore, the research shows that qualitative knowledge acts as a powerful regularizer. Precise numerical effects are not required; simply knowing the direction of a causal effect (amplifies vs. inhibits) is sufficient to significantly boost performance.
Empirically, PC-TES consistently achieved higher log-likelihood (better prediction accuracy) than vanilla Transformers (TES) and other specialized baseline models.
- Synthetic Performance: PC-TES successfully learned hidden causal rules, outperforming baselines across all four synthetic datasets.
- Real-World Gains: In the DeFi dataset, PC-TES achieved a roughly 5% relative gain over a standard Transformer simply by injecting the rule: "borrowing increases the probability of liquidation."
- Scaling Knowledge: On the BeigeBooks dataset, the model achieved its best results by incorporating eight different economic causal statements, demonstrating that the framework scales effectively to multiple pieces of domain knowledge.
Limitations and Future Directions
While powerful, the framework relies on the assumption of Strong Ignorability, meaning it assumes all important confounders are captured within the event history. If unobserved external factors exist, the causal estimates may be biased. Additionally, the model is sensitive to incorrect knowledge; providing flawed expert advice can lead the model astray.
Future work will explore extending this approach to Temporal Point Processes—where the exact continuous time between events is modeled—and developing methods to automatically discover causal pairs from text data, reducing the reliance on manual expert input.
Summary updated