Evaluating causal relationships in time-series data is notoriously difficult. Unlike standard machine learning, where we can just look at prediction accuracy, causal discovery requires knowing the "ground truth" of how variables influence each other over time—something rarely available in real-world datasets.
A new paper from the Causal AI Lab at UMBC introduces DTCD (Datasets and Toolkit for Causal Discovery), a modular framework designed to bring scientific rigor to this field. By moving away from static, "one-size-fits-all" datasets, DTCD allows researchers to stress-test algorithms against specific challenges like non-linearity, missing data, and non-stationarity in a controlled, composable way.
TL;DR
DTCD is a configurable toolkit for generating synthetic time-series data with exact ground-truth causality and counterfactual outcomes. It enables researchers to isolate specific algorithm failure modes—such as sensitivity to missing data or "cheating" via marginal variance shortcuts—that are typically hidden by aggregate benchmark scores.
What Problem Does This Paper Address?
Most researchers in time-series causal discovery rely on synthetic data because real-world ground truth is elusive. However, existing benchmarks suffer from two major flaws:
- Rigidity: They are often static files or hardcoded scripts. If you want to see how an algorithm handles only increased sparsity while keeping the noise distribution identical, you usually have to write new generator code from scratch.
- Evaluation Gaps: Most benchmarks only provide observational data. Evaluating "Individual Treatment Effects" (ITE) requires knowing what would have happened if a variable were different (a counterfactual). Existing tools rarely provide these exact "twins."
Furthermore, many benchmarks are "easy to game." Algorithms can sometimes achieve high scores not by finding causal links, but by exploiting statistical shortcuts like varsortability—where variables are simply sorted by their marginal variance.
Methods
The core of DTCD is its compositional architecture. As shown in Figure 1, the data-generating process (DGP) is broken down into five independent, orthogonal axes:
- Topology: The "skeleton" of the graph (e.g., Erdős-Rényi, Scale-free).
- Mechanism: The functional form of relationships (Linear, Polynomial, MLP, etc.).
- Noise: The distribution of stochastic innovations (Gaussian, Laplace, Cauchy, etc.).
- Mask: How data goes missing (MCAR - random, or MNAR - value-dependent).
- Intervention: Policies for "do-calculus" style changes to the system.
A standout feature is the Counterfactual Suite. By caching the exact noise realizations during a "factual" run and re-injecting them during an "interventional" run, DTCD produces counterfactual twins that are exact to floating-point precision. This allows for the first truly precise measurement of ITE in a temporal benchmark.
Datasets
The authors introduce the DTCD library and a reference benchmark consisting of:
- 9 Graph Templates: Varying in dimensionality ($d \in \{4, 6, 8\}$) and maximum lag ($\tau_{max} \in \{2, 3, 4\}$).
- 81 Discovery Configurations: Testing combinations of linear/nonlinear mechanisms, drift, and missingness.
- 18 CATE Configurations: For treatment-effect estimation.
They compare eight discovery algorithms (like PCMCI, DYNOTEARS, and VAR-LiNGAM) and four treatment-effect estimators (like TARNet and X-Learner).
Key Insights and Ideas
- The "Varsortability" Audit: The authors found that their raw benchmark had a varsortability of 0.93 (where 1.0 is perfectly sortable by variance). By adding a "standardization" toggle, they forced algorithms to actually learn structure. When standardized, the top-performing algorithm's (DYNOTEARS) F1 score dropped from 0.99 to 0.82, proving it was partially relying on the variance shortcut.
- Missing Data is a Hard Wall: Most discovery methods (6 out of 8) simply cannot handle
NaNvalues and fail immediately. Even for those that do, like CUTS+, there is a significant performance gap between random missingness (MCAR) and informative missingness (MNAR). - Aggregate Metrics Hide Nuance: A method might have a low True Positive Rate (TPR) not because it's "bad," but because of a scope mismatch. For example, CD-NOD is designed for contemporaneous edges and ignores lagged ones; DTCD's granular reporting makes this clear rather than just giving it a "failing" grade.
Key Results
The benchmark results (summarized in Figures 2 and 3) highlight clear winners and specific vulnerabilities:
- DYNOTEARS dominates clean, linear conditions with an F1 of ~0.99 but is the most affected by the varsortability shortcut.
- CUTS+ is the clear leader for missing data, outperforming mean-imputation strategies on both MCAR and MNAR variants.
- TARNet proved to be the most efficient CATE estimator, winning on both linear and nonlinear variants while being significantly faster than more complex architectures like SNet.
Limitations and Future Work
While powerful, DTCD is still a synthetic generator. The authors acknowledge a synthetic-to-real gap; high performance on DTCD is a prerequisite for, but not a guarantee of, success on real-world data.
Currently, the reference benchmark is limited to relatively small graphs ($d=8$), though the toolkit supports larger ones. Future work includes adding machine-readable identifiability tags to warn users when a configuration is theoretically impossible to solve, and expanding the CATE benchmark to include more recent architectures like Causal Transformers.
For engineers and researchers, DTCD provides a much-needed "diagnostic lab" to understand exactly where their causal models break. The code is open-source and available at github.com/hferdous/Composable_Causality.
Summary updated