Researchers from the Gatsby Unit and TU Dortmund have developed a new non-parametric test for "quasi-independence"—a statistical method to check if two variables are associated beyond a known temporal ordering (such as $X \le Y$). By leveraging Reproducing Kernel Hilbert Spaces (RKHS), their test can detect complex, non-linear dependencies in truncated and right-censored data where traditional methods often fail.
The Challenge of Ordered Data
In many real-world scenarios, data points come in pairs that are strictly ordered in time. For example, in a clinical trial, a patient’s recruitment time ($X$) must occur before their survival or exit time ($Y$). Because $X$ is always less than or equal to $Y$, these variables are technically never "independent" in the classical sense.
However, researchers still need to know if there is a hidden relationship between the two. Does the time a patient was recruited influence how long they survive, perhaps due to seasonal effects or changes in hospital protocol? This is the concept of quasi-independence: are $X$ and $Y$ uncoupled except for the fact that $X \le Y$?
Existing tests for this are often limited to simple, monotonic relationships (where variables strictly increase or decrease together). Furthermore, real-world data is often "right-censored"—meaning observation stops before the event of interest happens. This paper introduces a robust, non-parametric framework to handle both truncation and censoring while detecting a much wider variety of dependency patterns.
The Kernel Quasi-Independent Criterion (KQIC)
To address these limitations, the authors propose the Kernel Quasi-Independent Criterion (KQIC). The approach relies on several key innovations:
- RKHS Framework: Instead of looking for a specific type of correlation (like Kendall’s tau), the method maps the data into a high-dimensional feature space (an RKHS). This allows the test to look for any kind of functional dependency, including periodic or "V-shaped" patterns.
- Generalizing the Log-Rank Test: The KQIC acts as a non-parametric generalization of the classical log-rank test. While a standard log-rank test uses a specific weight function, KQIC effectively "optimizes" over an infinite space of weight functions to find the strongest evidence of dependence.
- Handling Censoring: The statistic is explicitly extended to account for right-censored observations (where we only know $Y$ is at least some value $T$).
- Wild Bootstrap: Because the distribution of the test statistic is complex, a "Wild Bootstrap" procedure is used to determine the threshold for statistical significance (the p-value), making the test practical for real-world datasets with unknown underlying distributions.
Datasets and Key Results
The authors evaluated their method on both synthetic and historical real-world datasets, demonstrating superior power and robustness.
Synthetic Data and Periodic Dependencies
Most existing tools assume that if $X$ goes up, $Y$ likely goes up (or down). KQIC’s primary advantage is that by using kernels, it can detect dependencies that "wiggle" or change direction, which are common in seasonal data.
In synthetic tests with periodic dependencies (where the relationship between $X$ and $Y$ oscillates, as seen above), KQIC maintained high detection power. In contrast, traditional methods like the Weighted Log-Rank (WLR) test saw their power drop to near zero because they could not capture the non-monotonic patterns.
Real-World Validation
The researchers also applied KQIC to three historical datasets:
- Channing House: Entry times and lifetimes of residents in a retirement community.
- AIDS Dataset: Incubation and lapse times for patients who contracted AIDS via blood transfusions.
- Abortion Data: A study on the timing of spontaneous abortions, characterized by a very high rate of censoring (over 90%).
The test correctly identified dependencies in the AIDS and Channing House datasets, aligning with—and in some cases providing stronger evidence than—existing specialized literature. Furthermore, KQIC proved remarkably resilient to high censoring. In the Abortion dataset (90.6% censored), KQIC successfully detected a dependency between entry time and event time that other general-purpose tests missed.
Conclusion and Future Work
The paper clarifies that quasi-independence is a deterministic constraint ($X \le Y$), whereas right-censoring is a stochastic property. Treating them correctly is vital for unbiased results. Despite being more powerful and flexible, the kernel-based approach is often more computationally efficient than permutation-based "minimal p-value" selection methods.
The authors note that the choice of kernel is important; they currently use "c0-universal" kernels (like Gaussian or IMQ kernels) to ensure the test is consistent. Future work could involve simplifying the specific conditions for kernels in the quasi-independence setting or exploring how these tests might be integrated into causal discovery frameworks where physical or temporal constraints naturally induce truncation.
Summary updated