Three-quarter Sibling Regression for Denoising Observational Data

Shiv Shankar, Daniel Sheldon, Tao Sun, John Pickering, Thomas G. Dietterich

Abstract

Many ecological studies and conservation policies are based on field observations of species, which can be affected by systematic variability introduced by the observation process. A recently introduced causal modeling technique called 'half-sibling regression' can detect and correct for systematic errors in measurements of multiple independent random variables. However, it will remove intrinsic variability if the variables are dependent, and therefore does not apply to many situations, including modeling of species counts that are controlled by common causes. We present a technique called 'three-quarter sibling regression' to partially overcome this limitation. It can filter the effect of systematic noise when the latent variables have observed common causes. We provide theoretical justification of this approach, demonstrate its effectiveness on synthetic data, and show that it reduces systematic detection variability due to moon brightness in moth surveys.

Introduction

In fields such as ecology, astronomy, and public health, researchers often rely on observational data that is inherently "noisy." For instance, moth surveys using light traps do not capture every individual; instead, counts are influenced by systematic factors like moon brightness, which reduces trap effectiveness. This creates a shared noise component across all observed species. While traditional methods like Half-Sibling Regression (HSR) can denoise independent variables sharing a common noise source, they fail when the variables themselves are correlated due to shared environmental drivers—a common occurrence in biological systems.

Methodology: The Three-Quarter Sibling Approach

The core contribution of this work is Three-Quarter Sibling Regression (3QS). The method is named after a kinship metaphor that describes the causal relationships between variables.

Causal structures for (a) Half-Sibling Regression and (b) Three-Quarter Sibling Regression.

In Figure 1(a), Half-siblings ($Y_1, Y_2$) share one "parent"—the shared noise $N$—but their signals ($Z_1, Z_2$) are independent. In Figure 1(b), Three-quarter siblings share the noise $N$, and their signals ($Z_1, Z_2$) are correlated because they share a common observed cause $X$ (e.g., seasonal environmental factors).

The 3QS algorithm proceeds in four steps:

  1. Model the Signal: Use observed environmental factors ($X$) to predict the target species count.
  2. Calculate Residuals: Identify the variation not explained by $X$. This residual contains both intrinsic biological variation and systematic noise.
  3. Denoise via Siblings: Use the residuals of other species to predict the target species' residual. Since intrinsic variations are independent given $X$, any remaining correlation in residuals is attributed to shared noise ($N$).
  4. Subtract Noise: Remove the predicted noise from the original observation to yield a denoised estimate.

Experimental Results

The authors validated 3QS using both synthetic simulations and a real-world dataset of moth observations from the Discover Life project.

Synthetic Validation

On synthetic data, 3QS significantly outperformed HSR in reducing Mean Squared Error (MSE) when common causes were present.

MSE performance comparison between 3QS and HS across different noise levels and sibling counts.

As shown in Figure 4, while HSR (dotted lines) struggles to maintain low error when signals are correlated, 3QS (solid lines) consistently achieves lower MSE by leveraging the observed process covariates ($X$) to protect the underlying signal.

Real-World Application: Moth Surveys

Applying 3QS to eight years of moth count data revealed its ability to "discover" and remove the effect of moon brightness without explicit instruction. Key findings include:

  • Noise Reduction: 3QS reduced the correlation between species counts and moon brightness by 7%.
  • Signal Preservation: Unlike HSR, which reduced data variance to ~80% of the original (indicating over-cleaning), 3QS maintained ~95% of the variance, preserving true biological fluctuations.
  • Predictive Power: Models trained on 3QS-denoised data were more accurate at predicting counts on "gold-standard" new-moon nights than those trained on raw or HSR-denoised data.

Conclusion and Future Work

3QS provides a robust framework for denoising observational data where variables share common environmental causes. By using causal independence assumptions rather than complex parametric physics models, it offers a flexible tool for ecological data cleaning. Future work will explore Bayesian frameworks to better handle uncertainty and address scenarios where noise and environment are themselves correlated.

Summary updated

Abstract

Many ecological studies and conservation policies are based on field observations of species, which can be affected by systematic variability introduced by the observation process. A recently introduced causal modeling technique called 'half-sibling regression' can detect and correct for systematic errors in measurements of multiple independent random variables. However, it will remove intrinsic variability if the variables are dependent, and therefore does not apply to many situations, including modeling of species counts that are controlled by common causes. We present a technique called 'three-quarter sibling regression' to partially overcome this limitation. It can filter the effect of systematic noise when the latent variables have observed common causes. We provide theoretical justification of this approach, demonstrate its effectiveness on synthetic data, and show that it reduces systematic detection variability due to moon brightness in moth surveys.
BibTeX
@inproceedings{shankar2019three,
  author = {Shiv Shankar and Daniel Sheldon and Tao Sun and John Pickering and Thomas G. Dietterich},
  title = {{Three-quarter Sibling Regression for Denoising Observational Data}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=eLhp9R_j1W},
}
Figure