Abstract
Many ecological studies and conservation policies are based on field observations of species, which can be affected by systematic variability introduced by the observation process. A recently introduced causal modeling technique called 'half-sibling regression' can detect and correct for systematic errors in measurements of multiple independent random variables. However, it will remove intrinsic variability if the variables are dependent, and therefore does not apply to many situations, including modeling of species counts that are controlled by common causes. We present a technique called 'three-quarter sibling regression' to partially overcome this limitation. It can filter the effect of systematic noise when the latent variables have observed common causes. We provide theoretical justification of this approach, demonstrate its effectiveness on synthetic data, and show that it reduces systematic detection variability due to moon brightness in moth surveys.
Introduction
In fields such as ecology, astronomy, and public health, researchers often rely on observational data that is inherently "noisy." For instance, moth surveys using light traps do not capture every individual; instead, counts are influenced by systematic factors like moon brightness, which reduces trap effectiveness. This creates a shared noise component across all observed species. While traditional methods like Half-Sibling Regression (HSR) can denoise independent variables sharing a common noise source, they fail when the variables themselves are correlated due to shared environmental drivers—a common occurrence in biological systems.
Methodology: The Three-Quarter Sibling Approach
The core contribution of this work is Three-Quarter Sibling Regression (3QS). The method is named after a kinship metaphor that describes the causal relationships between variables.
In Figure 1(a), Half-siblings ($Y_1, Y_2$) share one "parent"—the shared noise $N$—but their signals ($Z_1, Z_2$) are independent. In Figure 1(b), Three-quarter siblings share the noise $N$, and their signals ($Z_1, Z_2$) are correlated because they share a common observed cause $X$ (e.g., seasonal environmental factors).
The 3QS algorithm proceeds in four steps:
- Model the Signal: Use observed environmental factors ($X$) to predict the target species count.
- Calculate Residuals: Identify the variation not explained by $X$. This residual contains both intrinsic biological variation and systematic noise.
- Denoise via Siblings: Use the residuals of other species to predict the target species' residual. Since intrinsic variations are independent given $X$, any remaining correlation in residuals is attributed to shared noise ($N$).
- Subtract Noise: Remove the predicted noise from the original observation to yield a denoised estimate.
Experimental Results
The authors validated 3QS using both synthetic simulations and a real-world dataset of moth observations from the Discover Life project.
Synthetic Validation
On synthetic data, 3QS significantly outperformed HSR in reducing Mean Squared Error (MSE) when common causes were present.
As shown in Figure 4, while HSR (dotted lines) struggles to maintain low error when signals are correlated, 3QS (solid lines) consistently achieves lower MSE by leveraging the observed process covariates ($X$) to protect the underlying signal.
Real-World Application: Moth Surveys
Applying 3QS to eight years of moth count data revealed its ability to "discover" and remove the effect of moon brightness without explicit instruction. Key findings include:
- Noise Reduction: 3QS reduced the correlation between species counts and moon brightness by 7%.
- Signal Preservation: Unlike HSR, which reduced data variance to ~80% of the original (indicating over-cleaning), 3QS maintained ~95% of the variance, preserving true biological fluctuations.
- Predictive Power: Models trained on 3QS-denoised data were more accurate at predicting counts on "gold-standard" new-moon nights than those trained on raw or HSR-denoised data.
Conclusion and Future Work
3QS provides a robust framework for denoising observational data where variables share common environmental causes. By using causal independence assumptions rather than complex parametric physics models, it offers a flexible tool for ecological data cleaning. Future work will explore Bayesian frameworks to better handle uncertainty and address scenarios where noise and environment are themselves correlated.
Summary updated