Imagine you are trying to determine the truth about an event by consulting ten different online news outlets. If nine outlets report option A and only one reports option B, a simple majority vote suggests that option A is highly likely to be correct. But what if those nine outlets all scraped their text from the exact same unverified blog post? In reality, you do not have nine independent pieces of confirming evidence; you have a single, potentially flawed source whose error has been amplified through an echo chamber.
This scenario illustrates the core challenge of source dependency. Across machine learning, data integration, and sensor fusion, systems frequently aggregate information from multiple "weak" or noisy sources—such as heuristic rules, crowdsourced workers, web scrapers, or physical sensors. To make the mathematics of aggregation tractable, classic frameworks traditionally rely on the conditional independence assumption: they assume that, conditioned on the true underlying state, the errors made by different sources are statistically independent.
When this assumption is violated, aggregation mechanisms break down. Instead of resolving uncertainty, they double-count redundant data, leading to overconfident and systematically biased decisions.
The Mathematical Trap of Naive Aggregation
To understand why the conditional independence assumption is so pervasive—and so dangerous—we must look at how evidence fusion works mathematically.
In early perceptual reasoning and expert systems, such as the framework proposed by Thomas D. Garvey et al. in An Inference Technique for Integrating Knowledge from Disparate Sources, pooling information from multiple sensors required a way to update beliefs systematically. If two sources provide observations $E_1$ and $E_2$ about a hidden state $Y$, calculating the joint probability $P(E_1, E_2 | Y)$ is computationally expensive and requires knowing the full joint distribution. By assuming conditional independence, the math simplifies beautifully:
$$P(E_1, E_2 | Y) = P(E_1 | Y) \cdot P(E_2 | Y)$$
This simplification allows systems to update probabilities by simply multiplying the independent updates from each source. Early evidential reasoning systems, including those built on Dempster-Shafer theory (discussed by the authors of A Framework for Evidential-Reasoning Systems), relied heavily on this assumption to maintain tractability.
However, as Robert A. Hummel and Larry M. Manevitz demonstrated in Combining Bodies of Dependent Information, if $E_1$ and $E_2$ stem from a shared underlying cause or bias, multiplying their probabilities acts as an exponentiation of that shared bias. Enrique H. Ruspini similarly argued in Epistemic Logics, Probability, and the Calculus of Evidence that conventional rules for merging information fail in complex environments where pieces of evidence share hidden dependencies. When dependencies are ignored, the system misinterprets redundancy as independent confirmation, resulting in highly confident, incorrect conclusions.
How Source Dependency Manifests in Modern Systems
The tension between mathematical convenience and real-world dependency appears across two major domains in modern computer science: Truth Discovery in data integration and Weak Supervision in machine learning.
1. Truth Discovery and Data Integration
In large-scale data integration, systems scrape facts from thousands of web sources to resolve conflicts and find the ground truth. As Yaliang Li et al. note in A Survey on Truth Discovery, naive aggregation algorithms are easily compromised by "copycat" behavior, where multiple outlets replicate information from a single primary provider.
Laure Berti-Equille et al. formalized this in Sailing the Information Ocean with Awareness of Currents: Discovery and Application of Source Dependence, demonstrating that a single error can propagate across the web, creating a false majority that easily overwhelms a single, accurate, independent source. When data fusion frameworks fail to account for these copying relationships, they artificially inflate their confidence in widespread mistakes (Fusing Data with Correlations).
This echo-chamber effect is not unique to the web; it also severely distorts automated biosurveillance systems that ingest epidemiological signals from one another (Factors Influencing Performance of Internet-Based Biosurveillance Systems Used in Epidemic Intelligence for Early Detection of Infectious Diseases Outbreaks) and decentralized social networks where agents recycle redundant data (Introduction to the Issue on Signal Processing for Social Networks [Guest editorial]).
2. Weak Supervision and Heuristic Labeling
In modern machine learning, programmatic weak supervision frameworks (such as Snorkel) replace manual data labeling with a fleet of user-defined "labeling functions" (LFs)—such as keyword heuristics, regular expressions, or third-party models. These LFs programmatically annotate unlabeled data, and a generative model aggregates their outputs to produce training labels.
Without Dependency Modeling:
[LF 1: Keyword "free"] ──┐
[LF 2: Keyword "money"] ──┼─► [Naive Aggregator] ──► Overconfident / Biased Labels
[LF 3: Regex Match] ──┘ (Treats LFs as 3 independent votes)
With Dependency Modeling:
[LF 1: Keyword "free"] ──┐
├──► [Dependency Aware] ──► Balanced, Accurate Labels
[LF 2: Keyword "money"] ──┘ (Recognizes overlap; discounts redundant votes)
[LF 3: Regex Match] ─────►
The fatal flaw here is that human developers often write LFs that share underlying logic, look at the same features, or rely on the same pre-trained embeddings. Stephen Bach et al. showed in Learning the Structure of Generative Models without Labeled Data that a cluster of low-quality, highly correlated LFs can collectively overwhelm a high-quality, independent source. Because the aggregation model assumes independence, it treats the correlated cluster as multiple independent votes, skewing the synthesized labels.
Mayee F. Chen et al. quantified this phenomenon in Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimation, proving that unmodeled dependencies introduce a permanent "standing bias" into the inferred labels that cannot be cleared merely by adding more unlabeled data. This issue becomes even more pronounced when weak supervision is scaled to structured prediction tasks like sequence labeling or ranking, where errors correlate across different parts of the output space (Lifting Weak Supervision To Structured Prediction; Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler).
Mechanisms for Relaxing the Independence Assumption
To prevent systems from falling victim to correlated biases, researchers have developed several distinct strategies to detect, model, or bypass the source independence assumption.
1. Unsupervised Structure Learning
Instead of forcing users to manually specify how their labeling sources are related, modern frameworks attempt to learn the dependency structure directly from the patterns of agreements and disagreements among the sources themselves, without using any ground-truth labels.
In Learning the Structure of Generative Models without Labeled Data, the authors achieved this by optimizing an $\ell_1$-regularized marginal pseudolikelihood. The algorithm treats the true label as a latent variable and evaluates the covariance between the outputs of the labeling functions. If two sources agree significantly more often than their individual accuracy rates would predict, the model infers a statistical dependency (an edge in a graphical model) and automatically scales back their combined voting weight.
2. Adversarial and Distributionally Robust Optimization
Another approach abandons the attempt to precisely model dependencies altogether. Because estimating exact joint statistics is incredibly difficult without labels, these methods prepare for the worst-case correlation scenario.
Alessio Mazzetto et al. formulated this strategy in Adversarial Multi Class Learning under Weak Supervision with Performance Guarantees by framing label aggregation as an adversarial minimax game. The algorithm assumes that the weak sources may be arbitrarily correlated or even adversarially biased. By finding a predictor that performs robustly against the worst-case valid distribution of errors consistent with the observed data, they provide strong theoretical generalization guarantees without relying on conditional independence.
3. Constrained Optimization over Probabilistic Models
Rather than using complex generative models that require estimating probability distributions, some frameworks treat weak supervision as a geometric or optimization problem.
Chidubem Arachie and Bert Huang introduced Constrained Label Learning (CLL) in Constrained labeling for weakly supervised learning. CLL translates the accuracies and coverages of weak supervision sources into a set of linear constraints bounding the possible label space. By solving a simple convex optimization problem over these constraints, the framework remains naturally resilient to high redundancy and overlapping logic among heuristics, outperforming traditional generative models when sources are highly correlated.
4. End-to-End Joint Agreement
A final paradigm shift involves breaking down the traditional two-step pipeline of weak supervision (where labels are first aggregated, and a downstream model is then trained on those fixed labels).
In End-to-End Weak Supervision, the authors introduced WeaSEL, a framework that trains the label aggregator and the target downstream model simultaneously. By maximizing the mutual agreement between the aggregator's predictions and the downstream model's representations through a neural encoder, the system develops a natural immunity to highly correlated or redundant noise. The downstream model learns to identify the underlying features of the data, allowing it to ignore systematic, correlated errors made by the heuristic rules.
Summary
The assumption that information sources fail independently is a fragile foundation for AI systems. Whether integrating data from the web or combining heuristic rules to train deep networks, ignoring source dependencies leads directly to overconfidence and systemic bias.
By shifting from naive probabilistic pooling to dependency-aware paradigms—whether through unsupervised structure learning, adversarial optimization, or constrained optimization—modern frameworks successfully isolate independent signals from redundant noise. The result is a family of aggregation techniques that are robust enough to find the truth within highly correlated, real-world environments.
Key Foundational Work
An Inference Technique for Integrating Knowledge from Disparate Sources (1981) — contributed 1 concept elements to this direction.
Non-Monotonic Reasoning Using Dempster's Rule (1984) — contributed 1 concept elements to this direction.