Introduction
Knowledge Bases (KBs) such as Wikidata and YAGO serve as critical backbones for modern AI, yet they are inherently incomplete and subject to significant observation bias. We are far more likely to find recorded facts about prominent figures or specific geographic regions than about less visible entities. Traditional rule-based Knowledge Base Completion (KBC) methods often fail to account for this selection mechanism, leading to biased confidence estimates. This paper reframes KBC through the lens of Positive and Unlabeled (PU) learning, introducing a framework that utilizes propensity scores to mitigate these systemic biases.
The Challenge of Observation Bias
When estimating the precision or "confidence" of a rule (e.g., "If X is married to Y, then X lives in the same country as Y"), researchers typically rely on two restrictive assumptions because KBs only contain positive examples:
- Closed-World Assumption (CWA): Assumes any fact not present in the KB is false.
- Partial-Completeness Assumption (PCA): Assumes that if a subject has at least one known object for a specific predicate, all other objects for that subject-predicate pair are also known.
Both assumptions ignore the underlying selection mechanism, often overestimating the confidence of rules that align with existing data biases while penalizing those that do not.
Methodology: KBC as PU Learning
By bridging KBC with PU learning—a field designed to learn from positive and unlabeled data—the authors categorize KB population into a taxonomy of bias. This ranges from Selected Completely At Random (SCAR) to Selected At Random (SAR), where the probability of a fact being recorded (the propensity score) depends on its specific characteristics, such as the subject's popularity.
To address these biases, the paper introduces two novel estimators:
- Inverse Propensity Weighted (IPW): This estimator weights observed facts by the inverse of their likelihood of being recorded, effectively giving more weight to under-reported types of facts.
- IPW-PCA: A hybrid approach that combines the strengths of the PCA assumption with propensity weighting to improve robustness.
Experimental Evaluation and Results
The proposed methods were evaluated using Yago3-10. By treating the full dataset as ground truth and simulating various incomplete "known" KBs through biased selection mechanisms, the authors compared their estimators against traditional baselines using metrics like the Brier score.
As demonstrated in the experimental results, the IPW estimator achieves significantly higher accuracy in SAR scenarios. While the CWA estimator's performance degrades as the selection probability decreases, the IPW estimator remains remarkably stable and close to the true confidence level. Furthermore, the study found that even "noisy" or imperfect propensity scores (estimated within a $\pm 10\%$ margin) still yielded better results than assuming the data was unbiased.
Conclusion
This work demonstrates that formalizing KBC as a PU learning problem provides a mathematically rigorous way to handle observation bias. By moving beyond naive assumptions like CWA and PCA, the proposed IPW estimators ensure that rule-based completion systems do not simply reinforce the cultural or systemic biases already present in the data. Future work will explore extending these insights to embedding-based KBC methods and automating the estimation of propensity scores using external domain knowledge.
Summary updated