TL;DR
This paper tackles the challenge of building highly accurate knowledge bases (KBs) that include subjective information, like "is this restaurant romantic?" They introduce a probabilistic approach that models user consensus rather than binary true/false values, allowing for tunable precision. This method effectively handles ambiguity and improves KB performance, especially for subjective attributes.
What Problem Does This Paper Address?
Many applications, from recommendation engines to mapping services, rely on structured knowledge bases (KBs) to function. The accuracy of these KBs is paramount, especially when they provide information that can lead to poor user experiences if incorrect (e.g., a restaurant that's not wheelchair accessible). While factual attributes are relatively straightforward to verify, subjective attributes (e.g., "kid-friendly," "romantic") introduce significant ambiguity. Estimating the "truth" of these subjective attributes is difficult, making it challenging to build KBs with controllable precision (i.e., setting a specific false positive rate). Traditional methods often struggle with this ambiguity, leading to either overly conservative KBs that miss useful information or KBs with too many errors.
Methods
The core of this paper's contribution is a probabilistic model for estimating the "yes rate" of an entity-attribute pair. Instead of treating an attribute as simply true or false, they model the consensus among users. This is achieved by:
- Probabilistic Yes Rate Modeling: Each entity-attribute pair is assigned a latent "yes rate" (θ), representing the probability that a user would agree the attribute applies. This rate is modeled as a Beta distribution, which naturally captures uncertainty.
- Beta-Binomial Observation Model: Observed user votes (yes/no) are modeled as samples from a binomial distribution parameterized by the latent yes rate.
- Leveraging Side Information: To combat uncertainty, especially for pairs with few votes, the model incorporates "side information" associated with entities (e.g., text from a location's homepage). This information is used to learn embeddings for locations.
- Attribute Relatedness: The paper also analyzes correlations between attributes. This insight is used to develop architectures that share information across related attributes, leading to more accurate and confident predictions.
- Neural Network Architectures: Three neural network architectures are proposed to learn the parameters of the probabilistic model:
- Multi-task Baseline (ML): Learns a shared location embedding and attribute-specific functions.
- Alien Vote Architecture (AV): Directly leverages "alien votes" (votes for other attributes of the same entity) to learn attribute-attribute relationships.
- Independent AV Architecture (IAV): Uses separate networks for each attribute, allowing for more focused learning but potentially missing out on shared features.
Datasets
The primary dataset used is from Google Maps, containing millions of locations and over 70 attributes. This dataset includes:
- Observed Votes: Yes/no votes collected from users for location-attribute pairs.
- Side Information: Unstructured text data (e.g., homepage content) and structured tags associated with locations.
- Gold Labels: A subset of location-attribute pairs manually labeled by vetted workers to evaluate model performance.
Key Insights and Ideas
- Subjectivity is a Feature, Not a Bug: The paper reframes the ambiguity of subjective attributes as a source of uncertainty that can be probabilistically modeled, rather than a problem to be eliminated.
- Tunable Precision is Achievable: By modeling uncertainty, the KB can be configured to operate at a specific false positive rate, crucial for applications where errors are costly.
- Alien Votes are Powerful: Directly incorporating information about other attributes of an entity (alien votes) significantly improves model performance compared to indirect sharing via multi-task learning. This highlights the importance of attribute relatedness.
- Confidence Matters: The estimated uncertainty (τ) is a key indicator of model confidence. Higher confidence leads to more predictions, while lower confidence results in "No Prediction" outputs, effectively controlling precision.
- Attribute Relatedness Can Be Tricky: While generally beneficial, sharing information between attributes can be detrimental if one attribute is data-rich and the other is data-poor, potentially leading to negative transfer.
Key Results
- Improved F-scores: The proposed AV and IAV models consistently outperform the ML baseline and two empirical baselines in terms of F-score.
- High Precision Maintained: The models can be constrained to maintain a 95% precision rate, with the best model achieving comparable F-scores to a baseline that doesn't support tunable precision.
- Dominating Performance: When unconstrained, the best model (IAV raw) dominates empirical baselines by a significant margin (12% and 28% F-score).
- Well-Calibrated Models: Q-Q plots demonstrate that the learned models are well-calibrated, meaning their predicted probabilities accurately reflect the true likelihood of an attribute being present.
- Alien Votes Boost Confidence: The AV model, by leveraging alien votes, estimates higher confidence (τ) values, leading to more predictions and better recall compared to a model that ignores alien votes.
Limitations and Future Work
- Gold Label Imprecision: The evaluation using gold labels has limitations, particularly for subjective attributes where worker agreement might be lower, leading to a conservative estimate of model quality.
- Data Scarcity: For attributes with very few votes, relatedness estimates can be noisy and misrepresent true correlations.
- Future Work: The paper suggests that the framework is applicable to other domains involving crowdsourced categorical observations. Further exploration of attribute-attribute correlations and their impact on different information sharing strategies is also implied. The authors also note that the IAV architecture has more parameters and longer training times, suggesting potential for optimization.
Summary updated