Abstract
While traditional Commonsense Knowledge (CSK) research focuses on the plausibility of facts, e-commerce applications require a more nuanced understanding of salience—identifying which facts are most characteristic or important to a specific concept. This paper introduces the task of Commonsense Knowledge Salience Evaluation and presents BSEE, a novel benchmark dataset specifically designed for the e-commerce domain. We propose PMI-tuning, a method that leverages Masked Language Models (MLMs) and Pointwise Mutual Information to identify salient relationships. Our results demonstrate that salience is a distinct metric from plausibility, showing a moderate positive correlation with real-world click-through rates (CTR), and that our proposed method outperforms existing baselines in generalizing to unseen concepts.
Introduction
In the field of Commonsense Knowledge (CSK), the prevailing research paradigm focuses on plausibility: determining whether a statement like "Lions have tails" is true. However, in practical applications such as e-commerce search and recommendation, truth alone is insufficient. For a user searching for "running," the fact that "running requires shoes" is far more salient than the fact that "running involves breathing," even though both are true.
Existing knowledge bases often rank facts by confidence scores (truth probability) rather than salience (characteristic importance). This gap hinders AI agents from understanding the key traits humans naturally associate with queries. Modeling salience is challenging because it is inherently subjective and context-dependent. This paper addresses these challenges by formalizing salience evaluation and providing the tools necessary to bridge the gap between abstract knowledge and industrial utility.
Defining Salience: Necessity and Sufficiency
To move beyond subjective intuition, we decompose salience into two logical pillars:
- Necessity (Nec): Is the subject the primary reason for the object to hold? (e.g., "Running" is a primary reason to need a "weight running vest").
- Sufficiency (Suf): Does the property hold for most instances of the subject? (e.g., "Running" almost always involves "shoes").
We define the salience score ($S$) as a weighted combination of these two factors: $S = \lambda Suf + (1-\lambda) Nec$. This formulation allows for a more granular assessment of how essential a piece of knowledge is to a specific concept.
The BSEE Benchmark
We introduce the Benchmark dataset of Salience Evaluation in E-commerce (BSEE) to facilitate research in this area. The dataset consists of over 20,000 instances covering 10,783 entities and three core e-commerce relations: requires, complementary, and capable of.
Annotation and Splits
Human experts annotated triples using a three-point Likert scale for both necessity and sufficiency. To test model robustness, we provide two evaluation splits:
- Random Split: A standard split for general performance testing.
- Concept Split: A more rigorous test where concepts in the test set are entirely unseen during training. This evaluates the model's ability to generalize to out-of-vocabulary (OOV) e-commerce scenarios.
To prevent models from relying on shallow lexical shortcuts, we also included adversarial examples (e.g., changing "Running requires running shoes" to "Walking requires running shoes").
Methodology: PMI-tuning
We propose PMI-tuning to evaluate salience by probing the latent knowledge within pre-trained Masked Language Models (MLMs). Instead of standard fine-tuning, we calculate the Pointwise Mutual Information (PMI) between subjects and objects.
Our approach utilizes "soft prompts"—trainable embedding tensors—and a prompt encoder consisting of an LSTM and MLP. This setup allows the model to estimate the strength of the relationship between a subject and object within a specific predicate context. By focusing on PMI, the model captures the discrepancy between probabilities, identifying associations that are uniquely strong for a specific concept rather than just high-frequency word pairings.
Results and Discussion
Our experiments reveal several key insights into the nature of salience in AI:
The Generalization Gap
While models achieved high performance on the Random Split (up to 96.4 F1), performance dropped significantly on the Concept Split. The best baseline, PKGC, reached only 61.2 F1, while our PMI-tuning method achieved a state-of-the-art F1 of 63.4 and an AUC of 74.3. This highlights the difficulty of teaching AI to recognize the salience of entirely new product categories.
Industrial Relevance
We validated the practical utility of salience by comparing our model's predictions with real-world e-commerce search logs. We found a moderate positive correlation (Spearman coefficient of 0.42) between predicted salience scores and Click-Through Rates (CTR). This confirms that salience is a meaningful metric for predicting user intent and behavior.
Efficiency and Robustness
PMI-tuning proved remarkably efficient in low-resource settings. When trained on only 20% of the available data, it maintained higher performance than competing models, suggesting that it effectively leverages the rich latent knowledge already present in large-scale pre-trained models.
Conclusion and Future Work
This work establishes Commonsense Knowledge Salience as a critical task for practical AI applications. By releasing the BSEE benchmark and the PMI-tuning method, we provide a foundation for building AI systems that understand not just what is true, but what matters.
Future research will focus on expanding relation coverage beyond the current three types and addressing the inherent subjectivity of salience through personalized or demographic-specific modeling. Additionally, we aim to scale the method for long-form text and release a multilingual version of the BSEE dataset to support global e-commerce research.
Summary updated