Commonsense Knowledge Salience Evaluation with a Benchmark Dataset in E-commerce

Yincen Qu, Ningyu Zhang, Hui Chen, Zelin Dai, Chengming Wang, Xiaoyu Wang, Qiang Chen, Huajun Chen

Abstract

While traditional Commonsense Knowledge (CSK) research focuses on the plausibility of facts, e-commerce applications require a more nuanced understanding of salience—identifying which facts are most characteristic or important to a specific concept. This paper introduces the task of Commonsense Knowledge Salience Evaluation and presents BSEE, a novel benchmark dataset specifically designed for the e-commerce domain. We propose PMI-tuning, a method that leverages Masked Language Models (MLMs) and Pointwise Mutual Information to identify salient relationships. Our results demonstrate that salience is a distinct metric from plausibility, showing a moderate positive correlation with real-world click-through rates (CTR), and that our proposed method outperforms existing baselines in generalizing to unseen concepts.

Introduction

In the field of Commonsense Knowledge (CSK), the prevailing research paradigm focuses on plausibility: determining whether a statement like "Lions have tails" is true. However, in practical applications such as e-commerce search and recommendation, truth alone is insufficient. For a user searching for "running," the fact that "running requires shoes" is far more salient than the fact that "running involves breathing," even though both are true.

Existing knowledge bases often rank facts by confidence scores (truth probability) rather than salience (characteristic importance). This gap hinders AI agents from understanding the key traits humans naturally associate with queries. Modeling salience is challenging because it is inherently subjective and context-dependent. This paper addresses these challenges by formalizing salience evaluation and providing the tools necessary to bridge the gap between abstract knowledge and industrial utility.

Defining Salience: Necessity and Sufficiency

To move beyond subjective intuition, we decompose salience into two logical pillars:

  1. Necessity (Nec): Is the subject the primary reason for the object to hold? (e.g., "Running" is a primary reason to need a "weight running vest").
  2. Sufficiency (Suf): Does the property hold for most instances of the subject? (e.g., "Running" almost always involves "shoes").

We define the salience score ($S$) as a weighted combination of these two factors: $S = \lambda Suf + (1-\lambda) Nec$. This formulation allows for a more granular assessment of how essential a piece of knowledge is to a specific concept.

The BSEE Benchmark

We introduce the Benchmark dataset of Salience Evaluation in E-commerce (BSEE) to facilitate research in this area. The dataset consists of over 20,000 instances covering 10,783 entities and three core e-commerce relations: requires, complementary, and capable of.

Annotation and Splits

Human experts annotated triples using a three-point Likert scale for both necessity and sufficiency. To test model robustness, we provide two evaluation splits:

  • Random Split: A standard split for general performance testing.
  • Concept Split: A more rigorous test where concepts in the test set are entirely unseen during training. This evaluates the model's ability to generalize to out-of-vocabulary (OOV) e-commerce scenarios.

To prevent models from relying on shallow lexical shortcuts, we also included adversarial examples (e.g., changing "Running requires running shoes" to "Walking requires running shoes").

Methodology: PMI-tuning

We propose PMI-tuning to evaluate salience by probing the latent knowledge within pre-trained Masked Language Models (MLMs). Instead of standard fine-tuning, we calculate the Pointwise Mutual Information (PMI) between subjects and objects.

Our approach utilizes "soft prompts"—trainable embedding tensors—and a prompt encoder consisting of an LSTM and MLP. This setup allows the model to estimate the strength of the relationship between a subject and object within a specific predicate context. By focusing on PMI, the model captures the discrepancy between probabilities, identifying associations that are uniquely strong for a specific concept rather than just high-frequency word pairings.

Results and Discussion

Our experiments reveal several key insights into the nature of salience in AI:

The Generalization Gap

While models achieved high performance on the Random Split (up to 96.4 F1), performance dropped significantly on the Concept Split. The best baseline, PKGC, reached only 61.2 F1, while our PMI-tuning method achieved a state-of-the-art F1 of 63.4 and an AUC of 74.3. This highlights the difficulty of teaching AI to recognize the salience of entirely new product categories.

Industrial Relevance

We validated the practical utility of salience by comparing our model's predictions with real-world e-commerce search logs. We found a moderate positive correlation (Spearman coefficient of 0.42) between predicted salience scores and Click-Through Rates (CTR). This confirms that salience is a meaningful metric for predicting user intent and behavior.

Efficiency and Robustness

PMI-tuning proved remarkably efficient in low-resource settings. When trained on only 20% of the available data, it maintained higher performance than competing models, suggesting that it effectively leverages the rich latent knowledge already present in large-scale pre-trained models.

Conclusion and Future Work

This work establishes Commonsense Knowledge Salience as a critical task for practical AI applications. By releasing the BSEE benchmark and the PMI-tuning method, we provide a foundation for building AI systems that understand not just what is true, but what matters.

Future research will focus on expanding relation coverage beyond the current three types and addressing the inherent subjectivity of salience through personalized or demographic-specific modeling. Additionally, we aim to scale the method for long-form text and release a multilingual version of the BSEE dataset to support global e-commerce research.

Summary updated

Abstract

In e-commerce, the salience of commonsense knowledge (CSK) is beneficial for widespread applications such as product search and recommendation. For example, when users search for ``running'' in e-commerce, they would like to find products highly related to running, such as ``running shoes'' rather than ``shoes''. Nevertheless, many existing CSK collections rank statements solely by confidence scores, and there is no information about which ones are salient from a human perspective. In this work, we define the task of supervised salience evaluation, where given a CSK triple, the model is required to learn whether the triple is salient or not. In addition to formulating the new task, we also release a new Benchmark dataset of Salience Evaluation in E-commerce (BSEE) and hope to promote related research on commonsense knowledge salience evaluation. We conduct experiments in the dataset with several representative baseline models. The experimental results show that salience evaluation is a challenging task where models perform poorly on our evaluation set. We further propose a simple but effective approach, PMI-tuning, which shows promise for solving this novel problem. Code is available in \url{https://github.com/OpenBGBenchmark/OpenBG-CSK.
BibTeX
@inproceedings{qu2022commonsense,
  author = {Yincen Qu and Ningyu Zhang and Hui Chen and Zelin Dai and Chengming Wang and Xiaoyu Wang and Qiang Chen and Huajun Chen},
  title = {{Commonsense Knowledge Salience Evaluation with a Benchmark Dataset in E-commerce}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.2},
}
Figure