TL;DR
This paper tackles the complex problem of evaluating a candidate based on multiple criteria and expert opinions, where both criteria and experts have varying importance and reliability. The authors propose and compare two distinct approaches – the Transferable Belief Model (TBM) and Qualitative Possibility Theory (QPT) – to handle the inherent uncertainty and imprecision in such assessments, offering a framework for decision-makers to choose the most suitable method.
What Problem Does This Paper Address?
Choosing the "best" candidate from a pool, whether it's a job applicant, a product, or any abstract entity, is a common decision-making challenge. This process often involves evaluating the candidate against several criteria, with different experts providing their assessments for each criterion. The core difficulty lies in the fact that criteria themselves have varying importance, experts have different levels of competence and reliability, and the assessments themselves are often imprecise or uncertain. Traditional methods might struggle with this multifaceted uncertainty. This paper aims to provide a structured way to handle this by comparing two formalisms for representing and reasoning with uncertainty: belief functions (TBM) and possibility theory (QPT).
Methods
The paper presents two main approaches:
- Transferable Belief Model (TBM): This approach uses belief functions, which are analogous to subjective probabilities but allow for representing ignorance more explicitly. It handles uncertainty through numerical belief masses. The TBM approach involves:
- Representing expert opinions as belief functions.
- Discounting these beliefs based on expert reliability and confidence.
- Combining expert opinions for each criterion using Dempster's rule of combination.
- Aggregating criterion evaluations into a global "Goodness Score" using a compensatory approach (implicitly, through transformations).
- Transforming the final belief function into a pignistic probability for ranking.
- Qualitative Possibility Theory (QPT): This approach uses possibility distributions to represent imprecise information, often using linguistic terms or interval-valued scores. It offers a more symbolic and qualitative way to handle uncertainty. The QPT approach involves:
- Representing expert opinions and their confidence as possibility distributions.
- Discounting these distributions based on expert reliability.
- Merging expert opinions for each criterion using a weighted disjunction.
- Aggregating criterion evaluations into a global score using a conjunctive approach (weighted minimum).
- Using fuzzy pattern matching to assess the candidate's profile against criteria.
Datasets
Not discussed in the paper. The paper uses a synthetic example with 6 criteria and 4 experts to illustrate the methods.
Key Insights and Ideas
- Two Sides of the Same Coin: Both TBM and QPT can model imprecision, but they offer different perspectives and assumptions. TBM is numerical and can capture compensatory effects, while QPT is qualitative and excels at handling linguistic uncertainty and conjunctive reasoning.
- The Importance of Assumptions: The choice between TBM and QPT hinges on the underlying assumptions about how criteria are aggregated (compensatory vs. conjunctive) and how uncertainty is represented (numerical beliefs vs. qualitative possibility).
- Handling Imprecision is Key: The paper highlights that real-world assessments are rarely precise. Both frameworks provide mechanisms to deal with this imprecision, whether it's through interval-valued scores or belief functions.
- Expert Reliability Matters: Explicitly modeling expert reliability and confidence is crucial for robust decision-making. Both approaches incorporate mechanisms to discount or weight expert opinions accordingly.
- Explanation Capabilities: The qualitative nature of QPT offers a potential advantage in providing logical and understandable explanations for the assessment results.
Key Results
The paper demonstrates how both TBM and QPT can be applied to a multi-expert, multi-criteria assessment problem. While specific quantitative results are not the primary focus (as it's an illustrative example), the paper shows:
- TBM: Produces a belief function for the "Goodness Score" and a pignistic probability of 4.61 for the expected score, suggesting the candidate is "Good."
- QPT: Produces a possibility distribution for the final score, indicating the candidate is "certainly not a very good candidate, but most likely between bad and good." The example highlights how conflicts (e.g., on the "Crea" criterion) and unknown information (e.g., on "Dec") influence the final assessment.
Limitations and Future Work
- Unspecified Aggregation: The paper notes that the problem statement doesn't fully specify how criteria should be aggregated, leading to different choices (compensatory in TBM, conjunctive in QPT) which significantly impact the results.
- Commensurateness Hypothesis: Both approaches rely on assumptions about the commensurateness of scales (e.g., relating different ordinal scales).
- QPT's Interval Scale Assumption: The paper acknowledges that assuming an interval scale for QPT's distributions might be questionable.
- Future Work: The authors suggest exploring other aggregation operators within QPT (like median for compensation) and further investigating the explanation capabilities of both frameworks. They also point out the need for sensitivity analysis to understand the robustness of the conclusions to parameter changes.
Summary updated