What can be learned from satisfaction assessments?

Naftali Cohen, Simran Lamba, Prashant ReddyIcaif 2020 · DOI

TL;DR

This paper investigates the impact of "noisy" labels in customer satisfaction surveys, particularly those using ordinal scales. They demonstrate that common practices like unevenly segmenting responses can significantly degrade classification accuracy and propose a simple calibration method using textual descriptions to mitigate this error.

What Problem Does This Paper Address?

Customer satisfaction surveys are a cornerstone for businesses to gauge performance and identify areas for improvement. However, these surveys often rely on ordinal scales (e.g., 1-10) where respondents provide subjective numerical ratings. The authors highlight a critical issue: the "non-systematic bias" or "noise" inherent in these responses. This noise arises because a customer's true opinion might not perfectly translate to a single numerical score every time they are asked. This inherent variability, coupled with the common practice of grouping these scores into uneven bins (e.g., "Detractors," "Passives," "Promoters"), can lead to a compounding of errors, making it difficult to accurately learn from the data. The paper aims to quantify this error and explore ways to reduce its impact.

Methods

The core of the paper's methodology involves simulating and analyzing the effects of "intrinsic variability" on ordinal survey data. They model this variability as a discrete uniform distribution around a customer's "true" or "unbiased" opinion.

  • Simulation: They create synthetic datasets (SYNTH) where customers have a true score, and then introduce random noise (intrinsic variability) to simulate how their reported score might differ.
  • Analysis of Binning: They analyze how different ways of grouping these noisy ordinal scores into categories (e.g., 3-class or 2-class) affect classification accuracy and precision. This includes exploring "narrow," "medium," and "wide" binning strategies.
  • Calibration: They propose and test a "semi-ordinal, text-calibrated" survey design. This involves augmenting each numerical score on the ordinal scale with a short textual description (e.g., "7) My city is a decent place").
  • Estimation of Variability: They develop a method to estimate the intrinsic variability in real-world survey data by regressing biased scores on unbiased scores and equating the resulting RMSE to that of synthetic data.

Datasets

  • BRAND Dataset: Real-world Net Promoter Score (NPS) survey data from a large retail bank, consisting of 10,000 customer responses. This dataset is used to demonstrate the real-world impact of their findings and to estimate intrinsic variability.
  • SYNTH Dataset: A synthetically generated dataset used extensively for controlled experiments. It allows researchers to precisely manipulate intrinsic variability and study its effects on classification metrics.
  • CITY Dataset: An online survey conducted by the authors with approximately 200 employees of the retail bank. This dataset was specifically designed to collect both biased responses and approximations of true underlying opinions, and to explore the effectiveness of textual calibration.

Key Insights and Ideas

  • Noise Compounding: The paper strongly emphasizes that non-systematic error in ordinal surveys doesn't just cancel out; it compounds, leading to a significant and often exponential deterioration in classification performance.
  • Uneven Binning is Problematic: The common practice of segmenting ordinal survey responses into uneven bins (like the standard NPS categories) exacerbates the impact of intrinsic variability. This is because it can lead to a drastic imbalance in the resulting categories, particularly shrinking the "Passive" group.
  • Intrinsic Variability Sets an Upper Bound: The degree of intrinsic variability in survey responses directly limits the achievable accuracy and predictability of any classification model trained on that data.
  • Textual Calibration is Effective: Adding short textual descriptions to numerical survey responses can significantly improve the "uniformity" of responses and reduce the impact of non-systematic error, making the data more reliable.
  • Binning Design Matters: The way ordinal scores are binned into categories has a substantial impact on classification accuracy. "Medium" and "wide" binning strategies tend to perform better than "narrow" ones, especially as intrinsic variability increases.

Key Results

  • Even a small amount of intrinsic variability (e.g., ±1) can lead to a 20% decrease in the upper bounds of accuracy and precision scores for three-class classification.
  • As intrinsic variability increases, the "Detractor" category grows disproportionately, while the "Passive" category shrinks dramatically, leading to class imbalance.
  • For the BRAND dataset, the authors estimate an intrinsic variability of approximately ±2.5, suggesting an upper bound on accuracy scores of around 0.65.
  • The "calibrated" survey (with textual descriptions) in the CITY dataset showed a more uniform distribution of responses compared to the "uncalibrated" survey, with p-values for uniformity tests being orders of magnitude larger.
  • The NPS score is less sensitive to intrinsic variability than accuracy scores, as it primarily relies on the difference between "Promoters" and "Detractors," whose relative proportions remain more stable.

Limitations and Future Work

  • Uniformity Assumption: The study relies on the assumption of uniform intrinsic variability. While they show it can be estimated, real-world variability might be more complex.
  • Limited Scope of CITY Dataset: The CITY dataset, while innovative, was limited to retail bank employees and a specific set of questions, which might limit generalizability.
  • Future Work: The authors suggest exploring the deterioration effect using more general metrics like confusion matrices and ROC curves. They also propose further investigation into more sophisticated calibration methods and understanding the impact of different types of noise beyond simple uniform variability.

Summary updated

BibTeX
@inproceedings{cohen2020what,
  author = {Naftali Cohen and Simran Lamba and Prashant Reddy},
  title = {{What can be learned from satisfaction assessments?}},
  booktitle = {Icaif},
  year = {2020},
  doi = {10.1145/3383455.3422535},
}

Loading related content...

Figure