Reliable estimation of quality scores by a small calibrated listening panel

Iman Mossavat, W. Bastiaan Kleijn

Abstract

In this work we propose the calibrated-mean-opinion-score (CMOS), which accounts for varying levels of precision and bias across subjects in a listening test. We adopt the Bayesian statistical framework, where the hyper-parameters of priors are learned via empirical Bayes, and the posterior is approximated by a computationally inexpensive variational technique. As our experimental results show, CMOS is more robust to noisy and biased subjects than MOS. As a result, CMOS can be used to improve the reliability of listening test results when a small test panel is used. To correct for the subjects in the test panel, calibration signals are required. Calibration signals are rated by a panel larger than the test panel. The key to saving human labor and cost is that only a few calibration signals are required, and that it is possible to share calibration signals across listening tests.
BibTeX
@inproceedings{mossavat2013reliable,
  author = {Iman Mossavat and W. Bastiaan Kleijn},
  title = {{Reliable estimation of quality scores by a small calibrated listening panel}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2013},
  doi = {10.1109/icassp.2013.6639263},
}

Loading related content...

Figure