Analyzing the Relation Between Overall Quality and the Quality of Individual Phases in a Telephone Conversation

Friedemann Köster, Sebastian MöllerInterspeech 2016

TL;DR

This paper introduces a novel method to assess telephone conversation quality by breaking it down into distinct listening, speaking, and interaction phases. They found that the interaction phase has the most significant impact on overall perceived quality, and developed a linear model to predict overall quality based on these phase-specific ratings.

What Problem Does This Paper Address?

Current methods for evaluating telephone call quality, like the Mean Opinion Score (MOS), provide a single overall score but fail to offer diagnostic insights into why a call might be perceived as poor. This makes it difficult for service providers to pinpoint and fix specific issues. Furthermore, these methods often treat conversations as a monolithic experience, neglecting the distinct perceptual differences between actively listening, speaking, and the back-and-forth of interaction. This paper aims to bridge this gap by analyzing conversational quality on a phase-by-phase basis, providing more granular diagnostic information and a better understanding of how different conversational segments contribute to the overall experience.

Methods

The researchers developed a new experimental paradigm to allow participants to evaluate each conversational phase (listening, speaking, interaction) separately, as well as the overall conversation. This involved structured role-playing tasks and specific tests designed to isolate the impact of different degradations on each phase. They then used a multiple linear regression analysis to model the relationship between the perceived quality of each phase and the overall conversational quality. This approach allows them to determine the "weights" of each phase in contributing to the final perceived quality.

Datasets

The study utilized a newly designed conversational experiment involving 36 participants. They were exposed to 11 different simulated audio conditions, each designed to trigger specific phases or combinations of phases (e.g., sidetone for speaking, packet loss for listening, delay for interaction). The participants provided subjective quality ratings on a continuous scale, which were then transformed into MOS scores.

Key Insights and Ideas

  • Decomposition is Key: Breaking down a conversation into listening, speaking, and interaction phases provides a much richer understanding of quality than a single overall score.
  • Interaction Dominates: The "Interaction Phase" appears to be the most critical factor influencing overall perceived conversational quality. This suggests that the dynamic back-and-forth, including turn-taking and potential disruptions, is paramount to user satisfaction.
  • Phase Independence (Mostly): While the interaction phase is strongly linked to both speaking and listening, the speaking and listening phases themselves seem to be relatively independent of each other in terms of their direct impact on overall quality.
  • Speaking's Influence: The speaking phase has a greater impact on overall quality than the listening phase, possibly due to its connection with the interaction phase.
  • Linear Modeling Works: A linear model can effectively predict overall conversational quality based on the ratings of its individual phases, demonstrating a quantifiable relationship.

Key Results

  • The experiment successfully gathered phase-specific and overall quality ratings for 11 different audio conditions.
  • Statistical analysis (ANOVA) confirmed that the tested conditions significantly influenced all four quality ratings (overall, listening, speaking, interaction).
  • Correlations showed significant relationships between individual phase ratings and overall quality, with the interaction phase exhibiting the strongest correlation.
  • A multiple linear regression model was developed, estimating overall conversational quality (MOSco) using the formula: MOSco = -1.02 + 0.25 × MOSLI + 0.39 × MOSSP + 0.54 × MOSIN.
  • This model estimated subjective MOSco values with a high correlation (ρ = .94) and a low RMSE (0.27) compared to the actual subjective ratings.

Limitations and Future Work

The authors acknowledge that this is a pilot study and the developed model is based on a limited number of conditions. They suggest that further work is needed to:

  • Identify the specific weights of perceptual dimensions within each phase to further refine diagnostic capabilities.
  • Test a wider range of degradations and conditions to validate and generalize the proposed conversational model.
  • Conduct further research and experiments to create a more comprehensive quality profile.
  • The study also notes that the predictor coefficients in the regression were not statistically significant individually, likely due to high collinearity between the phase predictors. While the overall model is significant, this suggests potential redundancy in the phase information.

Summary updated

BibTeX
@inproceedings{koster2016analyzing,
  author = {Friedemann Köster and Sebastian Möller},
  title = {{Analyzing the Relation Between Overall Quality and the Quality of Individual Phases in a Telephone Conversation}},
  booktitle = {Interspeech},
  year = {2016},
}
Figure