Advanced time shrinking using a drop classifier based on codec features

Jochen Issing, Nikolaus Färber, Reinhard GermanInterspeech 2015

TL;DR

This paper introduces a more intelligent way to shrink audio streams in real-time for VoIP, improving quality by selectively dropping frames that are less likely to be noticed by listeners. By using features already available in audio codecs, they achieve a significant quality boost with minimal computational overhead, making it ideal for low-power devices.

What Problem Does This Paper Address?

In Voice over IP (VoIP) communication, network delays can cause audio packets to arrive out of order or too late, leading to audible glitches like dropouts. To combat this, adaptive playout mechanisms are used to buffer and manage packet arrival times. When network conditions demand it, these systems need to "shrink" the audio stream to avoid excessive delay. Traditional methods for shrinking audio often involve simply dropping frames at regular intervals or randomly, which can lead to noticeable quality degradation. The challenge is to shrink the audio effectively without significantly impacting the perceived quality, especially in a low-complexity environment like VoIP.

Methods

The core of this paper is an improved time-shrinking algorithm for VoIP that leverages a drop classifier. This classifier, built using decision trees, predicts whether an audio frame can be dropped with minimal impact on perceived audio quality. Crucially, the classifier relies only on audio signal features that are already available within the audio codec itself. This keeps the computational complexity very low.

The process involves:

  1. Feature Extraction: Extracting features like signal energy, zero-crossing rate, pitch, and periodicity from audio frames.
  2. Classifier Training: Training a decision tree classifier on manually classified frames (labeled as "dropable" or "audible") using these codec-available features.
  3. Adaptive Playout: During real-time communication, the classifier is used to decide which frames are candidates for dropping when time shrinking is needed. Frames classified as "dropable" are then removed.

The authors also explore weighting strategies during classifier training to prioritize the correct classification of "audible" frames, thereby further minimizing quality degradation.

Datasets

  • Training Data: The authors created their own dataset by encoding nine different audio stimuli (six speech, three music) using a low-delay version of the Unified Speech and Audio Coding (USAC) codec. These encoded bitstreams were then decoded and manipulated to create "drop audio files" for classification.
  • Evaluation Data: For subjective evaluation, they used six pairs of speech items and two music pieces from the McGill speech database. Background noise (street, car, office, music, babble) was added to some speech items.

Key Insights and Ideas

  • Codec-Internal Features Suffice: The most significant insight is that high-quality frame drop prediction can be achieved using only features already available within standard audio codecs. This bypasses the need for complex, computationally expensive feature extraction.
  • Classifier Outperforms Simple Dropping: The proposed classifier significantly improves audio quality compared to simple regular or random frame dropping, even at higher shrinking rates.
  • Low Complexity is Key: The method's low computational complexity makes it highly suitable for resource-constrained devices like smartphones and for VoIP applications where latency is critical.
  • Shrinking Rate vs. Quality Trade-off: The classifier allows for a higher effective shrinking rate (up to 10%) while maintaining better quality than simple dropping methods at lower rates.

Key Results

  • Objective Improvement: The classifier improves audio quality by approximately 0.5 Mean Opinion Score (MOS) points compared to the previous method without classification.
  • Computational Efficiency: The computational complexity is reduced by a factor of approximately 10⁴ compared to other advanced methods like WSOLA.
  • Classification Accuracy: The low-delay USAC classifier achieved a classification accuracy of 67.6% (with weight factor 2), with only 14.4% of audible frames incorrectly classified as droppable.
  • Subjective Evaluation: In listening tests, the classifier-based method achieved nearly fair audio quality for 10% shrinking, a notable improvement over simple frame dropping which was rated as poor. It performed particularly well for speech.

Limitations and Future Work

  • Music vs. Speech: The classifier performed better on speech than on music. Future work could focus on improving its performance for music content.
  • Feature Set Expansion: While the current approach uses codec-internal features, the authors suggest that a broader selection of signal features could potentially lead to even higher quality levels.
  • Adaptation Period: The current method is primarily for adaptation periods when network conditions change. Future work could explore more permanent adaptation strategies.
  • Shrinking Rate Limits: The improvement in quality diminishes at shrinking rates above 10%. Further research might explore different classification strategies for higher rates.

Summary updated

BibTeX
@inproceedings{issing2015advanced,
  author = {Jochen Issing and Nikolaus Färber and Reinhard German},
  title = {{Advanced time shrinking using a drop classifier based on codec features}},
  booktitle = {Interspeech},
  year = {2015},
}
Figure