TL;DR
This paper tackles a limitation in Spatial Audio Object Coding (SAOC) for interactive audio services, specifically the inability to perfectly remove vocal tracks in Karaoke modes. They propose a modified SAOC scheme that extracts and utilizes harmonic information from vocals to achieve cleaner elimination, significantly improving the user experience for interactive music applications.
What Problem Does This Paper Address?
Interactive audio services allow users to manipulate individual audio components (like instruments and vocals) to create personalized soundscapes. A key application is "Karaoke mode," where users want to remove the lead vocal to sing along. Spatial Audio Object Coding (SAOC) is a technology designed to reduce the bitrate of these services by transmitting a down-mixed audio signal along with spatial parameters, rather than individual object streams. However, the standard SAOC struggles with perfect vocal removal. This is because the SAOC's sub-band processing has limited frequency resolution, leading to residual vocal components in the reconstructed background music, especially when a vocal is fully suppressed. This degrades the listening experience, making it impossible to achieve a truly clean instrumental track.
Methods
The proposed solution introduces a "harmonic extraction and elimination structure" into the SAOC decoder.
- Harmonic Extraction: In the SAOC encoder, this module identifies and extracts key harmonic information from the vocal object. This includes the fundamental frequency (F0) and the amplitudes at harmonic frequencies. The process involves spectral whitening to flatten the power spectral distribution of the vocal signal, followed by a salience function calculation to estimate the fundamental frequency. This F0 is then calibrated to ensure accurate transmission.
- Harmonic Elimination: In the SAOC decoder, this module uses the extracted harmonic information to actively remove residual vocal components from the background music. It calculates an "elimination gain factor" based on the power spectrum of the down-mixed signal, the vocal object's gain factor, and the transmitted harmonic magnitudes. This factor is applied to the background music signal.
- Smoothing Gain Calculation: To mitigate potential audio quality degradation caused by the direct elimination of harmonic components (which can create discontinuities), a smoothing gain factor is introduced. This factor averages nearby spectral components to create a smoother transition, preserving the overall quality of the background music.
Datasets
The paper uses 5 popular Korean songs for evaluation. These songs were composed of 4-6 audio objects (vocal, instruments) and sampled at 44.1 kHz with 16-bit quantization. No new datasets were introduced.
Key Insights and Ideas
- Harmonic Information is Key: The core insight is that by precisely identifying the harmonic structure of vocals (fundamental frequency and its multiples), we can target and remove them much more effectively than relying solely on general spectral suppression.
- Karaoke Mode is a Critical Use Case: The paper highlights that while SAOC is good for general audio manipulation, its failure in perfect vocal removal significantly hinders its utility for popular interactive features like Karaoke.
- Trade-off Between Removal and Quality: Directly removing spectral components can lead to artifacts. The proposed smoothing mechanism is crucial for balancing effective vocal removal with maintaining acceptable background music quality.
Key Results
- Objective Performance: The proposed method showed better results than both the standard SAOC and a Non-negative Matrix Factorization (NMF) based vocal separation method in objective tests (measured by Symmetric Kullback-Leibler distance).
- Subjective Performance: Subjective listening tests (MUSHRA) indicated that the proposed method achieved higher scores than the standard SAOC, particularly for content without a prominent chorus. The results suggest the method effectively removes vocal objects while preserving sound quality.
- Specific Content Impact: The performance was noted to be affected by the presence of chorus sections, as their characteristics can be similar to vocals. However, the proposed method still demonstrated superior performance overall.
Limitations and Future Work
- Bit-rate for Harmonic Information: While the paper focuses on improving vocal removal quality, it doesn't explicitly detail the bit-rate overhead for transmitting the harmonic information.
- Chorus Object Handling: The paper acknowledges that chorus objects can be affected by the harmonic elimination process due to their spectral similarity to vocals.
- Future Work: The authors plan to investigate methods for reducing the bit-rate required for transmitting the harmonic information while maintaining sound quality, which is crucial for practical network and broadcasting applications.
Summary updated