TL;DR
This paper tackles the challenge of automatically matching a snippet of jingju opera singing to its corresponding musical score. By combining phonetic and duration information, the authors propose a novel approach that outperforms traditional melody-only methods, particularly for jingju music where melodic contours can be ambiguous. Their work demonstrates the effectiveness of Convolutional Neural Networks (CNNs) for acoustic modeling and Hidden Semi-Markov Models (HSMMs) for duration modeling in this specialized domain.
What Problem Does This Paper Address?
The ultimate goal is to automatically evaluate a student's jingju opera singing against a tutor's performance. A crucial first step in this process is "audio to score matching" – finding the correct musical score for a given singing phrase. Traditional methods for similar tasks, like Query-by-Singing/Humming (QBSH), often rely heavily on melodic information (pitch contours). However, the authors argue that for jingju music, this approach is insufficient. Jingju opera has distinct melodic modes (Xipi and Erhuang), and within these modes, different arias can share similar basic melodic contours. This similarity can lead to ambiguous matching if only melody is considered. To overcome this, the paper proposes a system that leverages both phonetic (lyrics) and duration information from the musical score, in addition to acoustic features extracted from the audio.
Methods
The proposed approach involves several key components:
- Acoustic Models: To extract phonetic information from the singing audio, the authors investigate three types of acoustic models:
- Gaussian Mixture Models (GMMs): A traditional baseline.
- Deep Neural Networks (DNNs): A more complex neural network approach.
- Convolutional Neural Networks (CNNs): A novel approach for this task, specifically designed to capture temporal features and timbre representations. The CNN architecture uses convolutional layers with varying filter sizes and a max-pooling layer, followed by a softmax output layer.
- Matching Network: This component uses Hidden Markov Models (HMMs) or Hidden Semi-Markov Models (HSMMs) to represent each possible lyric path in the musical scores. Each path is a left-to-right state chain corresponding to the phoneme sequence of the lyrics. The network outputs a posterior probability for each candidate score, which is used for ranking.
- Phonetic Duration Distributions: To incorporate duration information, the authors estimate phonetic duration distributions. This involves:
- Segmenting audio and scores into phonetic units.
- Calculating duration centroids for each phoneme class from the audio dataset.
- Distributing syllable durations from the score into phonetic durations based on these centroids.
- Normalizing phrase durations to match the query audio's duration.
- Modeling these durations using Gaussian distributions.
- Duration Models: Two models are explored for incorporating phonetic duration information into the matching process:
- Hidden Semi-Markov Model (HSMM): This model explicitly incorporates the phonetic duration distributions into the state transition probabilities.
- Post-processor Duration Model: This approach uses a standard HMM and then augments the log posterior probability with log duration probabilities.
Datasets
- Jingju A Cappella Singing Dataset: This dataset consists of two parts:
- Audio: Recordings of jingju opera singing from two role-types: dan (young woman) and laosheng (old man). The audio is annotated with phrase, syllable, and phoneme boundaries, using Chinese characters, pinyin, and X-SAMPA notations.
- Score: Musical scores for jingju opera a cappella singing, typed in stave notation with lyrics. These scores provide phonetic and duration information for each singing syllable.
The audio dataset is split into training, development (dev), and test sets. The score dataset is used for constructing the matching network and for parameter optimization and testing.
Key Insights and Ideas
- Melody is Not Enough for Jingju: The core insight is that relying solely on melodic information for audio-to-score matching in jingju opera is problematic due to the inherent similarity of melodic contours within modes. This justifies the need for complementary information like phonetics and duration.
- Phonetics as a Discriminator: Lyrics and their phonetic representations offer a more distinctive cue for matching jingju singing phrases, as stories and lyrics tend to vary even within the same melodic mode.
- CNNs Shine in Small Data: The study highlights the effectiveness of CNNs for acoustic modeling, particularly in scenarios with limited training data. Their ability to learn timbre representations and their inherent parameter sharing make them robust against overfitting.
- HSMMs Excel at Duration Modeling: The research demonstrates that HSMMs, which explicitly model duration distributions, outperform both baseline HMMs and the post-processor duration model for this task. This suggests that explicit duration modeling is crucial for accurate matching.
- Inconsistency in Post-Processor Performance: The post-processor duration model's performance was not as strong as expected, potentially due to the difference in matching unit size (phrases vs. keywords) and the larger vowel length standard deviation in jingju singing compared to previous studies.
Key Results
- Acoustic Model Performance: CNNs achieved the highest classification accuracy (0.484 for dan, 0.432 for laosheng) among the investigated acoustic models (CNN, DNN, GMM) on the test set.
- Duration Model Performance: HSMMs significantly outperformed the baseline HMMs, improving Mean Reciprocal Rank (MRR) by 13.2% for dan and 15.1% for laosheng. The post-processor duration model did not show significant improvement over the baseline.
- Overall Matching: The combination of CNN acoustic models and HSMM duration modeling yielded the best matching performance.
Limitations and Future Work
- Small Dataset: The authors acknowledge that the available dataset is relatively small, which can be a challenge for training deep learning models like DNNs.
- Future Work:
- LSTM RNNs: Exploring Long Short-Term Memory (LSTM) Recurrent Neural Networks (RNNs) for acoustic modeling, which have shown promise in speech recognition but require larger datasets.
- Deeper CNNs: Investigating deeper CNN architectures, which would also necessitate more training data.
- Data Augmentation: Planning to collect more jingju opera recordings and employ data augmentation techniques to improve acoustic model capabilities.
- Melody Fusion: Investigating methods to fuse melodic information with phonetic and duration information to further enhance matching performance.
Summary updated