Score-informed syllable segmentation for a cappella singing voice with convolutional neural networks.

Jordi Pons, Rong Gong, Xavier SerraarXiv 2017 · OpenReview

TL;DR

This paper presents a novel method for automatically segmenting syllables in Jingju (Chinese opera) a cappella singing by combining convolutional neural networks (CNNs) for onset detection with score information in a Viterbi decoding framework. This score-informed approach significantly outperforms existing methods, addressing the unique challenges of Jingju singing like prolonged syllables and subtle phonetic transitions.

What Problem Does This Paper Address?

The goal is to automatically evaluate Jingju a cappella singing, which requires precise segmentation of singing phrases into individual syllables. Jingju singing presents unique challenges:

  • Prolonged Syllables: Some syllables can be held for extended periods (up to 20 seconds), making traditional onset detection methods prone to false positives due to minor amplitude fluctuations.
  • Subtle Transitions: Jingju singing involves complex phonetic structures and techniques like "zu yin" (block sound) that can create pauses within a syllable, further complicating onset detection.
  • Lack of Clear Peaks: Standard onset detection functions (ODFs) often struggle to identify clear, distinct peaks for syllable onsets in Jingju singing, especially when compared to the clear spectral changes in other musical genres.

Existing methods, often relying on thresholding or peak-picking ODFs, are insufficient for this task. The paper aims to improve syllable segmentation by leveraging both the audio signal and the musical score.

Methods

The proposed method has two main components:

  1. Improved Syllable Onset Detection Function (ODF) Estimation with CNNs:
    • Novel CNN Architecture: The authors propose CNN architectures inspired by existing work but with a focus on using diverse filter shapes in the first convolutional layer. This allows the network to efficiently capture different time-frequency scales, which is crucial for detecting both abrupt (consonant-based) and gradual (vowel-based) syllable onsets.
  2. Temporal and Timbral Architectures: They explore separate CNNs designed to capture temporal dynamics and timbral features, respectively.
  3. Late Fusion: Predictions from temporal and timbral CNNs are combined using a late-fusion approach (multiplying their output probabilities) to leverage complementary information.
  4. Parameter Efficiency: The proposed architectures achieve state-of-the-art performance with fewer parameters than previous methods, suggesting greater efficiency and reduced overfitting.
  5. Score-Informed Viterbi Decoding:
    • A Priori Duration Model: Syllable durations are extracted from the musical score. This information is used to build a Gaussian-based a priori duration model, which estimates the probability of a syllable boundary occurring based on its expected duration.
  6. Hidden Markov Model (HMM): A Hidden Markov Model is constructed where hidden states represent candidate onset positions.
  7. Viterbi Algorithm: The Viterbi algorithm is employed to find the most likely sequence of syllable boundaries. Crucially, the state transition probabilities in the Viterbi algorithm are informed by the a priori duration model derived from the score. This allows the algorithm to favor boundary sequences that align with the score's rhythmic and durational information, effectively guiding the segmentation process and overcoming the limitations of ODF thresholding.

Datasets

  • Jingju A Cappella Singing Dataset: A newly curated dataset for this study, containing 39 interpretations of 31 unique arias sung by 11 Jingju singers, focusing on "dan" (female) and "laosheng" (old man) role-types. The dataset includes audio, pre-segmented phrases, and manually annotated syllable onset and offset ground truth. Syllable durations are transcribed from music scores.
  • Comparison Dataset: Kruspe's dataset of commercial pop songs is used to highlight the significant differences in syllable duration variability between Jingju singing and Western pop music.

Key Insights and Ideas

  • Score as a Constraint: The core idea is that musical scores, often overlooked in pure audio-based segmentation, contain invaluable temporal information that can significantly improve accuracy, especially for challenging genres like Jingju.
  • Multi-Scale Feature Extraction: The success of using diverse filter shapes in the first CNN layer highlights the importance of capturing features at various time-frequency scales for robust onset detection in singing.
  • Complementary Architectures: Fusing temporal and timbral CNNs demonstrates that specialized networks can learn complementary representations, leading to better overall performance.
  • Viterbi for Score Integration: Using the Viterbi algorithm with score-derived duration probabilities is an elegant way to incorporate prior musical knowledge into the segmentation process, moving beyond simple peak detection.

Key Results

  • State-of-the-Art Performance: The proposed score-informed method significantly outperforms existing state-of-the-art methods, including traditional approaches (Obin et al.) and deep learning-based onset detection (Schlüter et al.).
  • F-measure Improvement: The best performing model (Late-fusion CNN with 32 filters in the second layer) achieved an F-measure of 86.37%, compared to 81.93% for Schlüter et al. and 40.61% for Obin et al.
  • Parameter Efficiency: The proposed CNN architectures are more parameter-efficient, achieving higher performance with fewer parameters.

Limitations and Future Work

  • Score-Performance Mismatch: The method struggles when there are significant deviations between the sung performance and the provided score (e.g., syllable insertions/deletions not reflected in the score).
  • Ambiguous Transitions: Transitions between similar phonetic elements (e.g., vowel-to-vowel) remain challenging.
  • Future Directions:
    • Incorporating More Domain Knowledge: Suggests adding knowledge about padding characters or the expectation of prolonged final syllables to the decoding process.
  • RNNs for ODF: Proposes using Recurrent Neural Networks (RNNs) to further improve ODF estimation, as they have shown promise in similar tasks.
  • "Onset Regions" vs. "Onset Time Positions": For ambiguous transitions, the authors suggest detecting "onset regions" rather than precise time points, as these transitions often manifest as gradual spectral changes.

Summary updated

Abstract

This paper introduces a new score-informed method for the segmentation of jingju a cappella singing phrase into syllables. The proposed method estimates the most likely sequence of syllable boundaries given the estimated syllable onset detection function (ODF) and its score. Throughout the paper, we first examine the jingju syllables structure and propose a definition of the term "syllable onset". Then, we identify which are the challenges that jingju a cappella singing poses. Further, we investigate how to improve the syllable ODF estimation with convolutional neural networks (CNNs). We propose a novel CNN architecture that allows to efficiently capture different time-frequency scales for estimating syllable onsets. In addition, we propose using a score-informed Viterbi algorithm -instead of thresholding the onset function-, because the available musical knowledge we have (the score) can be used to inform the Viterbi algorithm in order to overcome the identified challenges. The proposed method outperforms the state-of-the-art in syllable segmentation for jingju a cappella singing. We further provide an analysis of the segmentation errors which points possible research directions.
BibTeX
@misc{pons2017score,
  author = {Jordi Pons and Rong Gong and Xavier Serra},
  title = {{Score-informed syllable segmentation for a cappella singing voice with convolutional neural networks.}},
  year = {2017},
  eprint = {1707.03544},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.03544},
}
Figure