Abstract
Vocal pitch extraction is a critical task in Music Information Retrieval (MIR), yet state-of-the-art deep learning models often fail when applied to Indian Art Music (IAM) due to a lack of annotated data and the unique complexities of the genre. This paper introduces the Saraga-Carnatic-Melody-Synth (SCMS) dataset, a large-scale, open-source resource for Carnatic vocal pitch extraction. By developing a novel "reverse-engineering" synthesis pipeline, the authors generated over 1,200 minutes of high-quality training data. Models trained on this repertoire-specific data achieved a significant performance leap, reaching ~91% accuracy compared to ~68% for models trained on Western music. These improvements directly translate to better performance in downstream tasks like automated melodic pattern discovery.
Introduction
While deep learning (DL) has revolutionized pitch extraction for Western Pop and Rock, its application to Carnatic music faces significant hurdles. The primary challenge is data scarcity: DL models require thousands of hours of ground-truth pitch data, which is difficult and expensive to annotate manually for the complex, sliding melodies of Carnatic music.
Furthermore, the musical complexity of the genre introduces specific technical difficulties. Carnatic music is characterized by gamakas (intricate microtonal ornaments) and a heterophonic texture where a violin often "shadows" the vocalist. Standard models frequently mistake the violin for the lead singer or smooth out essential ornaments, losing the musical identity of the performance.
The SCMS Dataset and Generation Pipeline
To overcome the lack of annotated data, the researchers proposed a "reverse-engineering" approach to create the Saraga-Carnatic-Melody-Synth (SCMS) dataset. Instead of attempting to annotate existing complex audio, they synthesized new audio from known pitch tracks, ensuring a mathematically perfect relationship between the audio and its "ground-truth" annotation.
The pipeline consists of several key stages:
- Leakage Removal: U-Net-based source separation was used to clean "close-mic" recordings from the Saraga dataset, removing instrumental bleed from the vocal tracks.
- Analysis/Synthesis: A preliminary pitch curve was extracted using a heuristic tracker (PMM), followed by a Harmonic plus Residual (HpR) model to resynthesize the vocal.
- Traditional Remixing: To ensure realism, the synthetic vocals were mixed with a synthesized tanpura (drone) and real instrumental tracks (violin and percussion). Specific constraints were applied to maintain a traditional concert balance, such as ensuring the vocal remained the predominant source.
The resulting SCMS dataset contains approximately 1,235 minutes of audio across 2,460 samples, providing a robust benchmark for training and evaluation.
Experimental Results
The authors evaluated the Frequency-Time Attention Network (FTA-Net) architecture, comparing a model trained on Western music (FTA-W) against one trained on the new SCMS data (FTA-C).
Performance Gains
The domain gap between Western and Carnatic music proved substantial. FTA-W achieved only 67.72% Overall Accuracy (OA) on Carnatic music, while the repertoire-specific FTA-C reached 90.99% OA. FTA-C also significantly outperformed Melodia, the long-standing heuristic standard, which scored 77.07%.
Addressing Repertoire Nuances
The training on SCMS allowed the model to overcome specific genre-related challenges:
- Violin Interference: FTA-C learned to ignore the "heterophonic lag" of the violin, a common failure point for Western models.
- Gamakas: Rather than treating rapid pitch oscillations as noise, the SCMS-trained model recognized them as essential melodic information.
- Cross-Tradition Generalization: FTA-C performed better on Hindustani music than the Western-trained model, suggesting shared acoustic logic across Indian Art Music traditions.
Impact on Melodic Analysis
The improvements in pitch extraction directly benefited downstream musicological tasks. In a test for finding repeated melodic motifs (sañcāras), the new model achieved an F1-score of 0.494, nearly doubling the performance of standard pitch extraction methods.
Conclusion and Future Work
This work demonstrates that repertoire-specific data is essential for accurate MIR in non-Western traditions. While the current synthesis method focuses primarily on vowels and can miss inharmonic content like consonants, it provides a powerful foundation for melodic analysis. Future research will focus on incorporating more advanced synthesis techniques to include consonants and expanding the methodology to other traditions, such as Hindustani music.
Summary updated