Musical Speech: A Transformer-based Composition Tool

Jason d'Eon, Sri Harsha Dumpala, Chandramouli Shama Sastry, Dani Oore, Sageev Oore

Abstract

In this paper, we propose a new compositional tool that will generate a musical outline of speech recorded/provided by the user for use as a musical building block in their compositions. The tool allows any user to use their own speech to generate musical material, while still being able to hear the direct connection between their recorded speech and the resulting music. The tool is built on our proposed pipeline. This pipeline begins with speech-based signal processing, after which some simple musical heuristics are applied, and finally these pre-processed signals are passed through Transformer models trained on new musical tasks. We illustrate the effectiveness of our pipeline -- which does not require a paired dataset for training -- through examples of music created by musicians making use of our tool.

Introduction

In contemporary and avant-garde music, transcribing the natural prosody of speech into musical notation is a well-established technique. However, manually capturing the rhythmic and melodic nuances of a spoken phrase is a labor-intensive process requiring significant musical expertise. While generative AI has advanced rapidly in music creation, there is a lack of tools that allow composers to use specific inputs, like their own voice, as a structural foundation for their work.

The primary challenge in automating this process is the absence of "paired data"—large datasets of speech aligned with corresponding musical transcriptions. This paper introduces a pipeline that sidesteps this requirement by using unsupervised learning and signal processing to transform raw speech into MIDI "musical outlines."

The Proposed Pipeline

The system transforms raw audio into a structured musical format through a multi-step pipeline involving feature extraction, sparsification, and Transformer-based refinement.

The proposed pipeline: (1) raw audio, (2) pitch extraction, (3) sparsification followed by (4) gap-filling, and (5) a parallel denoising path.

1. Feature Extraction and Sparsification

The process begins by extracting the fundamental frequency ($F_0$) and formants ($F_1, F_2, F_3$) from the speech signal. Because raw speech is "busy" and contains non-musical noise, the data must be "sparsified." The researchers employ two main techniques to identify perceptually significant moments:

  • Heuristic-based: Selecting pitches only during high-intensity (loud) moments.
  • Syllable-nuclei-based: Identifying the peaks of syllables to find the most important melodic anchors.
Visualization of the signal processing steps: (a) spectrogram, (b) raw pitch, (c) sparsified pitch, and (d) the final musical output.

2. Transformer-based Musical Refinement

The core of the system consists of two Transformer models trained on the MAESTRO dataset (approximately 200 hours of classical piano). Crucially, these models are not trained on speech, but on musical tasks that allow them to act as a "musical spell-checker":

  • Gap-filling Transformer: This model takes the sparse notes and "fills in the blanks" with notes that follow learned musical patterns from classical piano.
  • Denoising Transformer: This model takes the highly chromatic, non-traditional intervals of speech and "nudges" them toward standard musical intervals (e.g., perfect fifths or octaves).

Results and Evaluation

The effectiveness of the tool was evaluated both quantitatively and qualitatively:

  • Quantitative Analysis: Using the LibriSpeech dataset, the researchers demonstrated that the pipeline successfully shifted the interval distribution of speech. While raw speech is highly chromatic (moving mostly in 1-semitone steps), the processed output showed a distribution closer to professional music, introducing more "musical" intervals.
  • Qualitative Analysis: Professional musicians used the tool to create original compositions. In one example, "Moo-cow," a composer used the generated MIDI to drive various synthesizers, creating a piece where every instrumental note was derived from a 10-second speech clip.

Conclusion and Future Work

The "Musical Speech" tool provides a novel way for composers to bridge the gap between human voice and instrumental music. While currently limited to monophonic melodies and requiring a "human in the loop" for final orchestration, it offers a powerful starting point for creative exploration. Future work will focus on generating polyphonic harmonies and refining the rhythmic mapping of speech to the rigid grid of Western music.

Reference: d’Eon, J., Dumpala, S. H., Sastry, C. S., Oore, D., & Oore, S. (2021). Musical Speech: A Transformer-based Composition Tool. NeurIPS 2020 Competition and Demonstration Track.

Summary updated

Abstract

In this paper, we propose a new compositional tool that will generate a musical outline of speech recorded/provided by the user for use as a musical building block in their compositions. The tool allows any user to use their own speech to generate musical material, while still being able to hear the direct connection between their recorded speech and the resulting music. The tool is built on our proposed pipeline. This pipeline begins with speech-based signal processing, after which some simple musical heuristics are applied, and finally these pre-processed signals are passed through Transformer models trained on new musical tasks. We illustrate the effectiveness of our pipeline -- which does not require a paired dataset for training -- through examples of music created by musicians making use of our tool.
BibTeX
@inproceedings{deon2020musical,
  author = {Jason d'Eon and Sri Harsha Dumpala and Chandramouli Shama Sastry and Dani Oore and Sageev Oore},
  title = {{Musical Speech: A Transformer-based Composition Tool}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=_6yBwH5PgFz},
}
Figure