Introduction
Musicians have long looked to human speech for inspiration—from the rhythmic cadences of a preacher to the melodic lilt of a storyteller. However, manually transcribing speech into music is an exhausting task that requires an elite "ear" and significant time.
Existing automated tools often struggle because speech and music are fundamentally different. Speech is continuous and highly chromatic, utilizing microtonal frequencies that fall between standard piano keys. Music, conversely, is discrete and follows specific harmonic and rhythmic rules. When one simply maps speech pitch to MIDI notes, the result is often a chaotic barrage of sound that feels un-musical.
To bridge this gap, researchers have developed Musical Speech, a compositional tool that uses Transformer models to translate the prosody and rhythm of spoken word into structured musical melodies. By treating speech features as "noisy" or "sparse" musical constraints, the system allows creators to turn any vocal recording into a musical building block without requiring a specialized paired dataset for training.
The Musical Speech Pipeline
The authors propose a multi-stage pipeline that moves from raw audio to a polished musical "outline."
1. Feature Extraction
The system first extracts the fundamental frequency ($F_0$) and formants ($F_1, F_2, F_3$) from the speech recording. While $F_0$ represents the main pitch of the speaker's voice, formants provide alternative melodic paths based on the resonances of the vocal tract, offering users different "melodic interpretations" of the same sentence.
2. Sparsification
To prevent note-overload and chaotic outputs, the system uses two techniques to pick only the most "important" moments from the continuous speech signal:
- Heuristic-based: Selecting notes based on loudness thresholds.
- Syllable-nuclei-based: Using signal processing to find the "peaks" of syllables, mimicking how humans naturally perceive the "beat" or rhythm of a spoken sentence.
3. The Transformer "Brain"
The core innovation lies in how the system regularizes this sparse data. Instead of training on speech-to-music pairs—which do not exist in large enough quantities—the researchers trained two types of Transformers purely on music data (the MAESTRO V2.0.0 dataset, containing ~200 hours of professional classical piano performances):
- Gap-filling Transformer: Similar to how language models like BERT predict missing words in a sentence, this model takes the sparse notes from the speech and "fills in the blanks" with notes that make musical sense.
- Denoising Transformer: This model takes the raw, "messy" pitch sequence of speech and "corrects" it toward the nearest musical structures it learned during training.
Key Insights and Results
The most powerful idea in this research is using a Transformer as a "musical language model." By training on MAESTRO, the model learns the "grammar" of music—for example, that a perfect 5th interval is more likely than a dissonant tritone. When fed speech, it acts as a filter that forces the speech to "speak" in a musical tongue. By framing the problem as gap-filling or denoising, the authors successfully sidestepped the need for a parallel dataset of speech and its musical equivalent.
Quantitatively, the researchers analyzed the intervals (the distance between notes) of the generated outputs. Raw speech is highly chromatic, consisting mostly of 1-semitone jumps. The Transformer outputs successfully shifted this distribution to sit between raw speech and classical music, increasing the frequency of musically pleasing intervals like 4ths, 5ths, and octaves.
Qualitatively, the paper showcases several compositions created by professional musicians using the tool. In one example, a musician used the system to generate every single MIDI instrument in a 30-second track based on a 10-second speech clip, proving the tool's utility as a powerful "material generator" for actual art.
Limitations and Future Work
Currently, the tool only generates single-note (monophonic) melodies and cannot yet suggest harmonies or chords to accompany the speech. Furthermore, while the system adjusts timing, mapping the fluid, "off-grid" rhythm of human speech to a rigid musical tempo remains a complex area for improvement. Finally, while the tool generates raw MIDI material quickly, it still relies on a "human in the loop"—a skilled musician to orchestrate, mix, and refine the output into a finished piece of music.
Reference: d’Eon, J., Dumpala, S. H., Sastry, C. S., Oore, D., & Oore, S. (2021). Musical Speech: A Transformer-based Composition Tool. Proceedings of Machine Learning Research 133:253–274, NeurIPS 2020 Competition and Demonstration Track.
Summary updated