In the world of computer music, there is a long-standing divide between the "what" and the "how." Symbolic representations like MIDI are excellent at capturing the "what"—the specific notes, their timing, and their duration. However, they are notoriously poor at capturing the "how"—the subtle vibrato of a violin, the aggressive distortion of a guitar, or the rhythmic "swing" of a human drummer.
Recent research focuses on bridging this gap, moving beyond robotic MIDI playback toward systems that can synthesize high-fidelity audio with the expressive nuances of a live performance.
The Symbolic-Acoustic Gap
Traditional music synthesis often relies on a two-step process: first, a model generates a symbolic score (like MIDI), and then a synthesizer turns that score into sound. As Ben Maman and colleagues point out in Performance Conditioning for Diffusion-Based Multi-Instrument Music Synthesis, this often results in "flat" or "robotic" audio. MIDI simply lacks the bandwidth to describe a "sonic signature"—the unique combination of instrument timbre, performer style, and room acoustics.
This limitation is particularly evident in genres like black metal or math rock, where the "sound-mass"—the texture and spatiality of distortion—is more important than the underlying melody. Zack Zukowski and CJ Carr demonstrated in Generating Black Metal and Math Rock that to capture these genres, models must operate directly in the raw audio domain using architectures like SampleRNN, rather than relying on symbolic notation.
Performance Conditioning and Expressive Attributes
To give AI-generated music a human touch, researchers are developing "performance conditioning" techniques. The goal is to inject stylistic information into the generative process.
One approach is to use richer datasets that include expressive metadata. Chris Donahue introduced the NES Music Database, which includes not just notes but also the volume and tone shifts used by composers. Similarly, Adarsh Kumar and his team developed ShredGP, a model that generates guitar tablature. Unlike MIDI, tablature captures the "how" of guitar playing—bends, slides, and specific fingerings—allowing the model to imitate the distinct styles of legends like Jimi Hendrix.
In the audio domain, Ben Maman uses FiLM (Feature-wise Linear Modulation) layers to inject "performance IDs" into a diffusion model. This allows a user to specify a target "sonic signature," ensuring that a generated orchestral piece maintains a consistent instrumental texture rather than suffering from "instrument drift."
Intermediate Representations as a Bridge
Generating high-fidelity audio directly from scratch is computationally expensive and difficult to control. To solve this, many researchers use intermediate representations as a bridge between high-level intent and low-level sound.
In Foley Music, Chuang Gan and his team generate music from silent videos of musicians. Instead of mapping pixels directly to audio, they first extract human body keypoints (the "motion") and convert them into MIDI events. This intermediate symbolic step provides the structure, which is then synthesized into realistic audio that matches the physical mechanics of the instrument.
A similar logic applies to CycleDRUMS by Giorgio Barnabò and colleagues. They treat drum arrangement as an image-to-image translation task on mel-spectrograms. By using a CycleGAN, they can generate realistic drum tracks that sync with bass lines without needing perfectly paired training data, effectively "translating" the rhythmic feel of the bass into the acoustic texture of the drums.
Scaling to Raw Audio
The most ambitious approach to this problem is to abandon symbolic representations entirely and train massive models on raw audio. Jukebox, developed by Prafulla Dhariwal and the OpenAI team, uses hierarchical VQ-VAEs to compress audio into discrete codes. This allows a Transformer to learn long-range musical structure while the decoder handles the complex textures of singing and instrumentation.
While Jukebox is powerful, it is "offline." Newer models like VidMuse by Zeyue Tian and LORIS by Jiashuo Yu focus on "visual rhythm." They use latent diffusion to generate audio that is both high-fidelity and precisely synchronized with video events, such as the movements of a figure skater.
By combining the structural logic of symbolic music with the textural richness of raw audio synthesis, these researchers are moving toward AI that doesn't just play the right notes, but plays them with the right "soul."