TL;DR
This paper introduces a novel pattern-based music generation system using Wasserstein Autoencoders (WAEs) and a new "pianoroll" representation called PRC. This approach allows for more expressive and coherent generation of four-bar musical patterns, overcoming limitations of previous methods in terms of polyphony and dynamic expressiveness.
What Problem Does This Paper Address?
Generating realistic and musically coherent MIDI music has been a long-standing challenge. Existing deep learning approaches often fall into two camps: those treating music as sequential events (struggling with polyphony) and those using fixed-size patterns (often described as pianorolls). While pattern-based methods are promising, previous "pianoroll" representations have limitations. They can be ambiguous with long notes, discard dynamic expressiveness by using binary values, and can lead to musically uninteresting or abrupt transitions when generating longer sequences. This paper aims to address these shortcomings by proposing a more expressive pattern representation and a robust generative model.
Methods
The core of the system lies in two key innovations:
- PRC Description: This is a new pattern representation that extends the traditional pianoroll. Instead of just indicating note presence, PRC uses two channels for each track: one for note velocity (representing loudness/intensity) and another for note duration. This allows for richer musical expression compared to binary pianorolls. The authors also note that this representation is lossless, meaning a MIDI pattern can be perfectly reconstructed from its PRC tensor.
- Wasserstein Autoencoders (WAEs): The authors employ WAEs as their generative learners. WAEs are a type of autoencoder that uses the Wasserstein distance to regularize the latent space. This helps to avoid the "blurriness" or repetitive artifacts sometimes seen in music generated by standard VAEs, leading to more coherent and less repetitive outputs. The encoder and decoder architectures are based on DCGAN, with a small latent vector size (D=3) found to be sufficient.
For generating longer sequences ("pseudo-songs"), the paper describes two methods: interpolation between two learned patterns in the latent space and "swirls" using complex-valued parametric functions. To ensure smooth transitions and musically pleasing progressions, they employ a bottleneck path problem solution (similar to Dijkstra's algorithm) to subsample the latent trajectories.
Datasets
The authors created two new datasets specifically for this study, composed by professional musicians:
- Dataset A: Contains 910 patterns in acid jazz, soul, and funk genres, featuring drums, bass, Rhodes piano, and Hammond organ.
- Dataset B: Contains 968 patterns in high-pop and progressive trance genres, with ten instruments including guitar, pads, strings, choir, and lead, in addition to the four from Dataset A.
All patterns were quantized to 1/32 resolution, limited to four octaves, and transposed to avoid tonality variations.
Key Insights and Ideas
- Expressiveness through Velocity and Duration: The PRC description's ability to encode both velocity and duration is crucial for generating musically expressive content, moving beyond simple note on/off.
- WAEs for Coherent Latent Spaces: WAEs prove effective in creating a well-behaved latent space for music generation, mitigating issues like note repetition and blurriness that can plague other autoencoder variants.
- Latent Space Trajectories for Song Structure: The methods for generating "pseudo-songs" by interpolating or creating "swirls" in the latent space offer a way to generate longer musical pieces with a degree of coherence.
- Optimizing Transitions is Key: The bottleneck path problem solution highlights that simply sampling equally spaced points in the latent space doesn't guarantee musically smooth transitions, emphasizing the need for explicit optimization of these transitions.
Key Results
The paper reports that generated pseudo-songs with PRC are consistent in terms of harmony, rhythm, melodic flow, and style, with accurate instrument interplay. Quantitatively, Table 1 shows that PRC outperforms the standard PR description in terms of precision, recall, and mean absolute errors for both velocity and duration on test sets for both datasets. For example, on Dataset A, PRC achieved a precision of 32.8% and recall of 53.8% with mean absolute errors of 24.2 (velocity) and 0.98 (duration), compared to PR's 5.5% precision, 42.2% recall, and errors of 31.9 (velocity) and 1.34 (duration).
An API was also developed to allow musicians to access the trained networks, enabling them to generate interpolations and download MIDI files for use in DAWs.
Limitations and Future Work
The authors acknowledge that while equally spaced points in the latent space don't always correspond to equally spaced reconstructions, their bottleneck path solution helps mitigate this. However, abrupt transitions or "flat" progressions can still occur.
Future work suggested includes incorporating conditioning mechanisms by labeling dataset patterns with structural categories (e.g., intro, verse, chorus) to allow for more controlled and structured music generation.
Summary updated