Towards Mixed-initiative generation of multi-channel sequential structure

Cheng-Zhi Anna Huang, Sherol Chen, Mark J. Nelson, Douglas EckWorkshops @ ICLR 2018

The future of artificial intelligence in creative fields isn't just about "push-button" generation—it's about building a mixed-initiative partnership where humans and machines co-create. While deep generative models have become remarkably adept at imitating styles, they frequently struggle with maintaining long-term structure and accommodating human agency.

Most sequence models generate content chronologically, from the first note to the last. However, human creators rarely work this way; they scribble a motif here, draft a transition there, and constantly revisit earlier choices. Fully autonomous AI often ignores these subjective, non-linear workflows. To bridge this gap, researchers have proposed a framework where AI acts as a collaborative partner, filling in the gaps of a human's creative process, specifically within the complex domain of polyphonic music.

The Mixed-Initiative Spectrum

The traditional view of creative software places the human as the sole creator and the computer as a mere tool. Conversely, fully autonomous generative AI places the computer as the creator and the human as the audience.

The mixed-initiative spectrum, positioning CoCoNet as a collaborative partner between human-driven tools and fully autonomous AI.

This research frames interaction along a spectrum, aiming for the middle ground: Mixed Initiative. In this paradigm, the human provides the initiative by setting constraints or drafting initial ideas, and the AI takes the initiative by harmonizing and expanding upon those inputs across all channels.

CoCoNet: The AI "Colleague"

To realize this mixed-initiative approach, the authors utilize CoCoNet (Counterpoint by Convolutional Neural Network). Rather than predicting the next note in a sequence, CoCoNet is designed for "partial score completion."

  • Architecture & Representation: CoCoNet is a deep Convolutional Neural Network (CNN) trained similarly to an orderless Neural Autoregressive Distribution Estimator (NADE). It treats music as a multi-channel 2D binary matrix, where axes represent time and pitch, and channels represent different instruments or voices.
  • In-filling via Gibbs Sampling: The model uses blocked Gibbs sampling to generate music. It starts with a partial score provided by the human and iteratively rewrites the "empty" or "uncertain" parts until it reaches a musically coherent state.

By allowing users to input notes anywhere in a timeline, the AI transforms from a sequential "tape recorder" into a non-linear "canvas." It acts less like a tool executing commands and more like a colleague. If a user introduces a seemingly "wrong" note, the AI doesn't reject it; instead, it attempts to find a new harmonic context that makes the note work.

Change Propagation: From Surface to Deep Structure

A critical insight of this framework is how it handles structural edits. In music, a small "surface" change—like moving two notes in a melody—can necessitate "deep" structural changes, such as altering the underlying harmony or the key of the entire piece.

Diagram illustrating how surface-level changes in a melody propagate to deep structural changes in harmony.

CoCoNet handles this propagation automatically. When a user modifies a melody, the system regenerates the accompanying voices to ensure the new melody integrates seamlessly into the harmonic closure, allowing creators to quickly explore complex "what if" scenarios.

Case Study: Rewriting Beethoven

The researchers demonstrated CoCoNet's capabilities using Beethoven’s "Ode to Joy." When provided with the famous melody, the model successfully generated a coherent four-part harmonization. More importantly, when a user interactively modified the end of the melody to add a "twist," the model dynamically regenerated the other voices to support the new melodic direction. Users could even implicitly control the "abruptness" of these changes by limiting how much of the existing score the machine was allowed to modify during the sampling process.

Future Directions

While CoCoNet successfully demonstrates mixed-initiative co-creation in music, there are still hurdles to overcome:

  • High-Level Control: Currently, users interact primarily with "surface" elements (individual notes). Future iterations need "knobs" for high-level intent, allowing users to issue directives like "make this more minor" or "increase the tension."
  • Domain Expansion: The authors envision applying this multi-channel sequential approach to other complex structures, such as screenplays or video game narratives, potentially incorporating grammars and rulesets to give authors more narrative control.
  • User Experience (UX): Ultimately, the design of the interface—specifically how the human and AI "take turns"—is just as critical to true co-creativity as the underlying neural network architecture.

Summary updated

BibTeX
@inproceedings{huang2018towards,
  author = {Cheng-Zhi Anna Huang and Sherol Chen and Mark J. Nelson and Douglas Eck},
  title = {{Towards Mixed-initiative generation of multi-channel sequential structure}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
}
Figure