Relative Pitch and Duration Representations for Symbolic Music

48 concepts35 papers3 proposals

When a human listens to a melody, they do not perceive notes as absolute physical frequencies or isolated MIDI numbers. If you transpose "Happy Birthday" from C major to G major, or play it twice as slow, a human listener instantly recognizes it as the same song. Yet, to a naive machine learning model operating on absolute values, every single data point has changed.

Representing music via absolute coordinates—such as raw MIDI pitches (0–127) or absolute millisecond timestamps—forces models to learn the same underlying musical patterns repeatedly for every key, tempo, and instrument range. This absolute framing creates artificial distances between structurally identical sequences, severely limiting a model's ability to generalize.

To bridge this gap, a prominent research direction focuses on relative interval encoding. By transforming absolute musical data into relative relationships—pitch intervals and duration ratios—we can bake transposition and tempo invariance directly into the data representation and model architecture.


The Core Intuition: Shifting the Coordinate System

To understand why relative encoding works, imagine a ladder. An absolute representation describes the ladder by recording the exact height of every rung above the ground. If you move the ladder to a platform, every single height measurement changes. A relative representation, instead, records the distance between the rungs. No matter where you place the ladder, the description remains identical.

In symbolic music, this shift applies to both dimensions: pitch (vertical) and time (horizontal).

Absolute MIDI:  [60,  62,  64,  65]  --> Transposed (+2): [62,  64,  66,  67]
Relative Pitch:     [  +2,  +2,  +1]  --> Transposed (+2):     [  +2,  +2,  +1]

As Eric Brochu et al. demonstrated in early work on multi-modal music retrieval, models relying on absolute values fail to match human-generated queries because humans naturally fluctuate in pitch and key. By modeling melodies as first-order Markov chains of intervals rather than absolute pitches, the representation becomes inherently invariant to transpositions, allowing robust matching regardless of the performance's key.


Achieving Transposition Invariance in Pitch

Researchers have developed several distinct strategies to encode relative pitch, depending on whether the invariance is handled in the data representation or directly within the neural network architecture.

1. Interval-Based and Scale-Degree Representations

The most direct way to achieve transposition invariance is to alter the input tokens themselves:

  • Consecutive Intervals: Instead of absolute notes, the input tracks the step-by-step distance from the previous note. Dominik Hörnel utilized interval-based coding in early neural networks to capture structural symmetries like octave equivalence, reducing the model's learning burden. Similarly, Aggelos Pikrakis et al. leveraged note-to-note intervals to spot melodies within raw audio across different tunings and keys.
  • Root-Relative (Scale Degrees): Rather than tracking differences between adjacent notes, pitches can be encoded relative to a global tonic or local chord root. Eliya Nachmani et al. showed that encoding pitches as intervals relative to the scale's root note allows low-capacity models to learn universal musical patterns from very small datasets. Taking this further, Jingyue Huang et al. introduced a "functional representation" that maps notes to scale degrees and chords to Roman numerals. This preserves the functional role of each note within its tonal context, which is critical for capturing emotional valence.

2. Architectural Inductive Biases

Instead of changing the data format, we can design neural network layers that force a relative interpretation of absolute inputs.

  • Pitch-Axis Convolutions: Standard convolutional layers are shift-invariant. By treating the pitch space as a spatial dimension, convolutional filters can slide across pitches to detect chords and intervals regardless of their absolute position. Harsh Verma et al. applied this by convolving along a pitch-based axis to achieve translation invariance for harmonic structures. To handle the circular nature of musical keys (where a transposition can wrap around octaves), Furkan Yesiler et al. employed circular convolutions on doubled pitch-class bins followed by max-pooling, ensuring consistent embeddings across all 12 keys.
  • Relative History Windows: In autoregressive generation, John Thickstun et al. proposed centering the model's history window dynamically on the current pitch. This architectural choice transforms absolute notes into relative intervals during prediction, allowing the model to generalize reusable harmonic patterns across different registers.
  • Graph Neural Networks (GNNs): Music can be naturally modeled as a graph where notes are nodes. Emmanouil Karystinaios et al. designed a perception-inspired graph convolution block that explicitly encodes relative pitch and timing differences as edge features. This grants the network architectural transposition invariance without requiring extensive data augmentation.

Achieving Tempo Invariance in Time

Time poses a parallel challenge to pitch. If a performer slows down, the absolute duration of every note increases, yet the rhythmic structure remains unchanged. As Jamie Forth et al. noted, cognitive models must process time based on intervals between events rather than a linear timeline to achieve tempo invariance.

Duration Ratios and Inter-Onset Intervals

To capture rhythm independently of speed, researchers use proportional timing metrics. The authors of A Cross-Scape Plot Representation for Visualizing Symbolic Melodic Similarity, alongside Martin Rohrmeier, demonstrated that transforming symbolic data into logarithmic inter-onset interval (IOI) ratios allows robust melodic matching. Because a ratio divides one duration by another, the global tempo factor cancels out mathematically:

$$\text{Ratio} = \frac{\text{Duration}_{\text{Note 2}}}{\text{Duration}_{\text{Note 1}}} = \frac{c \cdot t_2}{c \cdot t_1} = \frac{t_2}{t_1}$$

For generative tasks, Taizan Suzuki et al. applied this concept to musical expression, encoding expressive parameters like tempo and volume as ratios relative to a parent segment's values. This allows expressive nuances to be seamlessly transferred across pieces of varying lengths.

Quantization and Hierarchical Structure

Continuous timing data from human performances is often noisy. To prevent minor timing variations from disrupting pattern recognition, continuous durations can be mapped into relative categories. Francesco Foscarin et al. achieved tempo invariance by using unsupervised 1D k-means clustering to group continuous note durations into discrete relative groups, stripping away millisecond-level noise to focus entirely on rhythmic relationships.

Furthermore, the underlying structure of time can be encoded hierarchically. Ziyu Wang et al. showed that utilizing a binary temporal encoding inside a tree-structured VAE creates a fractal latent structure. This structure naturally maintains translation invariance for duration, capturing the proportional essence of musical rhythm.


Why It Works: Downstream Benefits

Shifting from absolute to relative representations provides several profound advantages for music AI models:

  1. Data Efficiency: In an absolute paradigm, a model must see a musical motif played in all 12 keys and at various tempos to generalize. Relative encoding collapses these variations into a single canonical representation, allowing models like those by Eliya Nachmani et al. to achieve high coherence even when trained on tiny datasets.
  2. Robustness to Human Noise: In Query-by-Humming and version identification systems, user inputs are notoriously imperfect. Systems designed by Naziba Mostafa and Erdem Unal et al. show that converting absolute pitches and chords into relative interval sequences makes retrieval networks highly resilient to informal human queries and key transpositions.
  3. Cross-Cultural Applicability: Many world music traditions lack a fixed absolute pitch standard. For instance, in Indian Art Music, the absolute pitch of the tonic note (sa) varies dynamically based on the performer. Sankalp Gulati et al., along with the authors of Assessing the Tuning of Sung Indian Classical Music, demonstrated that calculating the distribution of relative intervals between pitch peaks allows computational models to analyze tuning and recognize phrases across diverse performers without needing a predefined absolute baseline.
  4. Latent Disentanglement: Relative encoding helps models separate what is being played from how or where it is played. J. Y. Zhao et al. enforced this by corrupting conditional melodies through random transposition before passing them to an adversarial discriminator. This forced the latent space of a Variational Autoencoder to discard absolute pitch information and represent harmonic movement in a purely relative, style-independent manner.

By aligning the input space with human cognitive invariants, relative interval encoding transforms music from a collection of rigid, absolute coordinates into a fluid web of relationships—enabling models to listen, analyze, and generate music much more like humans do.

Key Foundational Work

A Case Based Approach to the Generation of Musical Expression (1999) — contributed 1 concept elements to this direction.
MELONET I: Neural Nets for Inventing Baroque-Style Chorale Variations (1997) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 35 papers
  2. Version 1 · 2026-10-04 · 33 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_relative_interval_encoding_dc20bec6,
  title = {relative interval encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-interval-encoding/rdr_dc20bec692da465fb4c26f7a2afcdbf1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_relative_interval_encoding_dc20bec6_v2,
  title = {relative interval encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-interval-encoding/rdr_dc20bec692da465fb4c26f7a2afcdbf1/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake