Towards Robust Unsupervised Disentanglement of Sequential Data - A Case Study Using Music Audio

Yin-Jyun Luo, Sebastian Ewert, Simon DixonIJCAI 2022 · DOI

TL;DR

This paper introduces TS-DSAE, a novel two-stage training framework that significantly improves the robustness of disentangling sequential data, particularly music audio. By addressing the common issue of "global factor collapse" in existing methods, TS-DSAE achieves more reliable disentanglement and better reconstruction quality without relying on adversarial training or data augmentation.

What Problem Does This Paper Address?

Representation learning aims to uncover underlying factors that explain observed data. Disentangled representations are particularly valuable as they isolate these factors, making models more interpretable and enabling controllable data generation. While supervised methods can achieve disentanglement, the scarcity of labeled data necessitates unsupervised approaches. For sequential data, a common structural bias is to disentangle global (sequence-level) factors from local (time-varying) factors. The Disentangled Sequential Autoencoder (DSAE) is a probabilistic graphical model that attempts this but suffers from sensitivity to architecture, capacity, and a tendency for the global factor to collapse, meaning it's not effectively learned. This paper tackles the challenge of making unsupervised disentanglement of sequential data robust and reliable, specifically addressing the limitations of DSAE.

Methods

The core contribution is the Two-Stage Disentangled Sequential Autoencoder (TS-DSAE) framework. It builds upon the DSAE architecture but introduces a novel two-stage training process and additional regularization terms.

  • Two-Stage Training:
    • Stage 1: Constrained Training: The local encoder and transition network parameters are frozen. This forces the global encoder to capture essential sequence-level information, biasing its posterior distribution to represent global factors. This stage effectively isolates the global latent variable.
  • Stage 2: Informed-Prior Training: All parameters are unfrozen. The global prior is no longer a simple standard Gaussian but is set to the posterior learned from the constrained training stage. This "anchors" the global latent distribution to the learned sequence-specific prior, mitigating over-regularization and encouraging a more informative global latent space.
  • Factor Invariance and Manifestation Regularization: Four additional loss terms are introduced to explicitly encourage disentanglement. These terms work by:
  • Replacing latent variables (global or local) between different sequences.
  • Measuring how well the inferred latent variables of the modified sequences predict the original or swapped factors.
  • This encourages the global factor to remain invariant when local factors change, and vice-versa, promoting true disentanglement.

Datasets

The authors evaluate their method on two datasets:

  • dMelodies: An artificial dataset synthesized from monophonic symbolic music. It allows for controlled variation of global factors (tonic, scale, octave) and local factors (arpeggiation direction, rhythm).
  • URMP: A real-world music audio dataset containing violin and trumpet tracks. This dataset is used to demonstrate the applicability of TS-DSAE to realistic data.

Key Insights and Ideas

  • The "Shortcut Problem" is Real: The paper highlights how standard DSAE struggles with the "shortcut problem," where local latents can capture all necessary information, rendering the global latent useless. This is exacerbated by a simple, uni-modal prior for the global variable.
  • Two-Stage Training as a Powerful Inductive Bias: The constrained training stage acts as a crucial bottleneck, forcing the global encoder to learn meaningful sequence-level representations. This learned posterior then serves as a much richer prior for the full training, effectively guiding the model towards disentanglement.
  • Robustness Across Configurations: TS-DSAE demonstrates remarkable robustness. It performs well across different local latent space sizes and even with a simpler "factorised q" architecture, unlike baseline DSAE which is highly sensitive to these choices.
  • Disentanglement Without Adversarial Training: The proposed method achieves robust disentanglement without resorting to complex adversarial training or domain-specific data augmentation, which often require extensive hyperparameter tuning.

Key Results

  • Instrument Classification: TS-DSAE consistently achieves the highest macro F1 scores for instrument classification using the global latent space across various local latent space sizes on both dMelodies and URMP datasets. This indicates a more linearly separable and discriminative global latent space.
  • Reconstruction Quality: TS-DSAE effectively balances disentanglement and reconstruction quality. While other models improve reconstruction (lower FAD) at the cost of disentanglement (lower F1), TS-DSAE shows improvements in both.
  • Raw Pitch Accuracy (RPA): On the URMP dataset, TS-DSAE consistently improves RPA with increasing local latent space size, achieving comparable or better results than competing models while also disentangling factors.
  • Multiple Global Factors: The paper demonstrates that TS-DSAE can handle multiple global factors (e.g., instrument and octave) and successfully disentangle them, as shown in qualitative examples where swapping global factors leads to predictable changes in the generated audio.

Limitations and Future Work

  • Trade-off Between Disentanglement and Reconstruction: While TS-DSAE mitigates this trade-off, it remains a challenge. The authors suggest that scaling the regularization terms differently might offer further improvements.
  • Stationary Global Factor Assumption: The current model assumes a single, stationary global latent variable for the entire sequence. This might be too restrictive for sequences where global factors evolve slowly.
  • Future Work:
    • Evaluating TS-DSAE on multi-modal data generation.
  • Extending the framework to modalities beyond music audio.
  • Investigating hierarchical latent variable models that can capture information at multiple frame rates.
  • Exploring more complex decoder architectures, such as autoregressive decoders.

Summary updated

BibTeX
@inproceedings{luo2022towards,
  author = {Yin-Jyun Luo and Sebastian Ewert and Simon Dixon},
  title = {{Towards Robust Unsupervised Disentanglement of Sequential Data - A Case Study Using Music Audio}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/458},
}
Figure