Learning Discrete Representation with Optimal Transport Quantized Autoencoders

Xiaoyu Bie, Dexiong Chen, Xiaodong Cun, Xi ShenICLR 2023 · OpenReview

Abstract

Vector quantized variational autoencoder (VQ-VAE) has recently emerged as a powerful generative model for learning discrete representations. Like other vector quantization methods, one key challenge of training VQ-VAE comes from the codebook collapse, i.e. only a fraction of codes are used, limiting its reconstruction qualities. To this end, VQ-VAE often leverages some carefully designed heuristics during the training to use more codes. In this paper, we propose a simple yet effective approach to overcome this issue through optimal transport, which regularizes the quantization by explicitly assigning equal number of samples to each code. The proposed approach, named OT-VAE, enforces the full utilization of the codebook while not requiring any heuristics. We empirically validate our approach on three different data modalities: images, speech, and 3D human motions. For all the modalities, OT-VAE shows better reconstruction with higher perplexity than other VQ-VAE variants on several datasets. In particular, OT-VAE achieves state-of-the-art results on the AIST++ dataset for 3D dance generation. Our code will be released upon publication.
BibTeX
@inproceedings{bie2023learning,
  author = {Xiaoyu Bie and Dexiong Chen and Xiaodong Cun and Xi SHEN},
  title = {{Learning Discrete Representation with Optimal Transport Quantized Autoencoders}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=fcg9phFVzjd},
}

Loading related content...

Figure