Learning Sparsity of Representations with Discrete Latent Variables

Zhao Xu, Daniel Oñoro-Rubio, Giuseppe Serra, Mathias NiepertIJCNN 2021 · DOI · OpenReview

Introduction

Deep Latent Generative Models (DLGMs), such as Variational Autoencoders (VAEs), have become a cornerstone of unsupervised representation learning. However, these models typically yield dense latent representations where every feature is assigned a non-zero value. In contrast, many real-world phenomena are inherently sparse—only a small subset of potential features or concepts is relevant to any given observation. While traditional methods employ soft regularization (e.g., $L_1$ penalties) to encourage sparsity, they often lack the flexibility to adapt the degree of sparsity to individual data points or to enforce hard constraints required for resource-constrained environments like IoT devices.

This paper introduces the Sparse Deep Latent Generative Model (SDLGM), a framework that treats the sparsity level of a representation as a learnable, per-observation latent variable.

The SDLGM Framework

The core innovation of SDLGM is a hierarchical sampling strategy that explicitly separates the modeling of "which" features are active from "what" those features represent.

Two-Step Discrete Sampling

The model employs a two-step process to generate a sparse representation for an observation:

  1. Sparsity Modeling: For each observation, the model draws $L_0$ binary "gate" variables $\omega_{i, \ell}$ from a Bernoulli distribution. The sum of these gates defines the sparsity level (the number of active components) for that specific instance.
  2. Feature Modeling: Simultaneously, $L_0$ independent categorical features $z_{i, \ell}$ are sampled. The final representation is the element-wise product of these features and their corresponding gates, effectively masking out inactive components.
Architecture of the SDLGM encoder, illustrating the two-step sampling process for sparsity gates and feature selection.

To maintain end-to-end differentiability despite the discrete nature of these variables, the authors utilize the Gumbel-Softmax distribution. This continuous approximation allows the model to be trained using standard backpropagation.

Experimental Results

The SDLGM was evaluated on both unsupervised tasks (MNIST) and multi-label supervised tasks (Birds, Emotions, CAL500).

Unsupervised Representation Learning

On the MNIST dataset, SDLGM achieved a Negative Evidence Lower Bound (ELBO) of 99.05, outperforming standard Gumbel-Softmax (101.5) and other discrete latent variable models like DARN and MuProp. The model successfully learns to cluster digits in the latent space while adapting the number of active features to the complexity of the input.

Visualizations of learned representations: (left) t-SNE of MNIST latent space, (middle) distribution of learned sparsity levels, and (right) logit distribution.

Multi-label Classification and Sparsity Prediction

In supervised settings, SDLGM variants (Discriminative, Generative, and Conditional) consistently outperformed baseline models like Sparsemax and Logistic regression in terms of Micro- and Macro-averaged F1 scores. Furthermore, in a custom multi-digit MNIST task designed to test counting capabilities, SDLGM predicted the frequency of specific features with a Mean Squared Error (MSE) of 0.05, significantly better than the Random Forest baseline (0.16).

Conclusion

SDLGM provides a robust framework for learning sparse representations where the sparsity level is an emergent property of the data rather than a fixed hyperparameter. By leveraging differentiable discrete sampling, the model achieves state-of-the-art performance in both generative and discriminative tasks. Future work includes scaling this approach to high-dimensional data and exploring its application in sparse attention mechanisms for natural language processing.

Summary updated

Abstract

Deep latent generative models have attracted increasing attention due to the capacity of combining the strengths of deep learning and probabilistic models in an elegant way. The data representations learned with the models are often continuous and dense. However in many applications, sparse representations are expected, such as learning sparse high dimensional embedding of data in an unsupervised setting, and learning multi-labels from thousands of candidate tags in a supervised setting. In some scenarios, there could be further restriction on degree of sparsity: the number of non-zero features of a representation cannot be larger than a pre-defined threshold <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$L_{0}$</tex> . In this paper we propose a sparse deep latent generative model SDLGM to explicitly model degree of sparsity and thus enable to learn the sparse structure of the data with the quantified sparsity constraint. The resulting sparsity of a representation is not fixed, but fits to the observation itself under the pre-defined restriction. In particular, we introduce to each observation <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$i$</tex> an auxiliary random variable <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$L_{i}$</tex> , which models the sparsity of its representation. The sparse representations are then generated with a two-step sampling process via two Gumbel-Softmax distributions. For inference and learning, we develop an amortized variational method based on MC gradient estimator. The resulting sparse representations are differentiable with backpropagation. The experimental evaluation on multiple datasets for unsupervised and supervised learning problems shows the benefits of the proposed method.
BibTeX
@misc{xu2021learning,
  author = {Zhao Xu and Daniel Oñoro-Rubio and Giuseppe Serra and Mathias Niepert},
  title = {{Learning Sparsity of Representations with Discrete Latent Variables}},
  year = {2021},
  eprint = {2304.00935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.00935},
}
Figure