Deep Discrete Encoders: Identifiable Deep Generative Models for Rich Data with Discrete Latent Layers

Seunghyun Lee, Yuqi GuarXiv 2025 · DOI · arXiv

Introduction

Modern generative AI models, such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), have demonstrated remarkable success in modeling complex data distributions. However, these models often function as "black boxes" characterized by non-identifiability—where multiple parameter configurations yield the same output. This lack of uniqueness hinders interpretability and reproducibility, particularly in high-stakes domains like psychometrics, medicine, and education. This paper introduces Deep Discrete Encoders (DDEs), a class of deep generative models that combine the representational power of deep learning with the statistical rigor of identifiable latent variable models.

Methodology

Architecture: The Shrinking-Ladder Design

DDEs are directed graphical models composed of multiple layers of binary (0/1) latent variables. The architecture follows a "shrinking-ladder" design where each successive layer contains fewer nodes, effectively distilling fine-grained features into broader concepts.

The hierarchical architecture of DDEs. The left panel illustrates the general multi-layer structure, while the right panel demonstrates a practical application in hierarchical topic modeling.

Identifiability Theory

A core contribution of this work is the establishment of mathematical conditions for model identifiability. The authors propose a "pure child" requirement: for parameters to be uniquely recoverable, each latent variable should ideally influence at least three "children" (nodes in the layer below) that are not shared with other parents. Even when these strict conditions are not met, the model is shown to be "generically identifiable," meaning it remains unique for almost all possible parameter values in practical settings.

Two-Stage Estimation Pipeline

Training deep models with discrete variables is computationally challenging due to the non-convex optimization landscape. To address this, the authors propose a two-stage pipeline:

  1. Layerwise Double-SVD Initialization: Instead of random initialization, the model uses Singular Value Decomposition (SVD) and Varimax rotation to establish a statistically sound starting point.
  2. Penalized SAEM: A Stochastic Approximation Expectation-Maximization (SAEM) algorithm is employed. Unlike standard EM, which scales exponentially with the number of variables, this approach scales linearly, enabling the handling of high-dimensional latent spaces.

Experimental Validation

The DDE framework was evaluated across three distinct data modalities:

  • MNIST Handwritten Digits: Unsupervised learning of binary pixel data to identify structural components (e.g., loops and strokes).
  • 20 Newsgroups: Hierarchical topic modeling using count data (Poisson distribution) to group words into sub-topics and broad categories.
  • TIMSS Assessment Data: Multimodal data from international mathematics studies, incorporating both binary response accuracy and continuous response times.

Key Results

  • Interpretability: Binary latent variables act as "switches," representing the presence or absence of specific features (e.g., a student's mastery of "Algebra").
  • Performance: On the MNIST dataset, DDEs achieved a 92.6% test classification accuracy using a simple decision tree on learned features, outperforming traditional Latent Class Models in reconstruction clarity.
  • Topic Coherence: In the 20 Newsgroups task, DDEs produced disjoint, non-overlapping topics, surpassing Latent Dirichlet Allocation (LDA) in perplexity metrics.
  • Educational Insights: The model successfully mapped high-level latent traits (e.g., general interest in math) to specific skill masteries (e.g., Geometry), providing actionable insights for educators.

Conclusion and Future Work

Deep Discrete Encoders bridge the gap between deep learning and classical statistics by providing a framework for identifiable, interpretable latent representations. While currently focused on binary variables, future extensions could incorporate categorical variables and first-order optimization methods like Stochastic Gradient Descent (SGD) to further enhance scalability and flexibility in high-dimensional settings.

Summary updated

BibTeX
@misc{lee2025deep,
  author = {Seunghyun Lee and Yuqi Gu},
  title = {{Deep Discrete Encoders: Identifiable Deep Generative Models for Rich Data with Discrete Latent Layers}},
  year = {2025},
  eprint = {2501.01414},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.01414},
}
Figure