Variational Autoencoders (VAEs) are powerful tools for finding low-dimensional patterns in complex data, a process known as dimensionality reduction. However, their decoders are typically "black box" neural networks. This creates two major hurdles for scientists trying to extract meaningful insights from their data:
- Lack of Control: If a researcher knows a biological biomarker must decrease as a disease progresses, a standard VAE might still produce a "wiggly," non-monotonic curve because it has no concept of physical or biological constraints.
- Feature-Level Ambiguity: Standard VAEs find structure among samples (rows in a dataset), but they do not naturally group features (columns) that share similar functional behaviors over the latent space.
While previous work, such as BasisVAE, attempted to cluster features, it struggled to define broad "families" of behavior (like "all increasing functions") without extensive manual tuning. To address this, researchers have introduced BasisDeVAE, a new framework that constrains the black-box decoder by modeling the derivatives of features rather than the features themselves. This allows scientists to bake in prior knowledge—like monotonicity—to simultaneously reduce dimensionality and cluster features into interpretable groups.
Methods: The Derivative Trick and Behavior Clustering
The authors introduce two core models to achieve this interpretable clustering: DeVAE and BasisDeVAE.
The Derivative Trick (DeVAE)
Instead of the decoder mapping a latent variable $z$ directly to an output $x$ (i.e., $x = f(z)$), DeVAE models the derivative: $\frac{\partial x}{\partial z} = f_\theta(z)$.
By applying a softplus activation function to the output of this derivative network, the authors ensure the derivative is always positive. This mathematically guarantees that the resulting reconstructed function is monotonically increasing.
To obtain the actual feature value, the model integrates the derivative using Gauss-Legendre quadrature. This numerical method turns the integral into a weighted sum of neural network evaluations, making it fully differentiable and easy to implement in modern deep learning frameworks like PyTorch.
Clustering by Behavior (BasisDeVAE)
The researchers then embed this derivative-based decoder into a clustering framework. Features are assigned to one of $K$ "basis families" (e.g., Monotonic Increasing, Monotonic Decreasing, or Transient/Gaussian-like). The model learns which feature belongs to which family while simultaneously learning the latent representation of the data.
Experimental Validation
The authors evaluated their models on three distinct types of data to test recovery, real-world applicability, and scalability:
- Synthetic Data: A dataset of 500 samples with 50 features (mixed monotonic and transient signals) was used to verify if the model could accurately recover known ground-truth clusters.
- OASIS-3 (Brain Imaging): MRI data from 2,047 sessions. Features included volumes of brain regions (like the hippocampus) that are biologically known to atrophy (decrease) with age and Alzheimer's disease.
- Single-cell RNA sequencing (scRNA-seq): A high-dimensional "stress test" dataset of 8,509 mouse sperm cells and 5,216 genes, testing both scalability and biological interpretability.
Key Insights and Results
- Constraints via Range, not Architecture: It is notoriously difficult to constrain the functional shape of a neural network. By moving to "derivative space," the authors cleverly turn shape constraints into simple range constraints (e.g., forcing the output to be positive).
- Superior Fit and Extrapolation: Because the model "knows" the underlying shape of the function (e.g., it must keep going up), it performs significantly better than standard VAEs when trying to predict behavior at the edges of the latent space where data is sparse. On the OASIS brain imaging data, DeVAE achieved a higher predictive log-likelihood and a better ELBO (Evidence Lower Bound) compared to linear and standard VAEs, all while providing biologically realistic decreasing trajectories for brain volume.
- Efficiency through Parallelism: Integration sounds computationally expensive, but the quadrature method allows the model to evaluate the decoder at multiple points simultaneously. Surprisingly, BasisDeVAE was roughly 2x faster per epoch than the previous state-of-the-art BasisVAE on large genomic data.
- Scalability and Interpretability: In the scRNA-seq experiment, BasisDeVAE successfully clustered over 5,000 genes into interpretable "increasing," "decreasing," and "transient" groups. Unlike standard VAEs where clusters might be mathematically distinct but visually entangled, BasisDeVAE clusters have clear semantic meanings (e.g., "Genes in Cluster 1 are all up-regulated over time").
Limitations and Future Work
While BasisDeVAE represents a significant step forward for interpretable deep learning in the sciences, the authors note a few limitations. Currently, the derivative $\frac{\partial x}{\partial z}$ only depends on the latent variable $z$. Allowing it to also depend on the features $x$ themselves could lead to even more expressive models.
Additionally, the current work focuses on first-order derivatives (velocity/slope). Future versions could incorporate second-order derivatives to control curvature or acceleration. Finally, while the paper demonstrates monotonic and transient behaviors, incorporating other scientific priors—like periodicity for circadian rhythms—would require defining new derivative-based families.
Summary updated