Abstract
Many recent invertible neural architectures are based on coupling block designs where variables are divided in two subsets which serve as inputs of an easily invertible (usually affine) triangular transformation. While such a transformation is invertible, its Jacobian is very sparse and thus may lack expressiveness. This work presents a simple remedy by noting that subdivision and (affine) coupling can be repeated recursively within the resulting subsets, leading to an efficiently invertible block with dense, triangular Jacobian. By formulating our recursive coupling scheme via a hierarchical architecture, HINT allows sampling from a joint distribution p(y,x) and the corresponding posterior p(x|y) using a single invertible network. We evaluate our method on some standard data sets and benchmark its full power for density estimation and Bayesian inference on a novel data set of 2D shapes in Fourier parameterization, which enables consistent visualization of samples for different dimensionalities.
Introduction
Invertible Neural Networks (INNs) have become a cornerstone of generative modeling, offering exact density estimation and efficient sampling. Most contemporary INNs utilize coupling blocks, such as those found in RealNVP or GLOW. In a standard coupling block, input variables are split into two halves; one half remains unchanged and parameterizes the transformation of the other. While this ensures an easily computable Jacobian and efficient inversion, it results in a highly sparse Jacobian matrix. Consequently, these models often require stacking a large number of layers to capture complex dependencies, leading to increased depth and computational cost.
In contrast, autoregressive models provide a dense triangular Jacobian, allowing every variable to depend on all preceding ones. While highly expressive, they are sequentially constrained, making sampling computationally expensive. This paper introduces Hierarchical Invertible Neural Transport (HINT), which bridges this gap by providing the expressiveness of a dense triangular Jacobian with the parallel efficiency of coupling blocks.
The Recursive Coupling Block
The core innovation of HINT is the recursive coupling block. Instead of a single split, the variables are subdivided recursively. For an input vector, the first split creates $x_1$ and $x_2$. Subsequently, $x_1$ is itself split into $x_{1a}$ and $x_{1b}$, and this process continues. By nesting coupling blocks within each other, the zero entries in the Jacobian matrix are progressively filled.
This recursive structure ensures that in a single hierarchical block, a dense triangular Jacobian is achieved. Unlike autoregressive models, the sub-networks at each level of the hierarchy can be computed in parallel, maintaining the high throughput characteristic of coupling-based INNs.
Hierarchical Architecture for Inference
Beyond density estimation, HINT is designed for Bayesian inference tasks, such as estimating the posterior $p(x|y)$ given observations $y$. The authors propose a hierarchical organization where information flows from observed variables ($y$) to hidden parameters ($x$), but not vice versa.
This "lane" structure allows a single trained model to simultaneously represent the joint distribution $p(x, y)$ and function as an efficient conditional sampler for $x$ given $y$. This unified framework simplifies the training process for complex Bayesian tasks.
Experimental Evaluation
The performance of HINT was evaluated on standard UCI tabular datasets (Power, Gas, MiniBooNE) and a novel Fourier Shapes dataset. The Fourier Shapes dataset consists of 2D closed curves parameterized by Fourier coefficients. This dataset is particularly useful as it allows for high-dimensional modeling (up to 100D) while remaining visually interpretable, making it easier to assess whether the model captures sharp features and symmetries.
Key Results
- Expressiveness: On the "Lens" shape dataset, a single-block HINT outperformed a single-block RealNVP. RealNVP's inherent sparsity prevents it from modeling the full distribution in a single pass, whereas HINT's dense Jacobian captures the necessary interactions.
- Benchmark Performance: HINT consistently achieved higher log-likelihoods than RealNVP on UCI benchmarks, even under strict parameter budget constraints.
- Bayesian Inference: In conditional tasks, HINT produced visually superior samples with sharper angles and better-preserved symmetries compared to standard Conditional INNs (cINNs).
Conclusion and Future Work
HINT demonstrates that increasing the "density" of variable interactions within a layer can be more effective than simply increasing network depth. By linking the architecture to the theory of Transport Maps and Knothe-Rosenblatt rearrangements, the authors provide a theoretically grounded yet practical approach to building universal triangular maps.
While the current implementation faces some bookkeeping overhead, future optimizations using masking operations could further improve speed. Additionally, exploring the application of recursive coupling to convolutional architectures for image synthesis remains a promising avenue for future research.
This paper was presented at AAAI-2021.
Summary updated