Abstract
Many types of neural network layers rely on matrix properties such as invertibility or orthogonality. Retaining such properties during optimization with gradient-based stochastic optimizers is a challenging task, which is usually addressed by either reparameterization of the affected parameters or by directly optimizing on the manifold. This work presents a novel approach for training invertible linear layers. In lieu of directly optimizing the network parameters, we train rank-one perturbations and add them to the actual weight matrices infrequently. This $P^4Inv$ update allows keeping track of inverses and determinants without ever explicitly computing them. We show how such invertible blocks improve the mixing and thus the mode separation of the resulting normalizing flows. Furthermore, we outline how the $P^4$ concept can be utilized to retain properties other than invertibility.
Introduction
Normalizing flows are a powerful class of generative models that require neural network layers to be both invertible and have easily computable Jacobian determinants. Traditionally, researchers have used restricted architectures like triangular matrices or coupling layers to satisfy these requirements. However, triangular matrices are inherently sequential and slow on modern parallel hardware (GPUs), while coupling layers can struggle with "mixing" information across dimensions.
Directly optimizing general invertible matrices is computationally expensive because computing a determinant or an inverse typically costs $O(n^3)$ time, which doesn't scale well as the dimensionality ($n$) of the data increases. This paper introduces $P^4Inv$, a novel method for training invertible linear layers by optimizing rank-one perturbations instead of full weight matrices. This approach enables $O(n^2)$ updates for inverses and determinants, significantly speeding up the training of normalizing flows while maintaining high expressivity.
Methodology: Property-Preserving Parameter Perturbation ($P^4$)
The core innovation is the Property-Preserving Parameter Perturbation ($P^4$) update. Instead of updating a weight matrix $A$ directly at every optimizer step, the authors keep $A$ frozen and optimize a rank-one perturbation: $$A_{perturbed} = A + uv^T$$ where $u$ and $v$ are learnable vectors. Periodically (every $N$ steps), these perturbations are "merged" into the main weight matrix, and the vectors are reset.
The efficiency of this method stems from two classical linear algebra results: the Sherman-Morrison Formula for $O(n^2)$ inverse updates and the Matrix Determinant Lemma for $O(n^2)$ determinant updates. By using these formulas, the model can keep track of inverses and determinants throughout training without ever performing a full $O(n^3)$ computation.
Experimental Evaluation
The authors evaluated $P^4Inv$ across several benchmarks:
- Linear Toy Problems: Fitting 32-dimensional positive definite and 128-dimensional orthogonal matrices.
- 2D Toy Distributions: Density estimation tasks (e.g., rings, spirals) comparing against RealNVP.
- Alanine Dipeptide: A 66-dimensional molecular system used to test "Boltzmann Generators."
- MNIST: Studying how the reduced parameter space of rank-one updates affects non-invertible tasks.
Key Results and Insights
The $P^4Inv$ method demonstrates significant advantages in both computational efficiency and model performance:
- Massive Speedups: For high-dimensional problems (e.g., $n=4096$), $P^4Inv$ is roughly two orders of magnitude faster than standard inversion methods during the backward pass.
- Superior Density Estimation: In 2D tasks, $P^4Inv$ models reproduced multimodal ground truths more accurately than standard RealNVP models with a similar number of parameters.
- Better Mixing: In normalizing flows, replacing simple coordinate swaps with $P^4Inv$ layers leads to much better "mixing" of dimensions, resulting in cleaner mode separation.
- Robustness: The method can successfully navigate through "ill-conditioned" regimes where the matrix is temporarily non-invertible and still recover a correct inverse.
Discussion and Future Work
While $P^4Inv$ offers substantial benefits, it has some limitations. A rank-one update only has $2n$ degrees of freedom compared to $n^2$ for a full matrix, which can lead to slower convergence near the optimum. Additionally, when optimizing parameters of the inverse directly, the model can occasionally get stuck in local minima.
Future work includes extending the $P^4$ concept to other matrix groups, such as the orthogonal group $O(n)$, using Householder reflections or Givens rotations to preserve different structural properties.
Summary updated