Abstract
Flow-based generative models traditionally rely on $1 \times 1$ convolutions due to their tractability and invertibility, but these layers lack the spatial receptive field necessary to capture complex dependencies. This paper introduces a novel method for implementing invertible $n \times n$ convolutions by reformulating them as a composition of a learnable "Shift Function" and standard $1 \times 1$ convolutions. This approach provides the spatial flexibility of large kernels while remaining computationally efficient and easy to invert. Experimental results on ImageNet and CelebA-HQ demonstrate improved density estimation and high-quality image synthesis.
Introduction
Flow-based models like Glow and RealNVP learn data distributions through a sequence of invertible transformations. A major bottleneck in these architectures is the reliance on $1 \times 1$ convolutions. While mathematically convenient for calculating Jacobian determinants, $1 \times 1$ convolutions cannot model spatial relationships between neighboring pixels. Standard $n \times n$ convolutions offer this spatial awareness but are difficult to invert and computationally expensive within the constraints of normalizing flows. This work bridges this gap by providing the spatial benefits of $n \times n$ kernels without the typical mathematical overhead.
Methods
The core innovation is the reformulation of standard $n \times n$ convolutions. Instead of sliding a kernel over an input, the authors propose shifting the input and applying $1 \times 1$ convolutions. Mathematically, this is equivalent to a standard convolution but allows for easier inversion.
The architecture introduces an Invertible Shift Function ($S$), defined as a learnable linear transformation per channel: $$S(X_{c,i,j}) = \alpha_c X_{c,i,j} + \beta_c$$ where $\alpha$ and $\beta$ are learnable scale and translation parameters. When combined with an invertible $1 \times 1$ convolution, the network effectively simulates an $n \times n$ convolution. The Jacobian determinant of the Shift Function is simply the product of the $\alpha$ values, making it highly efficient.
Results and Discussion
The model was evaluated on CIFAR-10, ImageNet ($32 \times 32$ and $64 \times 64$), and CelebA-HQ. Key results in bits per dimension (BPD) include:
- ImageNet 32x32: 3.96 BPD (outperforming Glow's 4.09).
- ImageNet 64x64: 3.74 BPD (outperforming Glow's 3.81).
- CIFAR-10: 3.50 BPD.
The primary insight is that increasing the receptive field early in the flow improves generative quality. Furthermore, the proposed method is parameter-efficient, using fewer parameters than standard convolutions while maintaining tractability.
Conclusion
By viewing $n \times n$ convolution as a composition of spatial shifts and $1 \times 1$ operations, this work enables more expressive flow-based models. Future work will address memory constraints for high-resolution training and improving the stability of invertible matrix updates during optimization.
Summary updated