Most generative models are prisoners of the grid. Whether it is a GAN trained on $256 \times 256$ faces or a Diffusion model optimized for $512 \times 512$ landscapes, these architectures are typically hard-coded to a specific resolution. When pushed to generate images at non-native aspect ratios or higher resolutions, they often suffer from "object repetition" (e.g., a person with two heads) or structural collapse.
A growing body of research is moving away from these discrete pixel grids toward resolution-agnostic generation. The goal is to treat an image not as a fixed matrix of numbers, but as a continuous signal that can be queried at any coordinate and any scale.
The Secret Coordinate System of Convolutional Networks
To understand why standard models fail at new resolutions, we must first understand how they perceive space. In Positional Encoding as Spatial Inductive Bias in GANs, Rui Xu and colleagues revealed that convolutional neural networks (CNNs) rely on an accidental "secret sauce": zero padding. Because padding occurs at the boundaries, the network learns to use these zeros as anchors to figure out where it is on the canvas.
However, this implicit encoding is fragile. If you change the canvas size, the padding anchors shift, and the model’s spatial reasoning breaks. To fix this, researchers like Łukasz Struski in LocoGAN and the authors of ScaleParty propose replacing implicit padding with explicit positional encodings—such as Cartesian grids or sinusoidal waves. By feeding the model its exact $(x, y)$ coordinates, it can maintain structural consistency regardless of the output dimensions.
From Pixels to Continuous Functions
The most radical shift in this field is the move toward Implicit Neural Representations (INRs). Instead of predicting a grid of pixels, the model learns a function $f(z, \phi) \to RGB$, where $z$ is a latent code and $\phi$ is a continuous coordinate.
This approach, championed in works like Omni-GAN by Lingxi Xie and HyperCGAN by Ivan Skorokhodov, allows for "infinite" resolution. Because the function is continuous, you can query it at $1024 \times 1024$ or $4000 \times 4000$, and it will synthesize details that are consistent across scales. Aditya Agarwal extended this to video in INR-V, representing an entire video as a single point in a continuous latent space, enabling smooth interpolation in both time and space.
Scaling Diffusion without Retraining
While GANs were the early adopters of resolution agnosticism, diffusion models face a different challenge: their self-attention mechanisms scale quadratically with resolution, making 4K generation computationally prohibitive.
To bypass this, researchers have developed training-free tiling and modulation strategies. The intuition is to generate a large image by denoising small patches that "talk" to each other to ensure global coherence.
- MultiDiffusion and SpotDiffusion (by Brian Bernhard Moser) use overlapping or shifting windows to stitch patches together seamlessly.
- ElasticDiffusion by Guha Balakrishnan decouples the generation into global structure and local details, preventing the "double-headed monster" artifact common in naive scaling.
- HiPrompt uses Multimodal LLMs to provide hierarchical guidance, giving the model a "big picture" prompt for the whole image and "local" prompts for specific patches.
Flexible Architectures for Variable Latents
The final piece of the puzzle is moving away from rigid backbones like the standard U-Net. Zeyu Lu and colleagues introduced FiT (Flexible Vision Transformer), which treats images as sequences of dynamically-sized tokens. By using 2D Rotary Positional Embeddings (RoPE), FiT can process any aspect ratio during inference, even if it was only trained on squares. Similarly, the LEGO framework by Huangjie Zheng uses stackable "bricks" to allow for reconfigurable, variable-resolution modeling.
By combining continuous representations, explicit spatial biases, and flexible attention mechanisms, this research direction is transforming generative AI from a tool that produces fixed-size thumbnails into a engine capable of synthesizing vast, high-fidelity digital worlds.