Standard neural networks are rigid. A typical classifier expects a fixed-size vector; a standard CNN expects a specific number of input channels. In the real world, however, data is rarely so cooperative. A doctor might have three MRI sequences for one patient but only two for another. A satellite might capture a burst of five images over a target, while a drone captures twelve. A robot might navigate using four cameras today and six tomorrow.
To handle this fluidity, a growing body of research has moved away from "fixed-slot" architectures toward permutation-invariant aggregation. This approach allows models to ingest an arbitrary number of inputs in any order, producing a stable representation that scales with the available evidence.
The Geometry of Symmetry
The core challenge of variable-input learning is how to collapse a set of $N$ feature vectors into a single global descriptor without losing information or becoming sensitive to the order of the inputs. If a model treats the first input differently than the second, it cannot be truly "view-agnostic" or "modality-agnostic."
The solution lies in symmetric functions—mathematical operations where the output remains identical regardless of the input order. As Mohammad Havaei and colleagues demonstrated in HeMIS: Hetero-Modal Image Segmentation, simple statistics like the mean and variance across embeddings allow a model to handle missing medical modalities gracefully. By embedding each available scan into a shared latent space and then averaging them, the network creates a "consensus" representation that doesn't require "filling in" missing data.
This intuition extends to 3D vision. In ImVoxelNet, Danila Rukhovich shows that by projecting 2D features into a 3D voxel grid and averaging the features that land in the same coordinate, a model can perform object detection using any number of camera views. The voxel grid acts as a physical anchor, and the averaging operation ensures that more views simply lead to a "clearer" signal rather than a different architectural requirement.
Max-Pooling as a Feature Filter
While averaging provides a "consensus," max-pooling acts as a "filter" for the most salient information. This is particularly powerful in tasks where some inputs are "better" than others.
In PS-FCN: A Flexible Learning Framework for Photometric Stereo, Kai Han utilized max-pooling to aggregate information from an arbitrary number of light directions. In photometric stereo, different lights create different shadows. By taking the maximum activation across the stack of features, the model naturally ignores the low activations caused by shadows and prioritizes the high activations from well-lit regions.
Similarly, Fredo Durand and Miika Aittala applied this to Flexible SVBRDF Capture. When capturing material properties from handheld photos, some angles might show a distracting reflection while others show the true surface color. An order-invariant max-pooling layer allows the network to "pick and choose" the best features from each photo, hitting a sweet spot where the reconstruction quality improves as the user provides more images.
Beyond Simple Statistics: Attention and Recursion
Simple mean or max operations treat every input as equally important. However, in complex scenes, some perspectives are more informative than others. Modern architectures use self-attention to let the inputs "talk" to each other before aggregation.
In HeatFormer, researchers used a view-agnostic attention module to consolidate spatial features from a variable number of cameras. By calculating similarity across views, the model can weigh a clear, unobstructed view of a person more heavily than a view where they are partially occluded. This transforms the aggregation from a passive mathematical operation into an active, data-driven "neural optimizer."
For temporal data, such as satellite image bursts, Michel Deudon proposed a recursive fusion approach in HighRes-net. Instead of processing all frames at once, the model recursively updates a global hidden state as it sees each new frame. This is permutation-invariant because the fusion operator is designed to be symmetric, allowing the model to ingest any number of low-resolution frames to produce a single high-resolution output.
Why It Works: The "Evidence" Intuition
The reason these methods work across so many domains—from speech recognition to font style transfer—is that they treat inputs as evidence rather than slots.
In a traditional model, the "first input" is a specific slot (e.g., the Red channel). In a permutation-invariant model, the inputs are a "bag of observations." As Srinath Sridhar and Davis Rempe demonstrated with X-NOCS, this allows for dynamic scaling. A model trained on three views can be deployed with ten views at inference time; because the aggregation layer (like a mean or max) doesn't care about the count, it simply sees "more evidence" and produces a more accurate 3D reconstruction.
This shift from "fixed-input" to "set-input" architectures is fundamental for deploying AI in the messy, unpredictable real world, where sensors fail, cameras move, and data is rarely complete.