Systems that rebuild a moving 3D scene from video in one network pass ("feed-forward") avoid the minutes-to-hours of per-scene fitting that classic methods need. They still have two weaknesses. First, they treat each window of frames more or less from scratch, so what the model learned about a surface patch at frame $t$ is not carried in a usable form to frame $t+1$ once that patch has rotated or bent. Second, training them to refine their output over many frames and many update steps means backpropagating through a long unrolled loop, and the memory needed for that grows with the length of the loop. We propose a single mechanism that addresses both. The scene is represented by 3D Gaussians, which are small oriented ellipsoids that a renderer turns into images. Each Gaussian already carries a rotation, so it already defines a local coordinate frame. We store the network's per-Gaussian memory as vectors expressed in that frame. Moving these vectors between neighbouring Gaussians, or forward in time along a Gaussian's trajectory, is then a product of two known rotations. Nothing has to be precomputed, and the operation stays valid as the scene deforms.
Differential geometry has a name for this operation: parallel transport, the rule for moving a direction from one point of a curved space to another without arbitrary twisting. Mesh networks have used it to share filter weights across surfaces, but only on clean, static triangle meshes, and only with transport operators computed before training. Our observation is that a set of oriented primitives supplies its own transport for free. That turns a geometric weight-sharing idea into something that fits the streaming, deforming, topology-free setting of 4D reconstruction. We pair it with a reversible recurrent update, so that training memory does not grow with the number of refinement steps and frames. This last part is a secondary engineering contribution, and we test it against plain gradient checkpointing rather than assume it wins.
Core Claim
A recurrent feed-forward 4D Gaussian reconstructor whose hidden state is stored in each Gaussian's own frame and moved by relative rotations should do two things better than one whose state lives in a single global frame. It should keep its per-Gaussian memory consistent under non-rigid motion, and it should share weights across differently oriented parts of the scene. Both should show up as better temporal consistency and better novel-view quality on genuinely monocular dynamic video, at equal compute.
Concretely, let $R_i^t \in \mathrm{SO}(3)$ be the rotation of Gaussian $i$ at frame $t$, and $h_i^t \in \mathbb{R}^{3\times c}$ its $c$ vector-valued hidden channels written in that local frame. A message from neighbour $j$ to Gaussian $i$ is computed on the transported feature
$$ \tilde h_{j\to i}^t = (R_i^t)^\top R_j^t \, h_j^t , $$
so that the update rule sees neighbour features in $i$'s own coordinates. The same weights then serve every location and orientation. Carrying state from frame $t$ to frame $t+1$ uses the predicted per-Gaussian rotation change, so state written in local coordinates needs no correction when the frame co-rotates with the Gaussian. Any learned residual twist is an explicit, inspectable quantity.
One subtlety makes this cleaner than the mesh setting. A Gaussian's frame is defined only up to sign flips of its axes, and up to axis swaps when two scales are equal. The ambiguity group is therefore finite. We make the update exactly equivariant to the four-element sign-flip group after sorting the axes by scale. This is cheap, and it avoids the approximate continuous equivariance that mesh methods settle for. Exact continuous $\mathrm{SO}(2)$ equivariance with pointwise nonlinearities is not achievable, which is why both Gauge Equivariant Mesh CNNs and the Gauge Equivariant Transformer resort to Fourier-based or discrete cyclic approximations.
Relation to Prior Work
The geometric mechanism comes from gauge-equivariant learning. Cohen et al. framed moving a filter across a manifold as parallel transport. GEM-CNNs built direction-sensitive mesh convolutions by transporting vector features along edges. The Gauge Equivariant Transformer extended transport to attention between distant tangent planes. All three need precomputed tangent bases and transport along a fixed, clean 2-manifold mesh. As the research file notes, this breaks down when geometry is reconstructed on the fly or deforms. We keep their weight-sharing principle and change where the frames come from: the primitives themselves, rather than a mesh. That removes the precomputation entirely and does not require a manifold surface.
Vector Bundle Attention and Bundle Neural Networks learn orthogonal connections between node spaces on graphs. They are the closest relatives in spirit. Their connections are learned free parameters. Ours are fixed by rendered geometry that is supervised photometrically, which ties transport to physical rotation instead of leaving it as extra capacity.
The application regime comes from feed-forward reconstruction. GS-LRM predicts static Gaussians from a few posed images in about 0.23 seconds. DGS-LRM predicts deformable Gaussians and scene flow from monocular video in about 0.6 seconds, but reports blur and floaters away from the input trajectory. ReSplat refines Gaussians over four recurrent steps using rendering-error maps and k-nearest-neighbour attention, roughly 100 times faster than per-scene 3DGS fitting. It is static, and its neighbour attention works on global-frame features. Our method is closest to "ReSplat made dynamic and streaming, with frame-local state." The difference that matters is not adding time. It is that the recurrent state survives rotation of the primitives that carry it.
Monocular Dynamic View Synthesis: A Reality Check constrains our evaluation: benchmarks with teleporting cameras leak multi-view information, so we evaluate only on its effectively-monocular protocol.
The memory side borrows technique and constraint from reversible training. RevSNN uses additive coupling, $Y_1 = X_1 + F(X_2)$ and $Y_2 = X_2 + G(Y_1)$, to reconstruct activations during the backward pass. It reports 8.1 to 9.1 times memory reduction at about 33% more wall-clock time, and notes that some layers stay irreversible. Coupling is invertible whatever $F$ and $G$ are, so $F$ may contain the non-linear render-and-compare step. This is important because Sequence Accumulation only works for linear recurrences and so cannot handle a rendering feedback loop. We split each Gaussian's state into a geometry half and a latent half, and alternate their updates inside the coupling.
Research Program
We build a streaming reconstructor. A per-frame image encoder (irreversible, with its activations cached) produces tokens. A set of Gaussians with explicit rotation, scale, opacity, colour, and frame-local vector features is initialised from the first frames. For each new frame, the model runs $K$ refinement steps. Each step renders the current Gaussians, computes error maps against the observed frame, and lifts error features onto Gaussians. It then runs transported message passing over a k-nearest-neighbour graph, in which scalar channels attend normally and vector channels are rotated by the equation above, and applies a coupled update. Between frames, state is carried along trajectories in local coordinates.
Training unrolls $T$ frames by $K$ steps. Reversible coupling keeps activation memory roughly constant in $T \cdot K$, apart from the encoder cache and the drift of floating-point inversion. We measure that drift directly, comparing gradients from reconstructed activations against gradients from stored activations.
The analysis has three parts. The first measures how much of the gain comes from transport itself, by comparing against identical architectures with global-frame features or invariant-only features. The second tracks whether per-Gaussian features stay stable when the underlying surface rotates, using the cosine similarity of a Gaussian's feature across frames, binned by how much it rotated. The third asks whether the finite sign-flip ambiguity, if left unhandled, measurably hurts results.
Baselines
The naive reference point is per-window feed-forward reconstruction with no state carried across windows: DGS-LRM-style prediction applied independently to sliding windows. It fixes the floor for temporal consistency. Our main reference is DGS-LRM itself. The file reports its speed (about 0.6 s per video) but gives no quality figure on the Reality Check protocol, so no reported result was found on Lacuna for DGS-LRM novel-view PSNR or LPIPS. We will rerun it, or a faithful reimplementation if weights are unavailable, on the same splits.
The strongest competitor on this specific question is a dynamic extension of ReSplat that we construct. It has the same Gaussian count, the same encoder, the same $K$ refinement steps, and the same k-nearest-neighbour attention, and it carries state across frames in a global frame. Its state includes positions and rotations as input features, so it can in principle learn rotation-awareness. This is the competitor that isolates our idea, because it differs only in where features live and how they move. The file reports a roughly 100-fold speedup over 3DGS for ReSplat on static scenes but no dynamic-scene quality number, so no reported result was found on Lacuna for a dynamic ReSplat. It will receive the same training data, the same GPU-hours, and the same hyperparameter-search budget (an equal number of sweep trials). Beating it means higher masked PSNR and lower LPIPS on held-out monocular trajectories, and better temporal-consistency scores, with confidence intervals across scenes that do not overlap. Matching it at lower parameter count would count as a partial win for weight sharing.
The ablations are as follows. Invariant-only features remove vector channels entirely. Global-frame vector features are trained with random-rotation augmentation. Transport is applied in message passing but not along trajectories, and the reverse. The sign-flip equivariance is removed. A per-scene 3DGS or dynamic-Gaussian fit serves as an upper-bound reference for quality and a lower-bound reference for speed.
For memory, the baseline is the identical model trained with standard gradient checkpointing, plus a truncated-backpropagation variant. Both are compared at matched peak memory.
Why not just…?
Why not just feed global positions and rotations into ReSplat-style attention and let the network learn rotation handling? This is the most serious threat, and it is exactly what our strongest baseline does. Rotation-aware behaviour can be learned from data. The argument for building it in is sample efficiency and generalisation to motions rarer than those seen in training, not expressiveness. The decisive check is the gap between the two models as training data is reduced, and on held-out scenes with large articulated rotations. If the gap vanishes at full data, we will report that transport mainly buys data efficiency.
Why not just use mesh-based gauge networks such as GEM-CNN on a reconstructed surface? Those methods need a clean manifold mesh and precomputed transport (GEM-CNNs). Extracting and re-meshing a surface every frame from noisy monocular Gaussians would both add latency and fail on the non-manifold, holed geometry that the file flags as a known weakness of geodesic solvers. We will not run this as a full baseline. We will report how often a per-frame mesh extraction on our test scenes even yields a manifold.
Why not just use gradient checkpointing instead of reversible updates? Checkpointing is simple, exact, and its memory grows only with the square root of the unroll length, at a cost of about one extra forward pass. That compute cost is comparable to the 33% overhead RevSNN reports for reversibility. At short unrolls it may match or beat reversible training, and we will say so if it does. Reversibility should only win at long $T \cdot K$, where checkpointing's memory still grows. We will plot peak memory and wall-clock time against unroll length for both, and keep reversibility only in the regime where it is clearly better. Truncated backpropagation is cheaper still, but cuts credit assignment across frames. We expect it to hurt consistency over long horizons, and will measure this.
Proposed Experiments
All quality evaluations use the effectively-monocular protocol and masked metrics of Monocular Dynamic View Synthesis: A Reality Check. We report PSNR, SSIM, and LPIPS on held-out views, a temporal-flicker score computed from rendered frame differences after warping by predicted flow, and reconstruction latency per frame. For a geometry-only sanity check of the transport layer, we include the FAUST correspondence task used by GEM-CNNs. We treat mesh vertices as Gaussians with frames taken from normals and principal curvature directions, which tests whether primitive-derived frames lose anything relative to precomputed mesh transport.
The decisive check is the comparison between transported state and global-frame state at matched compute. We break it down by per-Gaussian rotation magnitude. The proposal predicts the gap should grow with rotation. If it is flat, the transport idea is not doing the work, and the result reduces to an architecture tweak.
A second check is feature stability across frames versus rotation, where transported features should stay stable while global ones decay. A third check is the memory scaling curve against checkpointing. The fourth is gradient fidelity of reversible reconstruction over long unrolls: if the relative error of the gradients exceeds a few percent at the target $T \cdot K$, we fall back to checkpointing and present the method without reversibility.
References
- Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs
- Gauge Equivariant Transformer
- Gauge Equivariant Convolutional Networks and the Icosahedral CNN
- VBA: Vector Bundle Attention for Intrinsically Geometric Representation Learning
- Bundle Neural Network for message diffusion on graphs
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
- DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos
- ReSplat: Learning Recurrent Gaussian Splats
- Monocular Dynamic View Synthesis: A Reality Check
- Memory-Efficient Reversible Spiking Neural Networks
- Sequence Accumulation and Beyond: Infinite Context Length on Single GPU and Large Clusters