Dynamic 3D scene reconstruction from multi-view video commonly represents the scene as a cloud of small colored blobs ("3D Gaussians") that move over time. The most accurate trackers give each blob a fixed identity from the first frame and let it move. Because no blob can be created or destroyed afterwards, a ball tossed into view mid-sequence has no blobs to represent it. The natural repair is to let the blob set change freely from frame to frame and then decide, after each frame, which new blobs continue old ones, which are newly born, and which old ones have died. This is a matching problem between two point sets of unequal size, and optimal transport (OT), the mathematics of moving mass between two distributions at minimum cost, is the standard tool for it. The practical version adds an entropy penalty and is solved with the fast Sinkhorn algorithm. That penalty spreads each blob's match across many neighbours, so motions collapse toward local averages and colors carried along a match get blurred. Removing the penalty gives a matching with no useful gradients. Differentiating through the iterative solver costs memory that grows with the number of iterations and the problem size, and Gaussian scenes contain about $10^5$ blobs.
We propose to link consecutive frames' Gaussians with unbalanced, quadratically regularized OT restricted to a sparse neighbour graph. The quadratic penalty, unlike entropy, yields exactly sparse, near one-to-one matchings. Unmatched mass is read directly as births and deaths. The matching cost is predicted by a small network, and its gradients come from a linear system defined only on the handful of matched pairs, solved without forming any matrix. The bet is that one choice fixes two problems: sparsity removes the entropic averaging, and the same sparsity makes exact implicit gradients cheap. If it holds, dynamic Gaussian reconstruction could support objects that enter and leave a scene without giving up the tracking accuracy of persistent-identity methods.
Core Claim
Entropic bias and differentiation overhead are usually treated as separate costs. In this setting they come from the same source: a dense transport plan. Debiaser Beware shows that entropic plans estimate conditional means rather than point-to-point maps. Displacements shrink toward the local mean as the regularization strength grows, which in a moving scene is exactly what blurs motion near boundaries and on thin parts. Debiased Sinkhorn barycenters shows that this smoothing is a property of the objective, not of stopping the solver early, so more Sinkhorn iterations will not remove it. A dense plan also makes the backward pass expensive. Unrolling Sinkhorn, as the differentiable particle filter of Differentiable Particle Filtering via Entropy-Regularized Optimal Transport does, has only been demonstrated with at most about 100 particles. Implicit differentiation of a dense plan requires a linear system over all $N \times M$ entries.
Our claim is that a sparse quadratic regularizer changes both properties together. The forward problem, posed over candidate edges $E$ (the $k$ nearest neighbours in position-plus-feature space), is
$$ \pi^\star(\theta)=\arg\min_{\pi\ge 0,\ \mathrm{supp}(\pi)\subseteq E}\ \sum_{(i,j)\in E}\pi_{ij}C_{ij}(\theta)+\frac{\gamma}{2}\|\pi\|^2+\frac{\lambda}{2}\left(\|\pi\mathbf 1-a\|^2+\|\pi^\top\mathbf 1-b\|^2\right). $$
Here $C_{ij}(\theta)$ is the learned cost of linking Gaussian $i$ at time $t$ to Gaussian $j$ at time $t{+}1$, and $a,b$ are their masses (opacity-weighted). The weight $\gamma>0$ sets how strongly the plan is regularized, and $\lambda$ sets how much unmatched mass (birth or death) is tolerated.
This is a nonnegative least-squares problem, precisely the form of unbalanced OT developed in Unbalanced Optimal Transport through Non-negative Penalized Linear Regression. We reuse that formulation for the forward solve. Two properties follow. First, the solution is supported on a small set $S$ of edges, roughly $N+M$ of them, so each Gaussian links to one or two successors rather than to a smear. Second, wherever the support is locally stable, the plan on $S$ satisfies a linear equation:
$$ \left(\gamma I+\lambda A_S^\top A_S\right)\pi_S=\lambda A_S^\top\begin{bmatrix}a\\ b\end{bmatrix}-C_S, $$
where $A_S$ is the incidence matrix of the bipartite graph formed by the support edges. The gradient of any downstream loss with respect to the cost therefore needs only one solve with a sparse, symmetric positive-definite matrix of size $|S|$. Conjugate gradients can perform it using only matrix–vector products.
This is the active-constraint reduction of Decision-Focused Learning via Tangent-Space Projection of Prediction Error, carried over to the transport polytope. That paper reduces the gradient system to the active constraints and needs curvature in the objective, which the quadratic term supplies. The solve is matrix-free, in the style of Matrix-Free Least Squares Solvers, which reports reverse-mode adjoints "5 to 10 times faster than unrolling the solver's forward pass." Neither paper touches transport or rendering. The contribution here is to notice that sparse transport is exactly the regime where their tools apply.
Research Program
The reconstruction runs in streaming fashion over multi-view video. At each new frame, the Gaussians from frame $t$ are copied forward. A short photometric fit then refines them, and splitting, cloning and pruning are allowed without restriction, so identities are not preserved. A link layer then solves the problem above between the frame-$t$ set and the frame-$(t{+}1)$ set. The cost $C_{ij}(\theta)$ combines squared position distance with the distance between per-Gaussian learned feature vectors produced by a small network. The feature vectors play the same role as the features in Split4D, but here they feed a matching rather than a contrastive clustering.
The resulting plan is used in three ways. Mass on matched edges transfers appearance (color coefficients, scale) from old to new Gaussians and defines per-Gaussian motion. Over matched pairs we apply the local rigidity and isometry terms of Dynamic 3D Gaussians. Unmatched mass in frame $t{+}1$ marks births and unmatched mass in frame $t$ marks deaths. The rendering loss over a sliding window of frames depends on the transferred attributes, so it depends on the plan. Its gradient flows through the reduced system above into the cost network. This is a predict-and-optimize loop in the sense of the End-to-End Constrained Optimization Learning survey: the network predicts a cost, a solver turns it into a decision (the matching), and the task loss (rendering error) trains the predictor.
Three engineering pieces make this tractable at $10^5$ Gaussians. The first is the $k$-nearest-neighbour candidate graph, in the same spirit as the KD-tree partitioning in Gaussian Herding across Pens and the memory-light kernels in Accurate Point Cloud Registration with Robust Optimal Transport. The second is warm-starting each forward solve from the previous iteration's support. This borrows the caching logic of Contrastive Losses and Solution Caching for Predict-and-Optimize, which matched full-solver quality while calling the solver only "5% of the time." The third is a matrix-free conjugate-gradient backward solve. When the support changes between iterations, the gradient is a one-sided derivative; we will log how often this happens and whether it destabilizes training.
The analysis component is a direct measurement of entropic centering bias in a scene-reconstruction setting. For each Gaussian, we stratify tracking error by distance to the nearest motion discontinuity, for both entropic and sparse plans across a sweep of regularization strengths. The prediction from Debiaser Beware is that entropic error rises sharply near boundaries and on thin parts, while sparse error stays roughly flat.
Baselines
The naive reference is persistent-identity tracking with no linking at all: Dynamic 3D Gaussians, with its first-frame Gaussian set and local rigidity, rotation and isometry terms. On PanopticSports it reports 28.7 PSNR, a median 2D tracking error of 1.57 pixels, and a 3D tracking error of 2.21 cm averaged over 5-second sequences, with training taking about two hours on a single GPU. These are the numbers to beat on sequences where every object is present from the first frame. On those sequences we expect, at best, to match them. We run it with its released settings on identical camera splits. On sequences with entering objects, beating it means lower post-entry PSNR error and lower post-entry tracking error, with no more than a 0.2 cm regression in 3D tracking error on the persistent objects.
The strongest existing transport competitor is the entropic Sinkhorn link: the same pipeline and the same candidate graph, but with entropy in place of the quadratic term and unbalanced KL marginals as in RobOT, differentiated by unrolling as in the differentiable particle filter. We also run it with the annealed warm-started schedule of Progressive Entropic Optimal Transport Solvers, so that it can approach small regularization stably. For fairness, both methods receive the same total GPU-hour budget per scene, and the regularization strength of each is tuned on the same held-out validation sequences using the same number of trials. Beating it means lower boundary-stratified tracking error at equal wall-clock time, together with lower peak memory in the backward pass. No reported result was found on Lacuna for entropic OT linking inside a 4D Gaussian pipeline, so this baseline's numbers will be produced here.
The ablations isolate the key ingredients. The first keeps the sparse plan but differentiates it by unrolling the active-set iterations. It measures what the reduced-system gradient buys in time and memory, holding accuracy fixed. The second keeps the sparse solver but freezes the cost at plain position distance with no learned features. It tests whether the predict-and-optimize gradient matters at all. The third sets $\lambda\to\infty$ (forcing balanced transport, so no births or deaths are possible). It tests whether reading unmatched mass as births and deaths is what handles entering objects.
Why not just…?
Why not just keep identities fixed and densify where new objects appear? This is the most serious threat. Persistent Gaussians plus unrestricted densification might handle entering objects with no matching layer. The difficulty is that densification spawns Gaussians wherever photometric error spikes. RobustSplat documents how this mechanism creates Gaussians that remain frozen in space after a moving object leaves. In that baseline, new Gaussians also have no defined motion history, so the rigidity terms cannot apply to them. We implement this variant explicitly, as Dynamic 3D Gaussians with per-frame densification. If it matches our post-entry tracking error, the transport layer is unnecessary and we will report that.
Why not just anneal Sinkhorn to a tiny regularization? Annealing does sharpen the plan, and Progressive Entropic Optimal Transport Solvers makes it stable, so this is a real competitor on forward accuracy. It does not remove the backward cost. Unrolling many small-regularization iterations stores every intermediate state. Implicit differentiation of an entropic plan involves every candidate edge, because no entry is exactly zero. The equal-budget comparison above settles the question. If annealed Sinkhorn reaches our boundary-stratified error within the same time and memory budget, the sparse-regularizer argument fails.
Why not just use nearest-neighbour matching, or a sliced-Wasserstein loss? Nearest-neighbour linking is cheap but not one-to-one. RobOT identifies the resulting cluster collapse as the reason to prefer transport in Accurate Point Cloud Registration with Robust Optimal Transport. We include it as a baseline and measure how many Gaussians are claimed by more than one successor. The sliced-Wasserstein regularizer of Improving the Convergence of Dynamic NeRFs via Optimal Transport matches pixel-color distributions. It does not produce 3D correspondences, and its authors note it assumes smooth temporal change and fails on sudden appearances. It is complementary rather than a substitute, so we add it on top of both our method and the entropic baseline to check that our gains persist.
Why learn the cost at all? A fixed position-and-color cost may already give good links. In that case the decision-focused gradient, which is the part motivated by optimization overhead, is ornamental. The frozen-cost ablation tests this directly. If it matches the learned cost, the contribution reduces to "sparse unbalanced transport for linking," which is still useful but narrower, and we would say so.
Proposed Experiments
Experiments run on three benchmarks. The first is PanopticSports from Dynamic 3D Gaussians, measuring PSNR and 2D and 3D tracking error. The second is the D-NeRF synthetic scenes from D-NeRF; their thin articulated parts, such as Mutant (31.29 PSNR for D-NeRF), are where entropic smearing should show most. The third is a set of multi-view Kubric sequences, generated with the same generator used in the RobustNeRF benchmark, in which objects enter and exit and ground-truth 3D trajectories are known. No reported result was found on Lacuna for tracking on such sequences. As a secondary check of birth and death detection, we compare the connected components of matched mass on the Multi-Human sequences against the decomposition of Split4D, which reports 0.893 mIoU.
Four checks decide the question. The first is the boundary-stratified tracking-error curve. The method succeeds only if sparse plans flatten the rise in error near motion discontinuities that entropic plans show, at matched regularization strength. The second is a scaling plot of backward time and peak memory against Gaussian count, from $10^4$ to $10^6$, for unrolled Sinkhorn, implicit entropic gradients, unrolled sparse gradients, and the reduced-system gradient. It must show the reduced system growing with support size rather than with edge count or iteration count. The third is post-entry rendering and tracking on the Kubric entering-object sequences against persistent identities with densification. The fourth is the rate of support changes during training and its correlation with loss spikes, which tests whether one-sided gradients are a practical problem. If the first check fails, the bias argument is wrong. If the third check is matched by the densification baseline, the application argument is wrong. Either outcome is informative and will be reported as such.
References
- Debiaser Beware: Pitfalls of Centering Regularized Transport Maps
- Debiased Sinkhorn barycenters
- Differentiable Particle Filtering via Entropy-Regularized Optimal Transport
- Unbalanced Optimal Transport through Non-negative Penalized Linear Regression
- Decision-Focused Learning via Tangent-Space Projection of Prediction Error
- Matrix-Free Least Squares Solvers: Values, Gradients, and What to Do With Them
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis
- End-to-End Constrained Optimization Learning: A Survey
- Gaussian Herding across Pens: An Optimal Transport Perspective on Global Gaussian Reduction for 3DGS
- Accurate Point Cloud Registration with Robust Optimal Transport
- Contrastive Losses and Solution Caching for Predict-and-Optimize
- Progressive Entropic Optimal Transport Solvers
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Improving the Convergence of Dynamic NeRFs via Optimal Transport
- D-NeRF: Neural Radiance Fields for Dynamic Scenes
- RobustNeRF: Ignoring Distractors with Robust Losses