Artists who want a smoke simulation to look like a painting, a brush texture, or a motif currently run an optimization for every frame. The best methods in the literature push gradients from an image-style loss back through a renderer into the smoke's velocity field. They take between 40 seconds and 13 minutes per frame, and they start from scratch for every new style image. We propose to replace this per-frame optimization with a single trained network. The network reads the current smoke state and outputs a style-specific velocity correction. Its weights are not fixed: a second, small network (a hypernetwork, meaning a network whose output is the weights of another network) generates low-rank weight updates from the style image. The correction is emitted as a vector potential, so its curl is divergence-free by construction. That property holds no matter what weights the hypernetwork produces.
In two sentences: we amortize transport-based fluid style transfer into one forward pass, and we let the style image rewrite the internal weights of the flow operator rather than enter as an extra input channel. We expect feed-forward stylization at interactive rates whose style fidelity approaches per-frame optimization. Incompressibility is guaranteed by construction, and the stylizer should generalize to style images never seen in training.
Core Claim
The claim has two parts, and each can be falsified on its own. The first part concerns amortization. The objective that Transport-Based Neural Style Transfer for Smoke Simulations (TNST) optimizes per frame can be learned by a style-conditioned neural operator trained across many simulations and many styles. The objective combines Gram-matrix style losses, rendered from multiple views, with a velocity field built from potentials. The trained operator's residual "amortization gap" on held-out styles should be small enough that a few optional refinement steps close it. The second part concerns conditioning. Style is a high-dimensional, highly varied context, so injecting it as generated low-rank weight updates at every layer should generalize to unseen styles better than concatenating a style code to the input or applying per-channel scale-and-shift.
TNST notes that a feed-forward proxy is an open problem. Lagrangian Neural Style Transfer for Fluids moves the optimization onto particles for a 20x speedup and notes that real-time 3D neural stylization remains unachieved. Neither work generates weights or trains anything that transfers across styles.
How This Relates to Prior Work
The fluid stylization line supplies our objective and our output parameterization. TNST shows that editing density voxels directly produces ghosting and disconnected clumps. It also shows that stylizing the velocity, split into an irrotational potential $\Phi$ for sharp motifs and a vector potential $\Psi$ for swirling textures, keeps the result plausible. The Lagrangian method and Neural Smoke Stylization with Color Transfer extend this to particles and color. All three remain test-time optimizers. We keep their loss and their $\Psi$ parameterization but train a predictor for the solution.
The dynamic-weight line supplies the conditioning mechanism and the case for why it matters. In Extending Fourier Neural Operators for Modeling Parameterized and Coupled PDEs, a hypernetwork injects layer-wise shifts into a Fourier neural operator (a network that learns PDE solution maps by mixing features in frequency space). That paper reports 55% to 72% lower errors than baselines, including input concatenation. Its context, however, is a handful of physical constants, not an image. Meta-learning Structure-Preserving Dynamics shows that low-rank modulation, down to predicting singular values over fixed bases, can preserve physical structure. That work stays on ODEs and states that fluid PDEs are unaddressed. Text-to-LoRA: Instant Transformer Adaption shows that a hypernetwork can map a free-form description to working low-rank adapters in one pass. It also reports sensitivity to ambiguous context encodings, which warns us that style encoding may be the weak point.
Our structure guarantee is simpler than the Meta-learning approach. We do not constrain the generated weights. We constrain the output: whatever weights are produced, the operator emits $\Psi$, and the stylization velocity is its curl.
The interactive-control line motivates the setting and marks the boundary of our claim. Hybrid Neural-MPM for Interactive Fluid Simulations in Real-Time uses a diffusion ControlNet that turns sketches into external force fields for a fixed-weight solver. It falls back to a classical solver when the neural one drifts. We address appearance stylization, not trajectory guidance, and we keep the base simulation classical. Our network only adds a correction, which sidesteps the drift problem that forced their safeguard.
Research Program
The per-frame model works as follows. At frame $t$ we have smoke density $d_t$ and simulated velocity $u_t$ on a grid, and a style image $s$. A frozen perceptual encoder (the VGG-19 features the Lagrangian method already uses) summarizes $s$ into a compressed style code $c$, built from channel means, variances, and a projected Gram matrix. A Fourier-neural-operator-style backbone $f$ maps $(d_t, u_t)$ to a vector potential $\Psi_t$. Each spectral and pointwise layer $\ell$ has shared weights $W_\ell$ plus a generated update:
$$W_\ell(c) = W_\ell + U_\ell\,\mathrm{diag}(\sigma_\ell(c))\,V_\ell^\top,$$
Here $U_\ell, V_\ell$ are learned rank-$r$ bases shared across styles, and $\sigma_\ell(c)$ is the hypernetwork's output for layer $\ell$. This is the SVD-like modulation of the Meta-learning paper moved into a PDE operator. The stylized density is advected by $u_t + \nabla\times\Psi_t$. Training minimizes TNST's multi-view style loss on renders of the advected density, plus a temporal term that penalizes disagreement between frame $t$'s stylization and frame $t-1$'s stylization carried forward by the flow. Training data are synthetic buoyant plumes from a standard grid solver, together with the real-world reconstructions in ScalarFlow. ScalarFlow provides 100 sequences, each at least 150 steps at 100×177×100. The style images come from a large painting and texture pool, with a held-out split of unseen styles and unseen style categories.
The plan runs in three stages. We first establish the 2D case, where per-frame optimization is cheap enough to compute exact per-style optima as a reference. We then move to 3D at ScalarFlow resolution. Finally we analyze how rank $r$ and the layers that receive modulation trade off against held-out style fidelity. That analysis addresses the open question from the file of how much rank stylization actually needs. An optional mode runs a few TNST gradient steps initialized from the network's output. We report it separately, because it tests whether amortization serves as a good warm start even where it is not a full replacement.
Baselines
The naive reference is the unstylized simulation rendered as-is. It anchors the style-loss scale and gives the ceiling for density preservation.
The strongest existing method is per-frame Lagrangian neural style transfer (Lagrangian Neural Style Transfer for Fluids), which reports 0.66 minutes per frame on its Smoke Jet scene, down from 14 minutes for Eulerian TNST. Eulerian TNST itself reports roughly 12 to 13 minutes per frame at 200×300×200 on a TITAN Xp. We found no reported style-fidelity number on Lacuna for either method, only runtimes. We will therefore rerun both on our scenes, on the same GPU, with the same style losses, views, and renderer. Each gets its published iteration budget, plus a sweep of reduced budgets that traces quality against wall-clock time. Eulerian TNST with our $\Psi$-only setting is the exact per-frame optimum of our training objective, which makes it the cleanest reference. We will count ourselves as beating it if, on unseen styles, we come within a pre-registered margin of its style loss at a small fraction of its per-frame time, and if we dominate every point of its reduced-budget curve at matched time.
To isolate the key ingredient, the conditioning ablations use the identical backbone, data, and tuning budget. The first concatenates the style code $c$ to the input channels, the shallow augmentation that hpFNO beat. The second applies per-channel scale-and-shift from $c$, the kind of feature modulation used in Deep Task-specific Bottom Representation Network for Multi-Task Recommendation and Meta-Inductive Node Classification across Graphs. The third trains a separate network per style, which serves as an oracle for seen styles. A final ablation replaces the $\Psi$ output with direct velocity prediction, which isolates the value of the structural guarantee.
Why not just…?
Why not just run the optimizer for fewer iterations? This is a real threat for any single style, since TNST-style losses may converge fast from a good start. The quality-versus-time sweep against Lagrangian and Eulerian optimization settles it directly. If truncated optimization matches our quality at comparable time, we will report that amortization buys only generality, not speed.
Why not just use scale-and-shift conditioning instead of generated low-rank weights? Per-channel modulation is simpler and may suffice. Arbitrary 2D style transfer is often done this way, and the file contains no fluid-specific evidence either way. The ablation at matched parameters and tuning decides it. If scale-and-shift matches low-rank generation on unseen styles, the paper's contribution narrows to amortized, divergence-free stylization, and we will say so rather than defend the hypernetwork.
Why not just stylize the rendered 2D frames with a feed-forward image network? TNST and the Lagrangian method both document that edits not carried by the flow flicker and "swim" across frames. Frame-wise edits are also inconsistent across camera views and give no 3D volume for relighting. We will include a frame-wise image stylizer and measure temporal warp error and multi-view consistency to confirm this on our scenes.
Why not just concatenate the style code to the input? For physical parameters, hpFNO reports that layer-wise hypernetwork injection beats shallow augmentation by 55% to 72% in error. We expect the gap to widen for style codes, which are richer, but the concatenation baseline tests that expectation rather than assuming it.
Proposed Experiments
The main experiment trains on synthetic plumes plus ScalarFlow and evaluates on held-out sequences crossed with seen styles, unseen styles, and unseen style categories. We report style loss on held-out camera views, density preservation relative to the input, temporal warp error, maximum and mean divergence of the stylization velocity, and seconds per frame. On the unseen-category split, a small preference study with artists compares our output with per-frame TNST.
Three results decide the proposal. The first is the gap to per-frame TNST on unseen styles at matched views. It must be small in absolute terms and must beat truncated optimization at equal time. The second is the unseen-style gap between low-rank generation and the scale-and-shift and concatenation ablations. This is what justifies weight generation. The third is the $\Psi$-versus-direct-velocity ablation. The direct variant should show measurable divergence and more ghosting, while ours stays divergence-free to solver precision. A secondary check sweeps $r$ from 1 to 32 and reports where held-out fidelity saturates. Low saturation would echo the Meta-learning paper's finding that a rank-5 update was often enough, and would be evidence that style variation also lives in a small weight subspace.
References
- Transport-Based Neural Style Transfer for Smoke Simulations
- Lagrangian Neural Style Transfer for Fluids
- Neural Smoke Stylization with Color Transfer
- Extending Fourier Neural Operators for Modeling Parameterized and Coupled PDEs
- Meta-learning Structure-Preserving Dynamics
- Text-to-LoRA: Instant Transformer Adaption
- Hybrid Neural-MPM for Interactive Fluid Simulations in Real-Time
- ScalarFlow: a large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning
- Deep Task-specific Bottom Representation Network for Multi-Task Recommendation
- Meta-Inductive Node Classification across Graphs