Compressing Relightable Gaussian Scenes Without Re-Baking the Lighting

Relightable 3D scenes can be reconstructed from photos by splitting each surface's appearance into its own material properties (base color, roughness, how metallic it is, surface direction) and the light falling on it. This split is what lets a captured object be dropped into a new lighting environment. These scenes are large. Compression is the obvious fix, and our central claim is that the obvious fix quietly breaks the split. Compressors for 3D scenes are tuned, and often fine-tuned, to reproduce the training photographs. Under the fixed lighting of those photographs, an error introduced in the surface directions can be hidden by painting the missing shading into the base color. The image still scores well, but the lighting has been baked back into the material, and the error shows up only when the scene is relit.

We propose a codec for relightable Gaussian scenes that addresses this directly. Its error budget for each material channel comes from how much that channel changes the rendered image averaged over many possible lightings, rather than under the training lighting. It stores those channels as fixed-length packed codes that the GPU unpacks inside the shading pass, at the moment it reads a pixel's surface attributes. The practical goal is scenes an order of magnitude smaller in video memory that stay relightable at interactive frame rates. The scientific goal is to show, and to measure, that standard compression pipelines reintroduce baked-in lighting artifacts.

Background and the gap

A 3D Gaussian scene represents the world as millions of small, soft, colored ellipsoids that are projected to the screen and blended. That makes rendering fast, but storage is heavy. The two surveys in the file, 3DGS.zip and Compression in 3D Gaussian Splatting, report 400 MB to several GB per scene. Both find that the view-dependent color coefficients take about 81% of the storage and that elaborate entropy decoders can drag down frame rates. KISS-GS reaches 85x–319x reduction by packing attributes into 2D image grids and then fine-tuning the scene against the codec, but it decodes on the CPU before rendering. All of this work compresses unlit radiance fields, where each Gaussian stores the color it shows under the capture lighting.

Relightable Gaussian methods replace those color coefficients with physical material channels. GS-IR derives normals from depth gradients, decomposes surfaces into the standard microfacet reflectance model, and bakes visibility into grids. Phys3DGS shows that blending many Gaussians per pixel lets buried Gaussians with wrong normals corrupt shading. It therefore shades once per pixel at an estimated surface point, running at 42–72 FPS on an RTX 4090. GS³ adds angular reflectance lobes and shadow splatting, and reports that high-frequency highlights "blink" across views. These methods motivate the proposal: they store every material channel uncompressed at full precision, and GS³'s observation tells us that small errors in normals and roughness are exactly the ones that become visible.

GPU-side compression supplies the machinery. Invariant Bit Packing exploits bit positions that rarely change across machine-learning tensors and decodes them at more than 95% of ideal throughput using groups of GPU threads working in lockstep. cuSZ-i and cuSZ enforce per-value error bounds at GPU speed on dense scientific grids. RLFC uses bounded-length integer codes so a light field can be read in random order in 1–3 µs per channel.

Two findings constrain the design. FCBench reports a median lossless ratio of only 1.16 on floating-point data, so the codec has to be lossy. Both surveys warn that variable-length decoders slow rendering, so the codes must have fixed length and support random access. No paper in the file compresses the material channels of a relightable scene, and none evaluates any compressed scene under novel lighting.

Core Claim

Compressing a relightable scene and then fine-tuning it against training photographs will reintroduce baked lighting. Most of this damage can be avoided without extra bits by bounding each channel's compression error according to its effect on the rendered image averaged over lightings.

We make this precise as follows. Let $\theta_c$ be the value of material channel $c$ for a Gaussian, $I(\theta; L)$ the deferred-shaded pixel under environment lighting $L$, and $p(L)$ a broad distribution of environment maps. We define the lighting-averaged sensitivity of channel $c$ as

$$w_c = \mathbb{E}_{L \sim p(L)}\, \mathbb{E}_{\text{pixels}} \left\| \frac{\partial I(\theta; L)}{\partial \theta_c} \right\|^2 .$$

Under a first-order expansion, the expected squared image error from a uniform quantization error of size $\epsilon_c$ scales as $w_c \epsilon_c^2$. We therefore set per-channel bounds $\epsilon_c = \tau / \sqrt{w_c}$ for a single quality target $\tau$, computed per spatial block. The training-lighting version of the same quantity is what current pipelines implicitly optimize. It underweights normals and roughness wherever the capture light happened to be dim or diffuse.

To measure re-baking, we introduce a re-baking index. It is the correlation between the change in recovered albedo (base color) caused by compression plus fine-tuning, and the diffuse shading those Gaussians receive under the training lighting. A value near zero means compression left the decomposition intact. A positive value means shading has leaked back into the base color.

Research Program

We would build three pieces.

The first is a set of channel-specific fixed-rate encodings suited to the physical ranges of material data. Normals would use two-component octahedral codes. Metallicness would use a 1-bit dielectric/metal flag plus a short residual, exploiting the two-cluster distribution noted in the file's gap analysis. Roughness and albedo would use per-block bounded integer codes in the style of RLFC. Each block would carry a small bit-invariance mask in the style of IBP. Bit widths per block come from the $\epsilon_c$ above.

The second is an in-situ decoder fused into a Phys3DGS-style deferred shader. Because shading happens once per pixel, only the Gaussians contributing to the surface hit are fetched and unpacked, in on-chip shared memory, so the full-precision scene never exists in video memory. We would also port the decoder to GS-IR's blended pipeline to test whether the approach works without single-hit shading.

The third is codec-aware fine-tuning. In the proposed version, the fine-tuning loss is taken under environment maps sampled from $p(L)$, and a penalty on the re-baking index is applied against the original decomposition. The analysis first confirms the premise by running KISS-GS-style codec fine-tuning on relightable scenes and measuring the re-baking index and relighting loss. If re-baking does not appear, the paper's contribution narrows to the in-situ decoder, and we would say so.

Baselines

Uncompressed reference. The naive reference is the uncompressed relightable model itself, the Phys3DGS renderer in full precision. This is the quality ceiling and the memory and throughput floor. On the TensoIR Synthetic Lego scene, Phys3DGS reports relighting PSNR of 26.417, against 24.158 for Relit3DGS (Phys3DGS), at 42–72 FPS on an RTX 4090. GS-IR reports view-synthesis PSNR of 35.333 on the same dataset (GS-IR). Staying within 0.5 dB of the uncompressed relighting PSNR at 10x or more lower material memory, with FPS no worse than the uncompressed renderer, would count as success.

Trivial quantization. This baseline stores every channel in FP16 or uniform INT8, with octahedral normals, decoded in the shader. It is the cheapest in-situ competitor and likely strong at low ratios. It gets the same trained scenes and the same renderer.

Strongest existing compressor, ported. We would apply KISS-GS's pipeline to the material channels, with image-grid packing and codec fine-tuning against training photos, and decode it on load. It is the strongest compressor in the file, reporting 738 MB to 3.2 MB on radiance-only scenes. No reported result found on Lacuna for any compressor applied to relightable Gaussian material channels, so we must run this port ourselves. It gets the same fine-tuning iteration budget as our method. Beating it means equal or smaller size with higher relighting PSNR under held-out lighting and a lower re-baking index. We do not expect to beat it on training-view PSNR.

Ablations. Two ablations isolate the key ingredient. The first uses training-lighting sensitivities in place of $w_c$. The second uses our bit allocation but fine-tunes under training lighting only. Every method uses identical trained starting scenes, GPU, resolution, and hyperparameter search budget.

Why not just…?

Why not just quantize everything to FP16 or INT8? This is a real threat below roughly 4x compression, and it may match us there. We report that honestly. The file's evidence is that uniform quantization tops out at 2–4x and handles unevenly distributed attributes poorly (Compression in 3DGS survey), and GS³ reports that normal and roughness errors produce blinking highlights. The rate–distortion curve under novel lighting settles the question. If INT8 with octahedral normals lies on our curve at every ratio above 10x, the bit-allocation claim is wrong.

Why not just use an existing compressor and decompress on load? Decompressing on load recovers file size but none of the video-memory savings. The surveys note that large scenes exceed 15 GB. The deeper problem is that these pipelines fine-tune against training photos, which is exactly what we hypothesize causes re-baking. The KISS-GS-port baseline tests this with the re-baking index. If the index stays near zero for that port, our diagnosis fails.

Why not just fine-tune under randomized lighting and skip the analytic bounds? This is the most serious alternative, because randomized-light fine-tuning may capture most of the benefit. We therefore treat it as part of the method rather than a rival, and ablate it explicitly. Keeping the sensitivity-derived allocation is justified only if it gives smaller files at matched relighting PSNR. We also expect the allocation to set bit widths before any fine-tuning, which matters when scenes must be streamed without a per-scene optimization step.

Why not bake materials into a mesh with hardware block-compressed textures? Extracting Triangular 3D Models does exactly this and inherits the GPU's texture compression. Meshes lose thin structures that Gaussians keep, such as GS-IR's bicycle spokes. We include this pipeline as a secondary comparison on DTU and NeRF Synthetic scenes, reporting relighting quality and memory side by side.

Proposed Experiments

Datasets. The TensoIR Synthetic dataset provides ground-truth albedo and relighting and is the primary testbed. SynCom tests relighting after placing objects into new scenes, with physically rendered ground truth. Mip-NeRF 360 and Tanks and Temples test memory, FPS, and view-synthesis fidelity on large real scenes, where no relighting ground truth exists.

Metrics. We report relighting PSNR, SSIM, and LPIPS under held-out environment maps; albedo PSNR; normal angular error; the re-baking index; material bytes per Gaussian; peak video memory; FPS; and decoder throughput as a fraction of memory bandwidth, the measure IBP uses.

Decisive checks. The first check is premise verification: the KISS-GS port should show a positive re-baking index and a gap between training-lighting and novel-lighting PSNR that grows with compression ratio. The second is the rate–distortion comparison under novel lighting against INT8 and the ablations. The third checks that fused decoding keeps Phys3DGS-level frame rates. The fourth measures temporal flicker of highlights along camera paths, compared with GS³'s reported blinking.

If the premise check fails, we report that finding. If the ablation shows analytic allocation adds nothing over randomized-light fine-tuning, we drop the allocation claim and publish the diagnosis and the decoder.