A 3D scene captured as a radiance field is a cloud of small colored blobs. 3D Gaussian Splatting (3DGS) is the standard example: each "Gaussian" is a fuzzy ellipsoid whose color changes with viewing angle. Owners of valuable or sensitive captures, such as a product scan, a heritage site, or a private interior, increasingly want an outside host to serve views of the scene without the host ever reading the asset. Fully homomorphic encryption (FHE) lets a server compute on encrypted numbers. It is usually dismissed for graphics because its cost explodes whenever encrypted values must be multiplied together. When too many multiplications stack up, the ciphertext's built-in noise must be reset by bootstrapping, a step that is slow even by FHE standards. This note observes that the right split between public and encrypted parameters avoids that explosion. If geometry and the angular basis functions are public, and only the per-blob weights on those functions are encrypted, then rendering a frame for a known camera is a linear map applied to ciphertexts. No two encrypted values ever multiply, so no bootstrapping is needed. The cost scales only with how many basis terms each blob carries.
This turns a graphics question into a cryptographic cost question. Graphics treats low-degree angular bases as a compromise. Spherical harmonics (SH), the standard choice, are smooth functions on the sphere, and keeping only low degrees blurs out sharp reflections. Under encryption, every extra basis term is an extra encrypted coefficient that must be stored, moved, and multiplied. The research question is therefore which public basis buys the most reflective detail per encrypted coefficient. We compare truncated SH, a small shared learned basis, and sharp reflection lobes whose shapes are public. Because the rendering map is public, cheap, and deterministic, it also supports a cryptographic version of the editorial firewall. An independent custodian holds half of the decryption key, recomputes the host's output, and releases decryption shares only for ciphertexts that are verifiably rendered images, never for the asset itself.
Core Claim
Write the appearance of Gaussian $i$ seen from direction $d$ as $c_i(d)=\sum_{k=1}^{K} a_{ik}B_k(d;\theta_{ik})$. Here $a_{ik}$ are weights, $B_k$ are angular basis functions with shape parameters $\theta_{ik}$, and $K$ is the number of terms. Splatting composites Gaussians front to back with weights $w_{pi}$ that depend only on geometry, opacity and camera. The pixel value is therefore
$$I_p=\sum_i w_{pi}\sum_{k} B_k(d_{pi};\theta_{ik})\,a_{ik}.$$
When geometry, opacities, $\theta$ and the camera are public, every factor except $a_{ik}$ is a plaintext number. The whole image is then a sparse plaintext matrix times the encrypted weight vector. Under the CKKS scheme (an FHE scheme for approximate real arithmetic), that costs plaintext–ciphertext products and slot rotations, with multiplicative depth one and no bootstrapping. This holds whether $B_k$ is a degree-3 harmonic, a learned function, or an anisotropic spherical Gaussian lobe, a sharp, stretched highlight shape, provided its axis and sharpness are public.
The claim has two halves. The engineering half says encrypted 3DGS rendering is practical at interactive-to-offline speeds when this split is respected. The scientific half says the quality-per-encrypted-coefficient frontier, not multiplicative depth, is the quantity to optimize. We expect shared learned bases and public-lobe reflection models to dominate truncated SH on that frontier for shiny materials.
Positioning Against Prior Work
Two papers hide information inside 3DGS, and both motivate the gap while illustrating what to avoid. GS-Hider replaces SH colors with a 16-dimensional feature splatted and decoded by public and private convolutional decoders. Its authors note that it loses view-dependent specular quality. SecureGS hides attributes in private MLP weights over an anchor representation and adds dense cover anchors to mask geometric leakage. Both are steganography: confidentiality rests on heuristic invisibility, not on a hard lattice problem. Both also route appearance through deep nonlinear decoders, which would be very expensive to evaluate under FHE. We keep the opposite design: the appearance path is strictly linear, so the encrypted computation stays shallow.
The FHE literature supplies the cost model and constraints. CryptoNets showed that packing many values into one ciphertext's slots buys throughput but not latency. SAFENet and Activation-Free Backbones show how much effort goes into making networks low-degree polynomials. We sidestep that effort entirely because rendering with public geometry has no nonlinearity in the encrypted variables. Circuit Bootstrapping documents the cost we aim never to pay. SecDOOD encrypts only salient feature channels to cut overhead. Our split is a principled version of that idea: we encrypt exactly the parameters that enter the output linearly.
The graphics literature supplies the candidate bases. Standard 3DGS uses degree-3 SH, which is 48 coefficients per Gaussian (GS-Hider). NeX expresses color as weights on a small set of learned global basis functions; those functions can be public while the weights stay secret. Spec-Gaussian adds anisotropic spherical Gaussian lobes on top of low-order SH, but feeds them through an MLP, which breaks linearity. We test a variant that keeps the lobes but drops the MLP, publishing lobe axes and sharpness and encrypting only lobe amplitudes. Neural-PIL is relevant as a cautionary contrast, since pre-integrated MLP lighting is fast in plaintext but deep under encryption.
The governance literature supplies the threat model rather than a technique. Venues route leaders' submissions through independent third parties (Preface), fully isolate conflicted editors (The ASHES 2020 special issue at JCEN), and redact deliberations from conflicted parties (ISMAR 2019 chairs' message). Legal scholarship argues for statutory independent bodies to separate moderation from interested parties (Recalibrating Free Speech). All of these enforce separation administratively. Our custodian is a narrow, verifiable instance of the same principle, built on threshold key setups from Multiparty Homomorphic Encryption. We do not claim to fix peer review. The analogy dictates who holds which key and what each party may release.
Research Program
The first component is the representation study. We train 3DGS scenes with four appearance families, each under a matched budget of $K$ encrypted coefficients per color channel. The first family is truncated SH at degrees 0 through 3, giving $K$ of 1, 4, 9 and 16. The second is a NeX-style basis of $K$ learned global functions, implemented as a small MLP over direction whose outputs are rendered as plaintext tables. The third is a public-lobe model: degree-1 SH plus $K-4$ anisotropic lobes with public axes and sharpness and encrypted amplitudes. The fourth is a hybrid where diffuse color is public and only the view-dependent residual is encrypted.
All models train in plaintext, where the owner holds everything. The basis parameters are then frozen and published, and the weights are encrypted. We will also measure how much "floater" geometry each basis induces, since Spec-Gaussian reports that weak angular bases push the optimizer to add spurious Gaussians. Here those floaters are public geometry and therefore leakage as well as artifacts.
The second component is the encrypted renderer. The image map $I = WA$ is sparse, with $W$ a plaintext matrix and $A$ the encrypted weight vector. Under CKKS, the dominant cost is rotations that move coefficients into the slots of the pixels they affect, not the multiplications themselves. We will design a packing aligned with 3DGS's 16×16 screen-tile rasterization, so that each ciphertext holds the coefficients of Gaussians touching one tile. Rotation counts then scale with per-tile overlap rather than image size. We will report how CKKS approximation error accumulates over long compositing sums. We will also report a cost model in which latency and bandwidth grow linearly in $K$ and in mean Gaussians per pixel, and validate it against measurement.
The third component is the custodian protocol. The decryption key is split two-of-two between the owner and an independent custodian. The host computes the encrypted frame. The custodian recomputes the same public linear map from the public ciphertexts and the requested camera, checks that the result matches, and only then contributes its decryption share. Because the computation has depth one, recomputation costs about as much as hosting and stays far below bootstrapped inference. We will measure that overhead.
The fourth component is a leakage audit, because the scheme deliberately publishes geometry, opacity and basis shapes. We train an attacker network to predict colors and appearance weights from public parameters alone, and report the PSNR of attacker renders. This tells owners exactly what they are not protecting.
Baselines
The naive reference is no encryption: the same scenes rendered in plaintext with standard degree-3 SH 3DGS. It sets the quality ceiling for the SH family and the zero-overhead cost floor. We run it on identical training schedules and hardware, and every encrypted variant is reported as a quality drop and slowdown relative to it.
The strongest plaintext quality competitor is Spec-Gaussian. It reports 37.70 dB PSNR on its Anisotropic Synthetic dataset against 33.82 dB for standard 3DGS. It is the best-documented model for sharp anisotropic highlights in the file, and the gap between those numbers is exactly what a cheap public basis would need to recover. We retrain it with the authors' configuration on the same scenes and treat it as an upper reference that cannot be encrypted cheaply. Beating standard 3DGS's 33.82 dB on that dataset with an encrypted model whose $K \le 16$ would count as success. Recovering half the gap to 37.70 dB at $K=9$ would count as strong success.
The strongest existing privacy-for-3DGS method is SecureGS. It reports a 1.16 dB overt-scene PSNR gain over GS-Hider, which itself reports 25.817 dB average on Mip-NeRF 360, and runs at 131 FPS. It addresses a different goal, hiding a secret inside a visible scene, so we compare on what is shared: fidelity of the protected content on Mip-NeRF 360 and the strength of the guarantee. We expect to lose badly on speed. The claim is a lattice-based guarantee at a stated cost, not a faster method.
The natural encrypted competitor is the most direct transfer of FHE practice. It encrypts standard degree-3 SH coefficients (16 per channel) and renders them with a generic diagonal-method matrix–vector product, without tile-aligned packing. It isolates our packing contribution and the benefit of non-SH bases at equal $K$. No reported result found on Lacuna for encrypted rendering of radiance fields, so we will establish this number ourselves. As the only published encrypted-inference cost anchor, we cite CryptoNets: 250 seconds for a packed batch of 4,096 MNIST images, at about 367.5 MB.
The key ablation makes lobe sharpness private as well. This forces a polynomial approximation of the exponential inside the encrypted circuit. It shows the depth and bootstrapping cost that the public-parameter split avoids, at matched quality.
Why not just…?
Why not just have the owner render the frames on their own server? If the owner can host, this is cheaper and should be used. The proposal targets the case where serving is outsourced to a host the owner does not trust, which is the norm for content delivery. In that case plaintext hosting means handing over the asset. The cost model will show where encrypted outsourcing becomes cheaper than owner-side rendering, and we will report honestly if it never does at interactive rates.
Why not just use a trusted execution environment on the host? This is a real threat. Enclaves would render at near-native speed. The difference is the trust root: an enclave trusts the hardware vendor and is vulnerable to side channels, whereas the procedural-firewall philosophy we borrow (Preface) asks for separation that does not depend on any single party. We will not claim superiority on speed. Instead, we report the slowdown factor at which an owner would choose each option.
Why not just encrypt the asset and ship it to licensed viewers? Viewers then hold the decrypted parameters and can copy them. Our protocol releases only images, gated by the custodian's check. A determined viewer can still refit a scene from many images, as from any video stream. The leakage audit will report how many views such refitting needs, so the protection claim stays bounded.
Why not just use GS-Hider or SecureGS? They provide concealment, not encryption, and neither lets a host compute on content it cannot read. Their decoders are deep and nonlinear, so making them encrypted would reintroduce the bootstrapping we avoid. The private-sharpness ablation quantifies a smaller version of that penalty.
Proposed Experiments
Experiments run on Mip-NeRF 360 for unbounded real scenes, NeRF Synthetic for isolated objects, and the Anisotropic Synthetic set from Spec-Gaussian for shiny materials. We report PSNR, SSIM and LPIPS of decrypted frames against held-out views. Separately, we report the PSNR between decrypted and plaintext renders of the same model, which isolates CKKS error. Costs are reported as encrypted asset size, per-frame latency on CPU and GPU CKKS backends, per-frame bandwidth, custodian recomputation time, and rotation count, all as functions of $K$.
The first decisive check is the quality-per-coefficient frontier on the anisotropic set. If at $K=9$ neither the learned basis nor the public-lobe model beats degree-3 SH ($K=16$), then the basis question is moot and the contribution narrows to the encrypted renderer.
The second decisive check tests whether encryption error is negligible. The decrypted-versus-plaintext PSNR must exceed 40 dB at practical CKKS parameters; otherwise the compositing sums need rescaling designs we have not budgeted for.
The third decisive check is the leakage audit. If an attacker predicts appearance from public geometry with error close to the owner's own model, then confidentiality of appearance alone is not worth paying for. We would then redirect the work toward encrypting opacity, which enters compositing nonlinearly, and measure what that costs.
References
- GS-Hider: Hiding Messages into 3D Gaussian Splatting
- SecureGS: Boosting the Security and Fidelity of 3D Gaussian Splatting Steganography
- CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy
- SAFENet: A Secure, Accurate and Fast Neural Network Inference
- Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models
- Circuit Bootstrapping: Faster and Smaller
- Secure On-Device Video OOD Detection Without Backpropagation
- NeX: Real-Time View Synthesis With Neural Basis Expansion
- Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting
- Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition
- Preface
- The ASHES 2020 special issue at JCEN
- Message from the ISMAR 2019 Science and Technology Program Chairs and TVCG Guest Editors
- RECALIBRATING FREE SPEECH IN INDIA’S DIGITAL AGE: BALANCING EXPRESSION, NATIONAL INTEGRITY AND THE GLOBAL DEMOCRATIC CHALLENGES
- Multiparty Homomorphic Encryption (or: On Removing Setup in Multi-Key FHE)