White Balance in One Ciphertext

Every camera has to separate two things it cannot directly observe: the true color of surfaces and the color of the light falling on them. A pixel records only their product. Estimating the light, a task called color constancy or auto white balance, is solved well by small learned models. If the best such model is a provider's private asset, however, a user who wants it must hand over raw photos, and raw photos are often private. Homomorphic encryption lets a server compute on data it cannot read. For images, though, its cost is severe. One reported single-image encrypted classification took about 549 seconds and moved roughly 56 GB (A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification).

This proposal argues that the physics of color constancy makes it unusually cheap to run under encryption, provided the computation is split at the right place. The phone converts the image to log-chroma coordinates, the logarithms of the ratios green/red and green/blue. In those coordinates a change of light becomes a pure shift of the image's color histogram. The phone builds a small histogram from these values in plaintext, which requires no model secrets. It encrypts the histogram into a single ciphertext and sends it. The server applies its secret filters, which amount to a linear convolution, then a low-degree polynomial readout. It returns three encrypted numbers, and the phone divides them to obtain the illuminant. Because the physics puts every nonlinear, encryption-hostile step (logarithm, division, binning, argmax) on the side where the model is not involved, the encrypted part is a shallow arithmetic circuit. It needs no bootstrapping and no batching across users. The approach succeeds only if it matches the accuracy of plaintext histogram methods at interactive single-image latency, and only in a setting where the model genuinely needs protecting. We test both conditions directly.

Core Claim

The multiplicative ambiguity $I = W \odot L$ (observed pixel $I$, white-balanced surface color $W$, illuminant $L$) turns into an additive shift in log-chroma histogram space (Convolutional Color Constancy). That shift structure is what lets a strong illuminant estimator be written as "public nonlinear featurization, then secret linear map, then public nonlinear decoding." We add a polynomial readout so the secret part stays inside a leveled encryption scheme. A leveled scheme supports a fixed number of sequential multiplications without the costly noise-reset step called bootstrapping. The secret computation is

$$\hat{\ell} = \frac{\sum_i (s\, z_i + 1)^p\, c_i}{\sum_i (s\, z_i + 1)^p}, \qquad z = F * H + B,$$

where $H$ is the client's 64×64 log-chroma histogram, $F$ the server's secret filter (toroidal convolution as in Fast Fourier Color Constancy), $B$ a secret bias map, $z_i$ the score at bin $i$, $c_i$ the bin's plaintext $(u,v)$ coordinate, $s$ a learned scale, and $p$ a small even power. The server computes the numerator's two components and the denominator homomorphically. The client performs the one division after decryption. This replaces softmax with the polynomial kernel-plus-normalization idea from PolyAttn in Activation-Free Backbones for Image Recognition. Here the normalization moves across the trust boundary to the client, so it costs nothing homomorphically. The multiplicative depth is about $2 + \log_2 p$: one level for the plaintext-filter convolution, $\log_2 p$ levels for the power, and one for the weighted sums. At $p \le 8$ that is five or six levels. A 64×64 grid fills exactly the 4096 slots of a CKKS ciphertext with ring dimension 8192. CKKS is the approximate-arithmetic scheme that packs a vector of reals into one ciphertext.

Relation to Prior Work

The research file reports no paper that runs color constancy under homomorphic encryption or secure computation. The closest work falls into three groups.

The first group motivates the choice of histogram features. Convolutional Color Constancy recasts illuminant estimation as locating a translation in log-chroma histogram space with learned filters. Fast Fourier Color Constancy wraps that histogram onto a torus, computes the convolution with FFTs, reaches high accuracy on 32×24 thumbnails, and runs in 1.44 ms on a phone. Both end with nonlinear decoding: an argmax or softmax in CCC, and a bivariate von Mises fit plus de-aliasing in FFCC. We keep their linear core, replace the decoding with a polynomial numerator and denominator, and hand the de-aliasing rule, which is public and gray-world based, to the client. The research file's own gap analysis treats log-chroma histograms as incompatible with encryption because logarithms and binning cannot be evaluated homomorphically. We sidestep that obstacle rather than approximate it: those steps involve no model secrets, so the client can perform them in plaintext.

The second group supplies alternative low-degree estimators and the constraints we must respect. Corrected-Moment Illuminant Estimation predicts the illuminant as a learned 19×3 matrix applied to image moments. It is the natural encrypted competitor, because the client can compute the moments and the server's secret is one matrix. Robust channel-wise illumination estimation and Cross-Camera Convolutional Color Constancy show what deeper plaintext models buy: robustness in the worst cases (CWCC) and cross-camera transfer (C5). C5 generates its filters from unlabeled calibration images. Doing that under encryption would turn the filters into ciphertexts and force ciphertext-ciphertext convolutions, so we treat C5 as a plaintext upper bound, not something to encrypt.

The third group supplies encryption methodology. CryptoNets introduced slot packing and square activations but amortizes latency over 4,096-image batches, which does not fit a single-photo query. NASS shows that encryption cost jumps discretely with depth, which is why we budget depth explicitly. SAFENet keeps exact ReLUs in selected channels through interactive protocols. All three target classification networks on raw pixels. None exploits a physics-derived feature space in which the secret computation becomes linear.

Research Program

We would build three components. The first is a client library that computes FFCC-style pixel and edge log-chroma histograms from a thumbnail and packs them into CKKS slots. The second is a server circuit that evaluates the toroidal convolution as a homomorphic linear transform, using the standard baby-step giant-step decomposition, then the power $p$ and two plaintext-weighted slot sums. The third is a training procedure that learns $F$, $B$, $s$, and $p$ in plaintext through the exact polynomial readout, so the model trained is the model deployed.

We would study two extensions, each adding one ciphertext-ciphertext multiply. The first is a degree-2 scoring model: an element-wise product of a coarse and a fine filter response, adapted from PolyConv in Activation-Free Backbones for Image Recognition. It asks whether one multiplicative interaction on the histogram helps with atypical scenes. The second combines pixel and edge histograms, which needs two ciphertexts or a larger ring.

Two analyses accompany the build. One measures how much scene information the client-side histogram leaks, which determines whether encryption is even needed. The other measures how many three-number responses a client needs to clone the server's model, since shallow secret models are easier to extract.

Baselines

The naive reference is Gray-World computed locally, which needs no server and no model. The file reports its mean angular error as 4.59° on NUS-8 (On Finding Gray Pixels.), 6.3 median on Gehler-Shi, and a maximum of 30.0 on SFU HDR (Corrected-Moment Illuminant Estimation). It marks the floor any server-side model must clearly beat to justify a round trip. The learning-free Grayness Index, also run locally, is a stronger training-free floor at a reported 2.91° mean on NUS-8.

The strongest encrypted competitor is Corrected-Moments with client-computed moments and a server-held 19×3 matrix applied to the encrypted moment vector. This costs one plaintext-ciphertext level and almost no rotations, so it is cheaper than our method. The reported 19-edge-moment result is a median of 2.0 on Gehler-Shi (Corrected-Moment Illuminant Estimation). Our method beats it only if its encrypted median and worst-25% errors are clearly lower on the same folds. The CCC paper reports a nearly 40% reduction in median error relative to Corrected-Moments, which suggests the margin exists in plaintext. We must show that the margin survives the polynomial readout and encryption noise.

The strongest plaintext methods set the accuracy ceilings: FFCC, CCC with softmax readout, CWCC, and C5. The file reports C5 at 1.77° mean on NUS-8, against a previous best of 2.05°. It reports FFCC only as about 20% lower mean error than CCC, with no absolute figure. No reported absolute CCC or FFCC error on Gehler-Shi appears in the file, so we would rerun both.

All learned methods would share the three-fold Gehler-Shi protocol, leave-one-camera-out on NUS-8 and on INTEL-TAU's three cameras, the same thumbnail resolution, and the same hyperparameter search budget.

Two ablations isolate the key ingredient. The first swaps our polynomial readout for FFCC's von Mises readout in plaintext, measuring what the encryptable decoder costs. The second compares the encrypted and plaintext versions of the identical model, measuring what CKKS approximation noise costs; this should be negligible, under 0.05°.

Why not just…?

Why not just run white balance on the phone? For most users this is the right answer. FFCC takes 1.44 ms on a 2016 phone, and C5 fits in about 2 MB. The proposal matters only when the estimator is a protected asset, for example a model a provider trained on data the client lacks. We would simulate that setting by training the server model on INTEL-TAU's 7,022 images and restricting the client-side model to Gehler-Shi. We then report the accuracy gap on held-out cameras. If a public local model closes that gap, the privacy argument for encrypted white balance fails, and we would report that as the main finding.

Why not just send the plaintext histogram, since histograms look anonymous? This is the most serious alternative, because it removes the encryption cost entirely. We test it directly. We train attackers to recover indoor/outdoor labels, the camera model, and coarse scene category from the pixel and edge histograms in INTEL-TAU. If these attacks are near chance, encryption protects little, and the contribution narrows to protecting the query from a curious server's aggregate analytics. If they succeed well above chance, as edge histograms plausibly allow, encryption is justified.

Why not just use a secure enclave or interactive secure computation? The reported enclave latency is about 0.14 s (A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification), and that same paper reports CrypTen at 0.25 s on its malaria model. Both would likely beat us on speed. Enclaves, however, require trusting the hardware vendor (Practical Enclave Malware with Intel SGX). Secure computation needs several network round trips, and the benchmarks behind these numbers were run on local networks. Our protocol is one message each way. We would report latency against CrypTen under emulated wide-area latency, and we would present the comparison as a choice of trust model, not a claim of superiority.

Why not just replace the ReLUs in CWCC with squares? That means encrypting pixels rather than histograms. The cost is many levels, 2D convolutions on image-sized slot layouts, and the instability of squared activations noted in CryptoNets and SAFENet. We would train this variant with the same budget and report its plaintext accuracy and its encrypted depth and latency, to show where it lands.

Proposed Experiments

Accuracy is reported as angular error (mean, median, worst 25%, maximum) on Gehler-Shi, NUS-8, and INTEL-TAU. Cost is measured on a single CPU core for one image: client featurization and encryption time, server latency, ciphertext bytes each way, and multiplicative depth. There is no batching.

Three checks decide the question. First, the encrypted histogram model must beat encrypted Corrected-Moments on median and worst-25% error at a server latency within an interactive budget, which we set at one second. It must also stay within 10% of plaintext FFCC's mean error. Second, a sweep over $p$ in {2, 4, 8} and over linear versus degree-2 scoring maps out the trade-off between accuracy and depth. Given NASS's reported cost cliffs, we expect a knee at the depth boundary where the ciphertext modulus must grow. Third, the histogram-leakage and model-extraction studies establish whether the setting is worth protecting at all. For extraction, we count the queries a distillation attacker needs to reach the server model's error. If extraction takes few queries, rate limiting becomes part of the protocol.

A negative result on the leakage check, or a closed gap on the local-model check, would be reported as the answer rather than hidden.