Keeping the Optical Key Local: Privacy Cameras as Federated Learning Clients

Federated learning trains one shared model across many devices, such as factory cameras or ATMs. Each device trains on its own images and sends only weight updates to a central server. That design leaves two holes. First, the server can often reconstruct training images from those updates, an attack called gradient inversion. Second, the raw frames still sit unprotected in the device's own memory, where malware can read them. A separate line of work builds privacy cameras. These scramble identity information optically, with a learned lens, a programmable light modulator, or tuned color-processing hardware, so that a clean image never exists in digital form. The obvious move is to put such cameras on federated clients. Nobody has done it, and doing it naively has a trap. Deep-optics methods train the lens jointly with the network, and the natural way to carry that into federated learning is to average the lens parameters along with the network weights. That hands the server the exact optical blur. The server can then invert the gradient and undo the blur in one optimization.

We claim that pre-capture optics protect a federated client against both threats only if the optical transform behaves like a secret key that never leaves the device. We propose federated training in which each client keeps a private lens code, either fixed per device or re-drawn every frame. The server aggregates only a digital model trained to work under any code in a shared family of privacy-preserving codes. We will measure, under a single threat model covering both device memory and server-side gradient attacks, how much this key locality reduces reconstruction. We will also measure how much differential-privacy noise it saves and what it costs in accuracy.

Where the two literatures stand

Deep-optics privacy work places a learnable optical element in front of the sensor and trains it end to end with a task network. Learning Privacy-preserving Optics for Human Pose Estimation shapes a lens with Zernike polynomials, which are smooth basis functions describing lens shape. On COCO keypoints it reaches 0.302 AP (clear lens 0.421) while pushing face recognition to near chance. OpticalDR does the same for depression scoring, with face-verification AUC around 0.51. CamPro instead tunes the camera's image-signal-processor color and gamma registers. It drops face identification from 84.3% to 0.3% while keeping 0.475 person-detection AP (raw 0.578), and an attacker who retrains on protected images reaches at most 11.2%. PrivateEye uses diffractive layers to steer identity features to sensor pixels that are never read out.

Learning a Dynamic Privacy-Preserving Camera Robust to Inversion Attacks (DyPP) matters most here. It notes that a fixed blur can be measured and deconvolved. Its answer is a generator that maps random noise to a family of lens codes, displayed on a spatial light modulator, with a new code per frame. It reports 36.0 COCO mAP against 33.9 for static privacy lenses.

All five papers motivate this proposal but train centrally. Their threat model is someone looking at, or deconvolving, the sensor image. DyPP also supplies our method: its code family and code-robust decoder are what we distribute.

On the federated side, Inverting Gradients shows that matching gradient directions (cosine distance) with a smoothness prior reconstructs training images. It reports about 14 dB PSNR on a trained ResNet-20 for CIFAR-10, versus about 7 dB for earlier distance-matching attacks. Reconstructions stay recognizable at batch size 100 and across multiple local steps. Gradient Inversion with Generative Image Prior strengthens the attack with a learned image prior. These papers constrain us: batching is not a defense, and any optical defense must be tested against attackers with strong priors.

FedVision shows that federated detection runs at industrial scale, cutting bandwidth from 50 MB/s to under 1 MB/s. It keeps clear frames on the device, so it is the deployment our clients would replace.

Core Claim

When a client trains on optically encoded images, a gradient attack recovers at best the encoded sensor image $y = h * x$. Here $x$ is the scene, $h$ is the lens's point-spread function (the blur pattern a single point of light makes on the sensor), and $*$ is convolution. The real question is whether the attacker can also recover $h$. We formalize the strongest attacker as solving

$$x^*, h^* = \arg\min_{x,\, h \in \mathcal{H}} \; \mathcal{D}_{\cos}\big(\nabla_W \mathcal{L}(f_W(h * x), y_{\text{label}}),\; g\big) + \alpha\, \mathrm{TV}(x).$$

In this objective, $g$ is the observed client update and $f_W$ is the shared model with weights $W$. $\mathcal{L}$ is the training loss and $y_{\text{label}}$ is the training label. $\mathcal{D}_{\cos}$ is cosine distance, $\mathrm{TV}$ is the total-variation smoothness prior with weight $\alpha$, and $\mathcal{H}$ is what the attacker knows about the optics.

$\mathcal{H}$ can take four forms. It can be a single known $h$, which is the case when the optics are federated. It can be the shared code family with the specific code unknown, or the family plus many rounds of pooled updates. Or it can be nothing at all.

Our prediction has three parts:

  • Federated optics, where $h$ is known, will give reconstructions close to undefended training.
  • A private static code per client will hold up only until the attacker pools enough rounds to estimate $h$ blindly.
  • Per-frame private codes from a family trained to resist inversion will keep identity recovery near chance at modest utility cost.

Research Program

We will build a federated simulator. Each client holds a DyPP-style differentiable optics model: Zernike lens codes produced by a shared code generator, followed by simulated diffraction and sensor noise. We add a second encoder family based on CamPro's differentiable color and gamma pipeline, so the findings are not tied to one kind of hardware.

The code generator is pretrained centrally on public data and then frozen. Each client gets a private random seed and draws its codes locally. We study two regimes: one fixed code per client, and a fresh code per frame. Only the digital model is aggregated, using FedAvg (weighted averaging of client weights).

During local training we add DyPP's privacy and non-invertibility losses. We also try one new term: the identity-recognition accuracy of a cheap learned inverter that maps gradients to images. This puts a gradient attacker into the training loop rather than only an image inspector.

Analysis has three parts. First, we map reconstruction quality against attacker knowledge for each optics regime. Second, we measure client drift: whether a digital model shared across clients with different codes converges as well as one trained centrally across the whole code family. Third, we find the differential-privacy noise level at which clear-image clients reach the same reconstruction quality as optical clients, and compare utility at that matched privacy level.

A small benchtop check is optional. It would use a static printed mask per client and CamPro's RV1126 register path. Its purpose is only to confirm that simulated codes behave like real ones. The main results are in simulation, and we say so.

Baselines

The do-nothing reference is FedAvg on clear images under the same partitions, model, rounds and local-step budget. It gives the attack ceiling and the utility ceiling. Geiping et al. report about 14 dB PSNR for their attack on CIFAR-10 with a trained ResNet-20. Reproducing roughly that figure on our clear-image setting calibrates our attack implementation.

The strongest existing method for the privacy camera itself is DyPP, trained centrally. It reports 36.0 COCO mAP. Hinojosa et al. report 0.302 keypoint AP, and CamPro reports 0.475 person AP with 0.3% face identification. Our federated variants are judged against these centralized numbers on utility. "Beating" them means staying within a few points of utility while adding protection against gradient attacks, which they never measured.

For gradient attacks on optically encoded inputs there is no reported result found on Lacuna. Every attack number in that cell is new.

The strongest federated defense is clipped, noise-added training (DP-SGD style) on clear images. We tune it with the same hyperparameter-search budget we give our method. We found no reported result on Lacuna for differential-privacy utility on these vision tasks, so we produce the full curve ourselves. Our method wins if, at equal identity-recovery rate from reconstructions, it gives higher task AP or accuracy. A combined variant tests whether optics plus small noise beats noise alone.

The ablation that isolates the key ingredient is federated optics: the lens parameters are averaged and broadcast like ordinary weights, which is the natural extension of centralized deep optics. A second ablation uses a private static code per client, to separate keeping the key local from refreshing it per frame. A third applies the identical learned transform in software on clear captured frames. It is equivalent for the gradient threat but leaves raw frames in memory, so it isolates what physical capture adds.

Why not just…?

Why not just apply the same learned transform in software before computing gradients? Against the server-side gradient attack this is mathematically the same as the optics, and our software-emulation baseline will show matching numbers. The difference is the device-memory threat, the post-capture gap stressed by CamPro and PrivateEye. We score it as identity accuracy on whatever image exists in device memory. For a static printed mask, the key lives in the glass. For a light modulator, the key lives in firmware and could be read by device-level malware. We will report that regime as weaker against device compromise. If a deployment trusts its devices, software emulation is sufficient, and the key-locality finding still transfers to it. We count that as part of the result, not a defeat.

Why not just add differential-privacy noise to updates? It gives formal guarantees that optics do not. The concern is utility on detection and pose estimation. The file has no reported number, so the matched-privacy comparison in this proposal is what settles it. If noise alone matches our utility at equal reconstruction quality, the optical contribution reduces to the device-memory threat, and we will say so.

Why not just use secure aggregation, so the server sees only summed updates? It hides individual updates from an honest-but-curious server. It does nothing for a compromised device. It is also complementary rather than competing, so we run it on top as a sanity check.

Why not just federate the optics along with the weights? This is the natural design, and we predict it fails because it broadcasts the key. It is our main ablation.

Proposed Experiments

We simulate 20 to 100 clients with non-uniform data splits. Benchmarks are CelebA (utility: smiling; private: identity and gender, as in PrivateEye), COCO person detection and keypoints (utility, as in Hinojosa, CamPro and DyPP), and CIFAR-10 to calibrate the attack. The attacks are cosine gradient matching with a total-variation prior and the generative-prior variant. Each runs under the four attacker-knowledge levels, with pooling over up to hundreds of rounds.

Metrics follow from the claim:

  • PSNR and SSIM of reconstructions measured against the scene $x$, not against the sensor image.
  • ArcFace identification accuracy on reconstructions and on the image held in device memory.
  • Task AP or accuracy.
  • Convergence speed relative to the centralized code-robust decoder.

Two results are decisive. First, federated optics should leak nearly as much as clear training, while per-frame private codes keep identification on reconstructions near chance even when the attacker knows the code family. If the family-aware attacker succeeds, the key-locality idea fails, and the code family needs stronger non-invertibility. Second, at matched identity-recovery rate, per-frame private codes should achieve higher utility than tuned noise addition alone. If the private static code survives pooled-round blind estimation as well as per-frame codes, then cheap printed masks are enough and the programmable modulator is unnecessary. That would be a practically important outcome either way.