A hospital, phone, or camera often has a degraded image (noisy, compressed, hazy, underexposed) and wants a cloud model to clean it up without the cloud seeing the content. One cheap way to hide an image from people is to cut it into small blocks and shuffle them with a secret key. The result looks like a scrambled jigsaw puzzle to a human. Neural networks can still classify and detect objects in such images (Human-Imperceptible, Machine-Recognizable Images). Nobody has asked whether the same trick works for restoration, where every pixel matters. The answer is not obvious. The strongest modern restorers mix a spatial branch with a frequency branch, and their frequency branch takes a Fourier transform of the whole image. Shuffling blocks destroys exactly the global phase structure those models depend on.
We propose to build the server-side restorer so that its output is guaranteed to be shuffled in the same way as its input, whatever the key. In the terms used below, it is "block-equivariant." The client unshuffles the result locally with its key. The design rests on a split of what restoration needs into three kinds of information. Content inside each block survives shuffling. Global statistics that do not depend on position, such as noise level, haze strength, illumination, and JPEG quantization, survive shuffling if they are computed as a function of the unordered set of block spectra. Context that crosses block boundaries is lost, and we measure its cost exactly. The bet is that the first two kinds of information carry most of the benefit of dual-domain restoration for degradations that are either local or global. Deblurring with large kernels is the expected failure case, and we will report it as one.
Core Claim
Let $x$ be the degraded image split into $B$ blocks of size $M\times M$, and let $P_K$ be the block permutation defined by key $K$. We build a restorer $f$ that satisfies
$$ f(P_K x) = P_K f(x) \quad \text{for every key } K, $$
so the server never needs the key and the client recovers $f(x) = P_K^{-1} f(P_K x)$.
The network has three parts. Each block passes through a spatial branch and a block-local frequency branch, either a per-block FFT or an 8×8 DCT aligned with the JPEG grid. A separate global-context branch pools the per-block amplitude spectra with a symmetric operation, attention without positional encoding, into a degradation descriptor. That descriptor modulates every block through scale and shift. This last step mirrors the Global Guided Local module of Frequency and Spatial Dual Guidance for Image Dehazing, which showed that haze lives mostly in the amplitude spectrum. Here the "global" signal is taken from the multiset of block amplitudes rather than from one whole-image FFT. A whole-image FFT is invariant to neither the order nor the content arrangement of shuffled blocks. The multiset of block amplitudes is exactly invariant to the order.
The quantitative expectation is that, without cross-block context, error concentrates near block edges. For a restorer whose effective local receptive radius is $r$ pixels, the fraction of affected pixels is roughly $4r/M$. That figure counts the pixels within $r$ of one of a block's four edges. So the gap to an unshuffled model should shrink roughly in proportion to $1/M$.
A client-side "halo" lets each transmitted block carry an $h$-pixel margin from its neighbours. It closes the gap but makes jigsaw reassembly easier. We treat $h$ as an explicit privacy-utility dial and measure both sides of the trade.
This differs from every close neighbour in the file. Keyed deformable convolutions (Privacy-Preserving Deep Learning Using Deformable Operators for Secure Task Learning) undo the permutation inside the network. That requires the key to be inside the model, and the method was tested only on classification and segmentation. PIRNet (PIRNet: Privacy-Preserving Image Restoration Network via Wavelet Lifting) hides the image in a natural-looking cover image rather than encrypting it, and it uses wavelets rather than Fourier amplitude and phase. Homomorphic CT denoising (Privacy-Preserving Encrypted Low-Dose CT Denoising) is lossless but needs a client round trip at every nonlinear layer. Our method needs one upload and one download, and the server never holds a key.
Research Program
The first piece is the architecture and the proof that it is equivariant. Per-block operations are equivariant by construction. The global branch is a symmetric set function, and the modulation applies the same descriptor to every block, so the composition is equivariant. We verify this numerically: random keys should produce bit-identical outputs up to permutation.
For JPEG, choosing $M$ as a multiple of 8 keeps each 8×8 DCT block intact. This lets us reuse the quantization-interval box constraint of D3: Deep Dual-Domain Based Fast Restoration of JPEG-Compressed Images unchanged inside every shuffled block. The per-block FFT branch borrows the frequency-dynamic filtering and the combined spatial-and-spectral L1 loss from When Fast Fourier Transform Meets Transformer for Image Restoration.
The second piece is the key design. The base key is a fresh block permutation for each image, which makes known-plaintext attacks across images moot. We add optional per-block key-chosen flips and rotations. These are handled by augmentation during training. They make adjacency harder to match without breaking the pixel-local structure that restoration needs.
We deliberately exclude shuffling pixels inside a block. That kind of mixing, as in the mixing-up scheme of Hao et al., destroys the local correlations that denoising and deblocking need.
The third piece is security measurement. This addresses an unstudied question in the file: does restoring the encrypted image make it easier to break? Cryptanalysis of Dual-Stage Permutation Encryption Using Large-Kernel Convolutional Neural Network and Known Plaintext Attack recovered permutations from block statistics alone, reaching around 20 dB reconstruction. That result motivates the measurement and also constrains the design: it warns that statistics preserved inside blocks are exploitable.
We will train a learned adjacency predictor, a large-kernel network in the style of that paper's LKCNN, to score whether two blocks were neighbours. It feeds a greedy jigsaw solver. We report the fraction of correct neighbour pairs and the PSNR of the reassembled image under three conditions: on the degraded encrypted input, on the server's restored-but-still-shuffled output, and as a function of block size $M$ and halo $h$. Restoration removes noise that may currently mask boundary continuity, so we expect restored outputs to be more vulnerable. Quantifying that effect is a result in its own right.
Baselines
The naive reference point has two parts. The first is the identity: return the degraded image. This anchors how much restoration happened at all. The second is a plaintext upper bound: the same backbone run on unshuffled images, at matched parameters and FLOPs. This defines the privacy cost.
The strongest existing restorer is SFHformer (When Fast Fourier Transform Meets Transformer for Image Restoration). It reports 43.03 dB PSNR on SOTS-indoor dehazing, ahead of MB-TaylorFormer by 0.39 dB, and gains of 0.98 dB over Retinexformer on LOL-v2-real and 0.62 dB over DRSformer on Rain200L. FSDGN reports 38.63 dB on SOTS-indoor with 2.73M parameters. We run SFHformer two ways. In the first, it is retrained directly on shuffled images with its whole-image FFT intact, the same training data, and the same number of iterations. This is the "let the network figure it out" competitor. In the second, it is run tile by tile on each shuffled block with weights trained on plaintext.
The tile-wise variant is the most dangerous baseline. It is trivially equivariant and differs from our method only by the global set-context branch and the block-aware frequency design. Beating it means a PSNR gain larger than the seed-to-seed standard deviation over three seeds. We expect that gain on haze, low-light, and denoising at unknown noise level, where the global descriptor matters. We do not necessarily expect it on JPEG, where the quantization table already supplies the global information.
For the medical case we compare against the homomorphic CT denoiser of Yang et al. on the 2016 NIH-AAPM-Mayo data. It reports 40.37 dB PSNR and 0.9789 SSIM, identical to plaintext. We expect to lose some PSNR to it. We will report how much, alongside communication rounds and wall-clock time.
For PIRNet on DIV2K, the file gives only relative gains over its baselines: +2.11 dB for denoising, +1.56 dB for deblurring, and +0.79 dB for 4× super-resolution. No absolute figure was found on Lacuna, so we rerun it under our noise and blur settings rather than quoting a number.
Two ablations isolate the key ingredient. The first removes the global set-context branch, which gives a per-block restorer only. The second replaces the per-block amplitude pooling with a whole-image FFT of the shuffled canvas. All methods get the same training sets, the same optimizer budget, and hyperparameter searches of equal size.
Why not just…?
Why not just restore each tile independently with an existing model? This is the real threat, and the proposal is built around testing it. Tile-wise inference already gives the equivariance property. If it matches our method on every task, the honest conclusion is that tiling plus the halo trade-off is the whole story, and the global branch adds nothing. The FSDGN amplitude-swapping evidence predicts otherwise for haze and illumination: a single block sees too little area to estimate a global amplitude correction. The dehazing and low-light experiments settle this directly.
Why not just train an ordinary dual-domain network on shuffled images? A non-equivariant network has to learn a separate behaviour for every one of the astronomically many orderings. Its whole-image FFT also mixes content across unrelated blocks, so the phase it sees is scrambled. The retrained-SFHformer baseline measures how much this costs. If it unexpectedly matches our method, the architectural constraint is unnecessary, and we will say so.
Why not use homomorphic encryption or a secure enclave? Homomorphic encryption gives exact results, but Privacy-Preserving Encrypted Low-Dose CT Denoising needs a client round trip at every ReLU. Enclaves require trusting hardware and the provider. Block shuffling is far weaker cryptographically, and our security experiments are designed to show how much weaker. The appropriate use is where a single non-interactive round trip matters more than formal guarantees.
Why not hide the image steganographically like PIRNet? PIRNet needs a cover image for every request, and its authors acknowledge it is vulnerable to steganalysis. Our method needs no cover, but the server can see that the image is scrambled. We compare the two on fidelity and on attack success rather than claiming dominance.
Proposed Experiments and Decisive Checks
The tasks span the local-to-global range of degradations. Denoising uses SIDD and synthetic DIV2K noise at unknown levels. JPEG artifact removal uses LIVE1, Classic5, and BSD500, with $M \in \{8, 16, 32\}$. Dehazing uses RESIDE ITS for training and SOTS-indoor for testing. Low-light enhancement uses LOL-v2-real. Deblurring uses GoPro, where we predict a large gap that the halo only partly closes. We report PSNR and SSIM, PSNR-B for JPEG, and latency.
The decisive checks are three.
First, the plaintext gap should fall roughly as $1/M$. If it does not, our account of where errors come from is wrong.
Second, our method must beat tile-wise SFHformer on dehazing and low-light by more than the seed variance. If it does not, the central claim fails.
Third, the curves of jigsaw-attack success versus $(M, h)$ must reveal an operating point where reassembled images stay unrecognizable. We set the threshold below the roughly 20 dB that the cryptanalysis paper reached. At that operating point, restoration must still retain most of the plaintext gain. If no such point exists for a task, we report that the task cannot be done privately under block shuffling.
References
- Human-Imperceptible, Machine-Recognizable Images
- Frequency and Spatial Dual Guidance for Image Dehazing
- Privacy-Preserving Deep Learning Using Deformable Operators for Secure Task Learning
- PIRNet: Privacy-Preserving Image Restoration Network via Wavelet Lifting
- Privacy-Preserving Encrypted Low-Dose CT Denoising
- D3: Deep Dual-Domain Based Fast Restoration of JPEG-Compressed Images
- When Fast Fourier Transform Meets Transformer for Image Restoration
- Cryptanalysis of Dual-Stage Permutation Encryption Using Large-Kernel Convolutional Neural Network and Known Plaintext Attack