Axis-Split Convolutions for Encrypted Volumetric Inference

Hospitals would like to send a CT or MRI volume to a cloud model and get a diagnosis back without the cloud ever seeing the scan. Fully homomorphic encryption (FHE) makes this possible in principle. FHE is a family of schemes that let a server compute directly on encrypted numbers. The price is steep: every encrypted operation is orders of magnitude slower than its plaintext version. Every working encrypted deep-learning system we know of handles flat 2D images or 1D token sequences. The common assumption is that going to 3D multiplies that cost by a factor that grows with the cube of the side length. This note argues that the assumption is mostly wrong, and that the right question is narrower and more useful.

Consider what happens when a small volume such as a 28×28×28 scan (21,952 voxels) is packed into the slots of a single ciphertext. The CKKS scheme, a common FHE scheme for approximate real-valued arithmetic, offers up to 32,768 parallel slots at the largest standard ring size, so the whole volume fits. In this packing, running a 2D convolution on every slice at once costs exactly what one 2D image would cost, because the slices sit side by side in the slots. Only mixing across slices costs extra. That extra cost is a small, countable number of additional ciphertext rotations (the expensive "shift" operation in FHE) and weight multiplications. Axis-split convolutions, borrowed from plaintext medical imaging, give each channel group a 2D kernel in a different orthogonal plane. They buy full 3D context for far fewer rotations and multiplications than a cubic kernel. We propose to build the encrypted operator that makes this choice explicit and to measure which volumetric operator mix gives the best diagnostic accuracy per second of encrypted latency on MedMNIST3D.

Core Claim

Under single-volume CKKS inference, the cost of a volumetric convolution is set by two quantities: the set of spatial offsets its kernels touch, and how channels are grouped against those offsets. Dimensionality itself is not the driver. A concrete cost model follows from the standard one-channel-per-ciphertext evaluation of convolution, in which each input ciphertext is rotated once per distinct offset, multiplied by weight plaintexts, and summed:

$$ T \approx \alpha \, C_{in} \, |O_\cup| \;+\; \beta \, C_{in} \sum_{g} C_{out,g}\, |O_g| \;+\; \gamma \, B . $$

The symbols are as follows. $C_{in}$ is the number of input channels. $O_g$ is the set of offsets used by output-channel group $g$, which has $C_{out,g}$ channels. $O_\cup$ is the union of all groups' offsets. $B$ is the number of bootstraps, the noise-refreshing operation that dominates FHE runtime. $\alpha, \beta, \gamma$ are measured per-rotation, per-multiplication and per-bootstrap costs.

Plugging in a 3×3×3 kernel shows where the costs land. A full cube has $|O_\cup| = |O_g| = 27$. An axis-split layer has $|O_g| = 9$ per group, but its union is only 19 (the cube minus its eight corners). A slice-parallel 2D layer has 9. A factorized (2+1)D layer costs 9 + 3 offsets but spends an extra multiplicative level, which can force extra bootstraps.

Our claim is that a network built mostly from slice-parallel and axis-split layers will sit on a better accuracy-versus-latency curve than both direct 3D convolution and the slices-as-channels "2.5D" shortcut. We also claim the cost model will predict the measured ranking. Both parts can be falsified, and either outcome is informative.

Prior Work and Its Role

The closest system is Orion. It compiles PyTorch models into CKKS circuits, packs strided 2D convolutions as Toeplitz matrix products to halve their depth, and places bootstraps by solving a shortest-path problem on the network graph. On CIFAR-10 ResNet-20 it reports 37 bootstraps against 58 for the previous compiler, Fhelipe, and a 2.38× speedup. Orion supplies our method for bootstrap placement and scale management. Everything it packs is 2D, and it offers no volumetric layout. Two other papers supply the polynomial ingredients. SMART-PAF trains low-degree polynomial substitutes for ReLU and reports a 14-degree approximation matching a 27-degree one at a 7.81× speedup. PolyNeXt and the polynomial Transformer conversion remove activations and softmax entirely. All three are evaluated only in 2D or 1D. We use them as fixed parts rather than as the variable under study.

On the dimensionality side, two papers motivate the proposal. Explainable 2D Vision Models for 3D Medical Data runs a 2D backbone over orthogonal slices and pools them with attention. It reports "top results in half of" the MedMNIST3D datasets with 63% fewer parameters than a 3D-Conv model. That result suggests full cubic kernels are not needed for accuracy at this scale. The paper runs entirely in plaintext and depends on softmax. MedMNIST v2 supplies the benchmark. It also supplies the axis-split (ACS) and 2.5D plaintext baselines, and it reports that 2.5D models struggle with full volumetric context. That finding constrains us: it predicts the cheapest encrypted option will lose accuracy, and we must show it.

CryptoNets fills slots by batching thousands of different images, which gives high throughput but 250 seconds per single inference. We instead fill the slots with one patient's volume, because the target is single-scan latency.

Our contribution differs from all of the above in one specific way. We treat "how 3D is this layer" as a declared, measurable property of an encrypted operator, and we study the accuracy-cost trade-off of that property directly.

Research Program

We will extend Orion with one module, an offset-set convolution. The module is declared by a list of spatial offsets per output-channel group over a flattened depth×height×width slot layout. From that declaration the compiler derives three things: the rotation steps and their evaluation keys (one key per distinct stride), the boundary masks that implement zero padding, and the cost terms in the model above.

Full Conv3d, Conv2d applied to all slices, axis-split convolution, (2+1)D factorization and depth pooling are all instances of this single module, differing only in configuration. That is the role of modular design here. The papers on reproducible research tooling, DeepDIVA and FaceX-Zoo, argue for freezing the harness so that only the variable under test changes. In our setting this guarantees that every architecture arm runs through identical packing, scale management and bootstrap placement, so differences in latency come from the operator and not from implementation quality. The concern that uniform interfaces leak when operations differ largely disappears here, because every arm genuinely reduces to "rotate by these strides, multiply, sum."

The main new engineering piece is strided downsampling in 3D. Halving a 28³ volume to 14³ leaves 2,744 occupied slots per channel. Without channel multiplexing, more than 90% of each ciphertext would be wasted. We will generalize Orion's single-shot multiplexed packing from two spatial axes to three, folding the masking for the stride along the depth axis into the weight plaintexts.

The networks will be small 3D-ResNet-18-shaped models. Within the network we vary only the per-layer operator type: slice-parallel, axis-split, cubic, or (2+1)D. Within one experiment we hold the activation fixed. The main runs use a SMART-PAF-trained low-degree polynomial ReLU, and a secondary run uses PolyNeXt-style Hadamard-product channel mixing, to check that conclusions survive the activation choice. Classification ends in polynomial global average pooling, which needs only logarithmically many rotations. One further ingredient connects the work to the 2D-pretraining problem: axis-split layers can be initialized from 2D weights. We will test whether initializing from models trained on MedMNIST2D helps on the small 3D datasets.

Baselines

The naive reference point is direct Conv3d everywhere. Concretely, this is the same ResNet-18-shaped network with every 3×3 kernel replaced by 3×3×3, compiled through our module with the same polynomial activation and the same Orion bootstrap placement. It is the method a reviewer would implement first, and it is the upper reference for plaintext accuracy among our arms. For plaintext context, MedMNIST v2 reports an average AUC of 0.863 across MedMNIST3D for a 3D-convolution ResNet-50. That number is not comparable to an encrypted, polynomial-activation model, so we report it only to anchor the scale. We found no reported result on Lacuna for any encrypted 3D convolutional inference, latency or accuracy, on MedMNIST3D or elsewhere.

The strongest existing method for accuracy at low cost is slice decomposition with attention pooling in the style of Ziller et al. To make it comparable we will build an encrypted version. It applies slice-parallel 2D convolutions along each of the three orthogonal axes, then pools slices with the polynomial attention kernel from PolyNeXt in place of softmax. It runs through the same compiler and receives the same training budget. This arm also tests whether cross-slice attention is worth its extra depth compared with letting convolutions mix across slices.

The cheapest competitor is 2.5D: depth is treated as input channels, collapsed by the first layer, and the rest of the network runs in 2D. It will likely be the fastest arm.

Fairness rules apply to all arms. Every arm gets the same training recipe and the same number of hyperparameter trials on the official MedMNIST3D validation splits. All arms run with the same CKKS parameters on the same single-threaded CPU, matching Orion's setting, and are timed per volume.

One ablation isolates the key ingredient. We will compare an axis-split network against a slice-parallel network with exactly the same layer count and channel widths. The two differ only in whether layers mix across the third axis. This separates "3D context" from "more parameters." "Beating" the competition means one thing: at some latency budget, an axis-split or mixed network achieves higher encrypted test AUC than every baseline at equal or lower latency, with at least one MedMNIST3D dataset showing a gain larger than seed-to-seed variation.

Why not just…?

Why not just port Conv3d into Orion and run a standard 3D CNN? This is the most serious alternative, and our own cost model says the rotation penalty is modest: 27 offsets against 19 for axis-split, not an explosion. The weight-multiplication term is where cubic kernels cost about three times more per output channel, and key memory grows with the number of distinct strides. If direct Conv3d matches axis-split on the accuracy-latency curve, that is a clean negative result, and the offset-set module still delivers the first encrypted 3D convolution. Our experiments run it as a full arm rather than arguing it away.

Why not just stack slices as channels and reuse a 2D encrypted network? MedMNIST v2 already finds that 2.5D models lose volumetric context in plaintext. Under FHE, however, 2.5D may be so cheap that a wider or deeper 2.5D network bought with the saved latency closes the gap. We test this directly by giving 2.5D a larger width at a matched latency budget, not just a matched architecture.

Why not just batch many patients per ciphertext as CryptoNets does? Batching optimizes throughput, not the 250-second single-inference latency CryptoNets reports. A 28³ volume already occupies two-thirds of the slots, so there is little room left to batch. Where per-patient latency does not matter, batching is a legitimate choice, and we say so. Our target is the interactive single-scan case.

Why not just use secure multi-party computation, which handles softmax and ReLU exactly? The systematic comparison of encrypted image classifiers shows interactive protocols can move tens of gigabytes per inference and degrade sharply under realistic bandwidth. That is a different trade-off, involving communication rounds and a different threat model. We keep it out of scope rather than claim superiority.

Proposed Experiments

All experiments use the MedMNIST3D datasets at 28³ with their official splits, including OrganMNIST3D, NoduleMNIST3D and SynapseMNIST3D. For each arm we report several quantities: plaintext AUC and accuracy for the polynomial model; AUC and accuracy after encrypted inference and decryption; the agreement rate between encrypted and plaintext predictions; measured rotations, weight multiplications and bootstraps per volume; total rotation-key memory; and single-volume CPU latency.

The first decisive check tests the cost model. If measured latency does not rank the arms the way the equation predicts, the offset-set view is wrong and the design guidance it implies is worthless. The second check is the slice-parallel versus axis-split ablation at matched width. If cross-axis mixing adds no AUC on these small volumes, then 3D context does not matter at 28³ under polynomial activations, and the cheapest slice-parallel design should be recommended. The third check is the full Pareto comparison against direct Conv3d, encrypted slice attention and widened 2.5D, repeated with the PolyNeXt-style activation to confirm the ranking does not depend on the polynomial choice. A secondary experiment compares 2D-initialized axis-split networks against random initialization on the smallest 3D datasets, testing whether 2D pretraining helps on scarce 3D data.