Depth-Free Temporal Fusion for Encrypted Evaluation of Video Models on Private Data

Many strong results in clinical and industrial machine learning cannot be checked by anyone else. The data are patient videos, handwriting recordings, or sensor logs that legally cannot leave the institution. A natural fix is to let an outside model be tested on the private data without either side seeing the other's assets. Homomorphic encryption (HE) makes this possible in principle. HE is a family of cryptosystems in which a server can add and multiply encrypted numbers without decrypting them. The obstacle is cost. Encrypted computation runs two to five orders of magnitude slower than plaintext, and video multiplies that cost by the number of frames.

This note proposes a way to make encrypted video evaluation cost roughly what encrypted image evaluation costs. The key fact is that some temporal summaries are just fixed weighted sums of frames. The clearest example is approximate rank pooling, which collapses a clip into one "dynamic image." Weighted sums are exactly the operations HE performs almost for free. So all temporal fusion can happen before any expensive encrypted multiplication. The encrypted network then spends its limited multiplication budget only on the interaction between motion and appearance. The goal is to turn "an outside lab verified this result on data it never saw" from an impractical idea into a routine, affordable procedure.

Core Claim

Two ideas from the file combine here. The first is from Dynamic Image Networks for Action Recognition, which summarizes a clip of $T$ frames $x_1,\dots,x_T$ as

$$d = \sum_{t=1}^{T} \alpha_t\, x_t,$$

where the coefficients $\alpha_t$ come from one gradient step on a ranking objective that rewards later frames scoring higher than earlier ones. The coefficients depend only on $t$ and $T$, not on the data. The second idea concerns how HE schemes such as CKKS (the standard HE scheme for approximate real-number arithmetic) handle this sum. Adding ciphertexts and scaling them by public constants uses essentially none of the multiplicative depth. Multiplicative depth is the number of sequential ciphertext-by-ciphertext multiplications a computation can afford before the cryptographic noise must be reset by the very expensive "bootstrapping" step, which Circuit Bootstrapping: Faster and Smaller identifies as the main FHE bottleneck. Adding and scaling ciphertexts never forces a bootstrap. At most, scaling by a constant uses one rescaling level if the coefficients are applied server-side. The consequence is that encrypted temporal fusion shrinks the payload by a factor of $T$ at essentially no depth cost.

The claim has three parts. The first is that linear temporal operators, meaning fixed rank pooling or a small bank of $K$ learned temporal filters, can carry nearly all the motion information a spatiotemporal classifier needs. The second is that a single multiplicative fusion of the motion summary with a static appearance frame recovers the "appearance-times-motion" filters that What Have We Learned From Deep Representations for Action Recognition? found inside two-stream networks. That fusion would be computed as a Hadamard (element-wise) product, as in PolyNeXt. The third is that the resulting model can be evaluated end to end under real CKKS. Its accuracy would come within a few points of plaintext two-stream baselines, at a per-clip cost close to one encrypted image. What is new is the placement of the multiplication budget in encrypted spatiotemporal models, together with the first measured ciphertext runtimes for a video backbone. PolyNeXt itself reports only plaintext GPU numbers.

The reproducibility use is the application regime. Consider a data holder, such as a hospital, and a model owner, such as another lab or a company. The data holder encrypts its private test clips under its own key. The model owner runs its model on the ciphertexts. The data holder decrypts the returned scores and publishes the metric. Evaluation of a test set is a throughput problem, not a latency problem, and throughput is where HE is strongest. CryptoNets packed 4,096 inputs into one ciphertext pass and sustained 58,982 predictions per hour on MNIST, even though a single pass took 250 seconds.

Research Program

The work has three parts.

Part 1: model family. We build an activation-free two-stream model. The motion stream receives $K$ temporal projections of the clip. With $K=1$ this is exactly approximate rank pooling. With larger $K$ the filters are learned, but they are frozen and treated as linear at inference. The appearance stream receives a middle frame. Retrieving actions in movies motivates this choice by finding that static shape is most discriminative mid-action while motion dominates at the boundaries. Each stream is a shallow PolyNeXt-style stack of PolyConv and PolyMLP blocks. The streams meet in one Hadamard product at a chosen depth $\ell$, followed by a small polynomial head.

The depth budget splits into the depth of the spatial backbone, one level for fusion, and the depth of the head. The temporal part contributes nothing to this budget. We sweep $\ell$ and $K$ to map accuracy against depth consumed. This matters because SecureNets shows that each extra polynomial degree eats noise budget.

Part 2: deployment. We run the models under CKKS. For the linear layers we reuse the rotation-minimizing matrix products of HETAL, which found slot rotations rather than multiplications to be the dominant cost. We report wall-clock time, ciphertext and key transfer size, bootstrap count, and throughput under SIMD packing.

Part 3: the protocol as a reproducibility instrument. We measure how faithfully encrypted evaluation reproduces the plaintext metric. The main quantity is the prediction flip rate, the fraction of test clips whose predicted class changes because of CKKS approximation error. We report it alongside the difference in the headline metric. We also bound how many queries a data holder can make before the returned scores would let it reconstruct the model, and we fix the evaluation set size accordingly. HE gives confidentiality, not proof that the correct model was run. We argue this matters less here, since the model owner never sees labels or inputs and so cannot tailor outputs to inflate its own score. We state this as an explicit threat-model assumption.

Baselines

Naive encrypted reference. Every frame is encrypted and passed through the same polynomial image backbone, and the scores are averaged. This is what one would do without the linearity insight. It costs roughly $T$ times our method in ciphertext volume and compute. It is run with the same backbone, CKKS parameters, and hardware. We beat it if we match or exceed its accuracy at a small fraction of its cost.

Plaintext accuracy ceiling. We use the published plaintext spatiotemporal models on each dataset. On UCF101, Dynamic Image Networks reports 57.9% for a single dynamic image versus 52.6% for mean pooling. It reports 89.1% on UCF101 and 65.2% on HMDB51 when dynamic, static, and trajectory features are fused. On handwriting, LSTM-CNN reports 96.2% accuracy on DraWritePD and 90.7% on PaHaW with only 0.084M parameters. On equine pain video, the two-stream C-LSTM reports 75.4% accuracy and 73.5% F1. These are not encrypted competitors but ceilings. We count success as landing within a few points of the matching plaintext number with an identical data split.

Strongest encrypted competitor. This is the split design used by SecDOOD, whose code is at Dystopians/SecDOOD, and in spirit by HETAL. The data holder runs a public spatiotemporal backbone in plaintext and encrypts only the resulting feature vector, and the model owner's proprietary part is a light encrypted head. I found no reported accuracy for SecDOOD in the file, and no reported result found on Lacuna for encrypted end-to-end video classification on UCF101, HMDB51, PaHaW, or equine pain. We run this baseline with the same public backbone family, the same training data and tuning budget for the head, and the same CKKS parameters. We beat it only if our end-to-end encrypted model is more accurate on the private-style domains, where public video backbones transfer poorly, at comparable cost.

Key-ingredient ablations. The first swaps rank pooling for mean pooling, both depth-free, to test whether ordering information matters. The second removes the Hadamard fusion so the streams are merely added, as in STM. The third moves the fusion depth $\ell$. A final reference is the spiking-network route of Nikfam et al.. It is cited only for cost: 930 seconds per inference for S-LeNet-5 versus about 31 seconds for LeNet-5, because encrypted time steps scale linearly.

Why not just…?

Why not just ship the model to the data holder and run it in plaintext? This is the real answer for most academic verification. Model-to-data evaluation, where the model is sent to run where the data lives, needs no cryptography. We scope the claim accordingly. HE is needed only when the model owner cannot disclose weights, as with corporate models like those in Retrieve and Copy-style settings, or when the data holder cannot run untrusted code. The contribution for the open-model case is nil, and the note says so.

Why not just use a public video backbone locally and encrypt only a head, as SecDOOD and HETAL do? This is the strongest threat. If it matches our accuracy, it wins on cost. Its weakness is that it evaluates a different model from the one whose claim is being checked. It also depends on public backbones transferring to domains like equine pain or pneumoconiosis radiographs, where Devnath et al. could not have results validated externally at all. The head-to-head comparison on handwriting and pain video settles it. If the split design is within noise of ours, we will recommend it for those domains and restrict our method to cases where the full model must stay secret.

Why not just release synthetic data instead? CorporateBench shows synthetic benchmarks help when real data cannot be disclosed. But a synthetic test set checks a claim on a different distribution. Encrypted evaluation checks it on the real one. We report both where synthetic versions exist.

Why not just encrypt frames and use a recurrent or 3D-convolutional model? Every recurrent step or temporal convolution across frames adds multiplicative depth, forcing bootstraps. The SNN-BFV measurements show latency growing linearly with time steps. The naive per-frame baseline above quantifies this directly.

Proposed Experiments

We first calibrate on public UCF101 and HMDB51, training plaintext polynomial models and confirming the rank-pooling-over-mean-pooling gap reported for ReLU networks survives without activations. We then deploy under CKKS and report accuracy, flip rate, seconds per clip at full SIMD packing, and gigabytes transferred. The transfer figure is compared against the 56 GB and 549 seconds per image that Haralampieva et al. measured for HE-Transformer on a deep CNN. Next, we move to the small-data, privacy-sensitive regimes: PaHaW and DraWritePD handwriting, where rank pooling is applied over coordinate sequences, and equine pain video. There we simulate the cross-institution protocol by holding out one data split as the "private" site.

Three results would be decisive. The first is whether the $K$-filter linear model with one fusion product stays within a few points of its plaintext counterpart and of the published plaintext numbers. The second is whether encrypted cost per clip stays within a small constant of one encrypted image rather than scaling with $T$. The third is whether it beats the split public-backbone design on the pain and handwriting data. If the first fails, linear temporal fusion is too lossy and the depth budget must be spent on time. If the third fails, the practical recommendation becomes the simpler split design, and the note's value reduces to the cost measurements and the flip-rate reproducibility audit.