Stencil codes update every cell of a large grid from its neighbors, one time step after another. They drive weather, ocean and fluid simulations, and they are limited by how fast data can be moved, not by arithmetic. The standard fix is temporal blocking: keep a small piece of the grid in fast on-chip memory and advance it several time steps before writing it back. Trusted execution environments (TEEs) are hardware enclaves that keep data encrypted whenever it leaves the chip or a protected memory region. They are the practical way to run a confidential simulation, such as proprietary reservoir data or sensitive climate inputs, on someone else's machine.
Inside an enclave, moving data across the trust boundary costs far more than ordinary memory traffic. That boundary may be a protected-memory cap, with encrypted paging beyond it, or an encrypted bounce buffer between CPU and GPU. Our claim is that this boundary should be treated as one more level of the memory hierarchy, with its own capacity and crossing cost. A compiler that reads the code's dependence graph can then choose deeper and differently shaped temporal blocks than any tuner built for open hardware.
The opportunity is real because the two literatures have not met. GPU stencil work keeps pushing temporal blocking deeper by reshaping how hardware resources are used. TEE work has only partitioned feed-forward neural networks along layer boundaries. The research file's coverage matrix records no work on temporal blocking inside an enclave, and no reported stencil performance under any TEE.
Core Claim
When the trust boundary is the expensive level of the memory hierarchy, two things should happen. First, the throughput-optimal temporal blocking depth should shift sharply upward relative to open hardware. Second, for multi-stage, multi-array legacy stencils, the right unit of blocking is a group of fused stages chosen on a stage-level dependence graph, under a hard capacity constraint set by the enclave. We expect a partitioner that sees both the graph and the boundary cost to recover most of the open-hardware throughput. We also expect it to clearly beat open-hardware tuners, even when those tuners are retuned with enclave cost constants.
The central trade-off can be written compactly. Consider a $d$-dimensional tile of interior side $b$, a stencil of radius $r$, and a blocking depth of $K$ time steps. Each block must load a halo-padded region of side $b+2rK$, which gives a protected-memory footprint of roughly $a\,s\,(b+2rK)^d$, where $a$ is the number of live arrays and $s$ the bytes per cell. The boundary-crossing cost per useful cell update is then approximately
$$ C(b,K) \approx \frac{c_x\, a\, s\,(b+2rK)^d}{K\, b^d} \;+\; c_f\left[\frac{(b+2rK)^d}{b^d}-1\right] \quad \text{subject to}\quad a\,s\,(b+2rK)^d \le M, $$
Here $c_x$ is the per-byte cost of crossing the trust boundary, which includes encryption and integrity work. The constant $c_f$ is the cost of the redundant updates in the shrinking halo, and $M$ is the protected capacity. The first term rewards deep blocking and the second penalizes it. Raising $c_x$ by the large factors that enclave crossings impose moves the optimum toward larger $K$, until the capacity $M$ binds. On open hardware, $c_x$ is just DRAM bandwidth and $M$ is shared memory or cache, so the tuned optimum sits elsewhere.
Real codes like CloverLeaf chain many kernels per time step over several arrays with different radii. For those, $a$, $r$ and the halo growth depend on which stages are fused, and choosing the fusion becomes a partitioning problem on a graph.
Where the Boundary Actually Costs
Being honest about the hardware decides where this matters. Classic Intel SGX caps protected memory at roughly 90–128MB, and ARM TrustZone devices expose under 16MB. Goten and DarkneTZ both document this, and Goten reports TEE-only execution 6.84x slower than its partitioned design (Goten, DarkneTZ). A multi-gigabyte grid on such a device either pages through encrypted swaps or must be tiled explicitly by the application. On NVIDIA Hopper confidential computing, the reverse-engineering study by Gu et al. shows that CPU–GPU traffic passes through encrypted staging buffers with cryptographic bounce routines (NVIDIA GPU Confidential Computing Demystified). On that platform the costly boundary is host-to-device traffic, which matters for out-of-core grids and for halo exchange between GPUs, not for kernels that fit in GPU memory. VM-based TEEs without a hard cap may impose only a modest tax.
The first job of the project is therefore to measure the "enclave tax" on stencils in each of these regimes. We then direct the optimization only where the tax is large. If the tax is negligible on VM-based TEEs, we will say so.
Research Program
The work has three parts.
The first part builds a stage dependence graph from legacy code. Here we borrow method. Stencil-Lifting represents loop iterations as invariant subgraphs and lifts Fortran/C stencils to Halide deterministically. It is formalized only for single-output kernels with regular loops. We do not need a full verified lift. We need a coarser graph in which each node is a stencil sweep, and each edge carries the array it transmits and its neighbor offsets. Building that graph at the loop-nest level follows the program dependence graph construction of Reps and Yang. Their slicing result is what justifies scheduling a fused group of stages independently of the rest of the program. Where Stencil-Lifting succeeds, we start from its Halide output. Where it fails on multi-output kernels, we fall back to the coarser stage graph. Measuring how often this happens on CloverLeaf, WRF, and the MG solver is itself a result.
The second part is the boundary-aware partitioner. It chooses stage groups, tile shapes and blocking depth to minimize the cost model above, summed over groups. Two hard constraints apply. The live footprint of each group must fit within the enclave capacity at the outer level, and within shared memory or registers at the inner level. The resulting two-level scheme has an inner level of ordinary on-chip temporal blocking, reusing EBISU-style or Girih-style kernels. The outer level advances enclave-resident super-tiles over many steps before sealing them out to untrusted memory. This is the hardware–software co-design element. The cost constants $c_x$, $c_f$ and $M$ are measured per platform by microbenchmarks rather than assumed, and the schedule is generated against them.
The third part is a leakage constraint, which acts as a design constraint rather than a feature. Gu et al. show that transfer timing and plaintext metadata leak on GPU confidential computing. Stencil access patterns do not depend on data values, so a static schedule reveals only grid shape and step count. We require the generated schedule to stay data-independent. We also report the cost of optional padding that hides grid dimensions, so users can see what concealing problem size costs.
Baselines
The naive reference point lifts or ports the code to Halide, or uses the original CUDA, and runs it unmodified inside the enclave. Encrypted paging or bounce buffering handles whatever does not fit. This is what a practitioner would do today, and it defines the enclave tax.
The strongest existing open-hardware method on GPUs is EBISU, the low-occupancy temporal blocking system of Revisiting Temporal Blocking Stencil Optimizations. It reports a 1.49x geometric-mean speedup (up to 2.53x) over the best prior method per benchmark, including AN5D and STENCILGEN, and reaches 88% and 80% of modeled attainable performance for 2D and 3D stencils. We also run AN5D as the automated temporal blocking reference. For legacy multi-kernel applications on CPU enclaves, the reference is Stencil-Lifting's Halide output, for which the file reports 7.62x CPU and 83.85x GPU speedups over legacy Fortran on CloverLeaf. That measurement was on open hardware. There is no reported result on Lacuna for any stencil framework running inside a TEE.
For fairness, every baseline runs on the same enclave platform, grid sizes and step counts. Each receives the same autotuning budget in wall-clock tuning time over its own parameter space. The decisive ablation retunes each baseline's own depth and tile parameters using our measured enclave cost constants. This isolates whether gains come from the cost model alone or also from our extra structure. A second ablation runs our partitioner with open-hardware constants, which isolates the boundary-aware cost. A third disables stage fusion, which isolates the graph.
Beating the baselines means a geometric-mean throughput gain of at least 1.3x over the best retuned baseline in the regimes where Phase 1 shows a large tax. It also means recovering at least 70% of the same code's open-hardware throughput, against whatever fraction the naive monolithic run achieves.
Why not just…?
Why not just run the code monolithically in a VM-based TEE and accept the overhead? This may be the right answer on some platforms, and Phase 1 is designed to find out. On capped enclaves, the evidence that it fails is strong: Goten's 6.84x TEE-only slowdown, and DarkneTZ's overhead once its roughly 14MB of secure memory fills. On GPU confidential computing, the bounce-buffer path that Gu et al. document lies on the critical path of any out-of-core or multi-GPU stencil. If the measured tax on a VM-based TEE is within about 10%, we drop that regime from the optimization claims.
Why not just retune EBISU, AN5D, or a Halide scheduler with enclave-aware cost constants? This is the most serious threat, because the core insight is partly a change of constants. The retuned-baseline ablation tests it directly. These tuners optimize one memory level at a time and, for the GPU frameworks, a single stencil. Neither handles a hard enclave capacity at an outer level, nor fusion across a CloverLeaf-style multi-array kernel chain. If retuning comes within 10% of our partitioner on both single kernels and multi-stage applications, we will report that the contribution is the characterization and the cost model, not the graph partitioner.
Why not just outsource the arithmetic to an untrusted GPU with Goten-style masking? Linear stencils satisfy $S(x+m)=S(x)+S(m)$, so additive masking appears to carry over from Goten. Combining it with SparStencil's matrix reformulation might even fit Tensor Core hardware. The catch is unmasking. The enclave must subtract $S^K(m)$, and computing that costs as much as computing $S^K(x)$, unless it is precomputed offline. Goten precomputes its triplets offline, which is a cheap per-layer correction for matrix multiplication; for stencils the offline cost would equal the online cost. Floating-point masking over hundreds of steps also raises precision and leakage questions. We will implement this as a secondary baseline on one linear kernel to quantify the gap, not as a main path.
Why not use homomorphic encryption or multi-party computation instead of hardware? The file's evidence is decisive at this scale. Enclaves were 120x faster than homomorphic encryption in PRESAGE, and Goten had a 132.64x throughput advantage over Falcon multi-party computation. Iterative stencils over billions of cells compound those costs across every time step.
Proposed Experiments
Phase 1 measures the enclave tax. We run the standard iterative suite (j2d5pt through j3d27pt and Poisson, as used by EBISU and the scratchpad deep-blocking study of Zhang et al.) and StencilMark kernels at grid sizes from below to roughly 50x above protected capacity. The platforms are SGX with a capped protected memory region, OP-TEE on TrustZone for small grids, and Hopper confidential computing in out-of-core and two-GPU halo-exchange configurations. We fit $c_x$ and $M$ per platform and check that measured optimal depths move in the direction the cost model predicts. If they do not, the model is wrong and we stop to revise it.
Phase 2 compares the partitioner against all baselines and ablations on the same suite, reporting throughput in cell updates per second, bytes crossing the boundary, and achieved blocking depth.
Phase 3 targets real applications: CloverLeaf, the NAS MG solver, and the PW advection and NEMO tracer advection kernels used by Stencil-HMLS, at its 8M–134M point grids. Here stage fusion should matter most, and we report how often the stage graph had to substitute for a failed full lift.
The decisive checks are three. Does boundary-aware depth beat open-hardware depth when both run on the same enclave? Does the graph-based fusion add measurable gains beyond retuned single-kernel tuners on multi-stage codes? Does the data-independent schedule keep the transfer-timing trace independent of grid contents? We test that last property by running identical shapes with different inputs and comparing the distributions.
References
- Goten: GPU-Outsourcing Trusted Execution of Neural Network Training
- DarkneTZ: towards model privacy at the edge using trusted execution environments
- NVIDIA GPU Confidential Computing Demystified
- Stencil-Lifting: Hierarchical Recursive Lifting System for Extracting Summary of Stencil Kernel in Legacy Codes
- The Semantics of Program Slicing and Program Integration
- Revisiting Temporal Blocking Stencil Optimizations
- AN5D: Automated Stencil Framework for High-Degree Temporal Blocking on GPUs
- SparStencil: Retargeting Sparse Tensor Cores to Scientific Stencil Computations via Structured Sparsity Transformation
- PRESAGE: PRivacy-preserving gEnetic testing via SoftwAre Guard Extension
- Exploiting Scratchpad Memory for Deep Temporal Blocking: A case study for 2D Jacobian 5-point iterative stencil kernel (j2d5pt)
- Stencil-HMLS: A multi-layered approach to the automatic optimisation of stencil codes on FPGA