A holographic display shapes the light wave itself rather than just pixel brightness, so in principle it can pre-distort light so that a viewer's own imperfect eye focuses it sharply. That would let people skip glasses inside AR and VR headsets, and it would also correct irregular aberrations that glasses cannot fix. Two things block this in wearables. First, computing holograms is expensive: a neural hologram generator with a built-in aberration-correction module runs at 5 frames per second on a phone (End-to-end learning of 3D phase-only holograms for holographic display). Second, the correction is not fixed. When the eye rotates, the pupil moves and the eye's aberrations are sampled differently across the field of view, so the pre-distortion has to change within milliseconds.
We propose to treat the eye's aberration as a spatially varying optical operator and to approximate it as a weighted sum of $K$ shift-invariant blur kernels whose weights depend on gaze. The number of kernels $K$ then becomes a shared design knob: it controls both how well the eye is corrected and how much memory traffic and compute a custom accelerator must handle. We would search jointly over the hologram network, $K$, and the accelerator's dataflow. An event-based eye tracker would update only the $K$ mixing weights, not the network. The two-sentence version is this: vision correction on a holographic display is cheap when the eye's blur is identical across the field and expensive when it varies with position and gaze. We would measure how much variation actually matters, then build hardware whose cost scales with that measured amount instead of the worst case.
Core Claim
Much of the literature treats "aberration correction" as a single multiplicative phase at the pupil. That treatment is exact only for a blur that is the same everywhere in the image and a pupil that stays in place. Under that assumption, correction adds one complex multiply per pixel to an FFT-based propagation, and the accelerator problem reduces to the generic problem of accelerating hologram generation.
Our claim is that realistic eye models become spatially varying once gaze moves and accommodation changes, and that this variation is low-rank. We write it as
$$ h(\mathbf{x};\,\mathbf{g}) \approx \sum_{k=1}^{K} w_k(\mathbf{x},\mathbf{g})\, h_k(\mathbf{x}), $$
where $h(\mathbf{x};\mathbf{g})$ is the local wave-optics response of the eye at image position $\mathbf{x}$ when the gaze state (pupil position and accommodation) is $\mathbf{g}$. Each $h_k$ is a fixed shift-invariant kernel, and each $w_k$ is a smooth per-position weight map.
This structure mirrors the multi-channel convolution model that Waveguide Holography: Towards True 3D Holographic Glasses calibrated interferometrically to undo scrambling inside the waveguide. We reuse that method but apply it to the viewer's eye rather than the waveguide, and we make the weights driven by gaze.
The hardware consequence is concrete. The $K$ kernels are precomputed per user and stored on chip, so a gaze update only rewrites the small weight maps. The accelerator's buffer sizes and FFT tiling can then be matched to the smallest $K$ that meets a target retinal fidelity. What would be new is a hologram generator, a correction operator, and an accelerator searched together, with a measured rank of eye aberration linking fidelity to energy.
Research Program
The first step is measurement, and it could falsify the whole program. Using wave-optics eye models with Zernike aberration profiles (both typical and irregular higher-order profiles), we would compute the exact spatially varying response across a sweep of fields of view and gaze. The gaze trajectories would come from the Annotated Event-based Gaze and Eye Position Dataset by Angelopoulos et al.. We would then fit the low-rank model for increasing $K$. The output is a curve of retinal-image fidelity against $K$ for each field of view. That curve tells us whether the 8° field of current waveguide prototypes needs any spatial variation at all, and how quickly the need grows at wider fields.
The second step builds the differentiable pipeline. A HOLONET-style generator of the kind used in Artifact-Resilient Real-Time Holography, which reaches 85 FPS on a desktop but without refractive correction, would produce a phase pattern. Angular-spectrum propagation implemented with WaveBlocks (Learning to Model and Calibrate Optics Via a Differentiable Wave Optics Simulator) would carry that pattern to the eye. The low-rank eye operator would then form the retinal image.
The third step extends the co-search of DNA: Differentiable Network-Accelerator Co-Search. DNA searches network operators jointly with loop tiling, loop order, and processing-element arrays, but only for image classifiers built from ordinary convolutions. We would add three operator types to its accelerator design space: complex-valued FFTs, per-position multiplies by the kernel weights, and multi-plane propagation. We would also add $K$ and numerical precision as searchable variables. The memory-traffic concern raised by AQPIM guides the cost model: we expect off-chip movement of multi-plane complex fields to dominate energy, and $K$ multiplies that movement directly.
The fourth step is a streaming interface. Gaze events would update the weight maps between frames, so that correction latency is set by weight recomputation rather than by a full network pass.
Implementation would target an FPGA prototype plus an analytic, cycle-level cost model for ASIC projections. Fidelity would be validated on a benchtop holographic display, using a camera behind phase plates that emulate aberrated eyes. Because of the phase-randomness vulnerability documented in Artifact-Resilient Real-Time Holography, its phase-randomness regularizer would be kept, so the corrected holograms stay robust to eyelashes and floaters.
Baselines
The do-nothing reference is an uncorrected hologram viewed through the aberrated eye model. It fixes the floor for retinal-image PSNR and SSIM.
The space-invariant correction baseline applies a single conjugate Zernike phase at the pupil plane, updated for pupil shift. This is the $K=1$ special case and the cheapest credible method. Its role is to isolate the proposal's key ingredient, the spatially varying low-rank operator. It will use the same generator, data, and hardware cost model as the proposal. We count the proposal as beating it only if larger $K$ improves retinal PSNR by more than 1 dB, averaged over gaze trajectories, at a field of view we name in advance.
The strongest existing fidelity baseline is the neural 3D hologram pipeline with an aberration-correction module from End-to-end learning of 3D phase-only holograms for holographic display. On MIT-CGH-4K-V2 it reports 29.6 dB PSNR and 0.947 SSIM, and 5 FPS on an iPhone 13 Pro. We would retrain it on the same aberrated-eye targets and run it on the same FPGA and cost model through a standard toolchain, with INT8 quantization and an equal tuning budget. Beating it means matching its PSNR within 0.5 dB at 60 FPS or more, at lower modeled energy per frame, and under gaze-varying aberration, which it does not model.
The sequential-design baseline trains the best generator first and then separately searches for the accelerator that best fits it. It isolates the value of joint search. DNA reports 3.04× better FPS than the HS-Co-Opt baseline on the same FPGA budget and a 324× Energy-Delay Product reduction over NASAIC, in classification (DNA). No reported result was found on Lacuna for co-search applied to hologram generation. We would therefore report our own joint-versus-sequential gap, with equal search compute for both.
Why not just…?
Why not just multiply by the conjugate aberration phase and spend all effort on generic hologram acceleration? This is the most serious threat. At the 8° field of Waveguide Holography, the eye's aberrations may barely vary across the field. The rank-measurement study is designed to settle this before any hardware is built. If $K=1$ stays within 0.5 dB of the exact operator across realistic gaze trajectories and fields up to about 30°, we drop the low-rank operator. The contribution then narrows to co-designing the gaze-update path, and we will say so.
Why not just run a fast network on a commercial phone neural processor with quantization? The only reported mobile number for aberration-aware 3D holography is 5 FPS (End-to-end learning of 3D phase-only holograms for holographic display). Wearable power and heat limits are a stated constraint in Lightweight holographic near-eye display system with self-charging capability using solar energy. The strongest-baseline row runs exactly this alternative on equal hardware. If quantization alone reaches the target, the joint search has not earned its cost.
Why not just use prescription inserts or a varifocal lens and correct nothing digitally? Physical lenses fix defocus and regular astigmatism well. However, Vision Correcting Displays Based on Inverse Blurring and Aberration Compensation notes that such lenses cannot fix irregular higher-order aberrations. We would test a hybrid in which an insert corrects the low-order terms and our system corrects only the residual. The residual likely has lower rank, which would mean smaller $K$ and cheaper hardware. If the hybrid wins, that partition becomes our recommended design.
Proposed Experiments
All image experiments use MIT-CGH-4K-V2 scenes rendered through simulated aberrated eyes. The aberration profiles cover typical and irregular higher-order Zernike terms, with accommodation from 0 to 5 diopters. Gaze sequences are replayed from the Angelopoulos event dataset. We report retinal PSNR and SSIM as functions of gaze angle. For hardware we report FPS, modeled energy per frame, off-chip bytes per frame, and gaze-to-photon latency, all on the FPGA prototype and the ASIC cost model.
There are three decisive checks. The first is the rank curve described above, which decides whether spatially varying correction matters. The second is joint versus sequential search at a matched energy budget, which decides whether co-design matters for this workload. The third is benchtop capture through irregular-aberration phase plates while pupil position is steered. There we must show that corrected contrast holds across a 3 mm or larger pupil excursion, the scale that Artifact-Resilient Real-Time Holography identifies as the minimum robust eyebox. Updates would be weight-only, with no network re-run.
References
- End-to-end learning of 3D phase-only holograms for holographic display
- Waveguide Holography: Towards True 3D Holographic Glasses
- Annotated Event-based Gaze and Eye Position Dataset by Angelopoulos et al. "Event-Based Near-Eye Gaze Tracking Beyond 10,000 Hz"
- Artifact-Resilient Real-Time Holography
- Learning to Model and Calibrate Optics via a Differentiable Wave Optics Simulator
- DNA: Differentiable Network-Accelerator Co-Search
- AQPIM: Breaking the PIM Capacity Wall for LLMs with in-Memory Activation Quantization
- Lightweight holographic near-eye display system with self-charging capability using solar energy
- Vision Correcting Displays Based on Inverse Blurring and Aberration Compensation