Many low-power cameras look at the same scene in two different wavebands: visible light by day and near-infrared (NIR) at night, when an infrared illuminator switches on. Matching a person seen in daylight to the same person seen in NIR is hard because the two images differ in color, texture and contrast. The usual fix is software that splits each image's features into a "shared" part that describes the person and is the same in both bands, and a "specific" part that holds cues peculiar to one band. That software runs after the full image has been read out and digitized, which costs the most energy in a small edge device. We propose moving this split into the lens. A thin meta-optical element (a flat surface patterned with sub-wavelength pillars that reshapes incoming light) is designed together with a small digital network. It sends light onto three groups of detector pixels: one broadband group trained to respond identically to a scene in either band, and two groups sitting behind pixel-aligned spectral filters, so that each can only ever see its own band.
This matters because optical front-ends only save energy when the number of detector readouts is very small. Once that is true, the real design question is what those few readouts should measure. Our bet is that spending them on a physically separated shared and specific code, instead of on a mixed code learned for the task alone, gives better cross-band matching and better behavior when one band is missing, at the same readout budget. If that holds, it would be the first case where an optical encoder's structure is dictated by a representation-learning principle rather than by task loss alone.
Core Claim
Under a fixed, small budget of detector readouts, a co-designed optical encoder has two properties. First, it routes band-specific information through pixels that are spectrally isolated by physics. Second, it computes a band-invariant code on the remaining broadband pixels. Such an encoder beats a task-only optical encoder, and beats digital binning followed by digital disentanglement, on visible-to-NIR matching and on robustness when one band is absent.
The physical part of the separation is one-directional and should be stated honestly. A spectral filter guarantees that NIR light cannot leak into the visible-specific pixels, up to the filter's extinction ratio. Physics does not guarantee that the broadband shared pixels are free of band-specific content. That half still has to be learned. The research file's Gap 4 frames physical orthogonality as a substitute for decorrelation losses. We reject that framing: orthogonal channels are not statistically independent features. Instead, we measure how much of the disentanglement burden the hardware actually removes.
Relation to Prior Work
The motivation comes from multimodal disentanglement work showing that mixing modalities into one representation loses information. MISA splits text, audio and video into invariant and specific subspaces. It aligns the shared parts with a moment-matching loss, keeps the subspaces apart with a soft orthogonality penalty, and loses accuracy when the specific parts are removed. CH-SIMS documents how training only for a shared label contaminates per-modality signals. On our target task, JSIA-ReID separates identity from band style for visible-infrared person re-identification. It shows that global distribution matching alone reaches 32.1% Rank-1 on SYSU-MM01, while disentanglement plus instance-level alignment reaches 38.1%. We reuse the style of loss from MISA and the matching protocol from JSIA-ReID. All of these methods act on fully digitized images using per-modality digital backbones. The file notes this multiplies cost (the dual VGG16 encoders of Wang et al.), and that cost is exactly what an optical front-end could remove.
The method comes from optical encoders. Chang et al. replaced a network's first convolution with a diffractive element. Photonic advantage of optical encoders co-designs silicon-nitride meta-optics end to end under ordinary incoherent light. The spectral CNN chip places pixel-aligned spectral filters on a CMOS sensor and computes spectral inner products on natural light. We combine the meta-optic of the second with the per-pixel filters of the third. The difference is that none of these encoders is asked to produce a structured shared/specific code, and none sees more than one waveband as distinct inputs.
Three papers come closest, and each leaves our question open. Polarization diffractive networks multiplex several independent linear transforms into one diffractive volume by polarization, but they test only random synthetic matrices and assign no semantic role to the channels. LUMEN-PRO searches over which diffractive layers are shared versus task-specific across tasks, but works on single-modality images, and its "specific" means per task, not per sensor band. TOMFN runs multimodal fusion on photonic waveguide meshes. Its fusion is monolithic and low-rank, with no shared/specific split, and it uses one-dimensional waveguides rather than the two-dimensional free-space optics that Free-space optical encoder for computer vision argues suit images.
One paper constrains the design. Photonic advantage of optical encoders found about a 10% experimental gain over digital binning on MNIST at 3×3 to 4×4 readouts, and parity above 8×8, because conversion costs erase the savings. We therefore confine our claim to the low-readout regime and treat the readout budget as the independent variable. We do not claim advantage at full resolution.
Research Program
We use an incoherent-light imaging model, because a surveillance camera sees ordinary light, not lasers. Each detector zone $z$ records
$$y_z = \int F_z(\lambda)\,\big(\mathrm{PSF}_\lambda \ast I_\lambda\big)\,d\lambda,$$
The terms are defined as follows. $I_\lambda$ is the scene intensity at wavelength $\lambda$. $\mathrm{PSF}_\lambda$ is the wavelength-dependent point-spread function (the blur pattern the meta-optic produces for a point of light), set by the pillar layout. $\ast$ is 2D convolution. $F_z(\lambda)$ is the fixed transmission of zone $z$'s pixel filter: broadband for the shared zone $S$, visible-pass for $P_{vis}$, and NIR-pass for $P_{nir}$.
The pillar layout and a small digital back end are trained jointly with
$$\mathcal{L} = \mathcal{L}_{id}(S, P) + \alpha\,\mathcal{L}_{align}\big(S^{vis}, S^{nir}\big) + \beta\,\mathcal{L}_{\perp}(S, P),$$
The terms are defined as follows. $\mathcal{L}_{id}$ is the identity-matching loss. $\mathcal{L}_{align}$ is a moment-matching plus instance-level alignment term, applied to shared-zone features computed from visible-only and NIR-only renderings of each identity, following MISA and JSIA-ReID. $\mathcal{L}_{\perp}$ is the soft orthogonality penalty between shared and specific features. Cross-band matching at test time uses only $S$.
What we build is a differentiable simulator. It models dispersion of the pillars across 450–950 nm, filter extinction ratios, fabrication phase error, detector shot and read noise, and 8-bit readout, which the file lists as the main sources of the gap between simulation and hardware. We also build a readout-budget sweep that varies total pixels from 3×3 up to 16×16 feature maps and varies the fraction allocated to $S$. The allocation fraction is a research object in itself: we will report the fraction at which cross-band accuracy peaks, and how that fraction moves with the budget.
A tabletop validation follows. A fabricated meta-optic is placed in front of a filter-patterned sensor, and a subset of test identities is displayed under switchable visible and NIR illumination. It checks whether the simulated leakage and accuracy survive contact with hardware.
The encoder is static once made. The file's first tension says that static optics cannot re-route when data distributions drift. Our setting sidesteps part of this, because the day/night switch is a known change of band, not an unknown drift, and the digital back end remains trainable for everything else.
Baselines
The naive reference is digital binning. A standard lens and RGB-NIR sensor image the scene, and the image is block-averaged down to the same number of readouts as our encoder. The same small digital network is then applied, trained with the same identity loss. This mirrors the binned baseline in Photonic advantage of optical encoders and represents doing nothing optically.
The strongest existing method on this question is full-resolution digital disentanglement in the style of JSIA-ReID. That paper reports 38.1% Rank-1 and 36.9% mean average precision (mAP) on SYSU-MM01 all-search single-shot, and 48.1% Rank-1 on RegDB. RegDB uses thermal rather than NIR imagery, which our optics cannot see through one aperture, so we cite it only as context. We rerun a MISA-loss disentanglement model on the same full-resolution images with our back-end family, giving a reproducible ceiling. We do not expect to beat it at tiny budgets. Success means closing most of the gap to it while using orders of magnitude fewer readouts than it does.
The competitor that decides the question is a task-only optical encoder. It has the identical meta-optic parameterization, simulator, noise model, readout budget and back end, but no zone filters, no alignment term and no orthogonality term, and it is trained only for identity matching.
A second strong competitor is "optics then digital disentanglement". The same task-only optical encoder is followed by MISA-style shared/specific heads applied after readout.
Two ablations isolate our ingredients. One removes the pixel filters, so specific zones become broadband and separation relies on losses only. The other keeps the filters but drops $\mathcal{L}_{align}$.
For fairness, every optical variant gets the same number of optimizer steps, the same hyperparameter search budget over $\alpha$, $\beta$ and learning rate, and the same noise draws. Results are reported over five fabrication-noise seeds.
Beating a baseline means one of two things, at equal readouts and with non-overlapping seed intervals. Either Rank-1 is higher on visible-to-NIR matching, or accuracy drops less when one band is removed at test time.
Why not just…?
Why not just train the optical encoder for the task and let it discover whatever code it needs? This is the real threat. The task-only encoder has strictly more freedom, because it can mix bands in every pixel. If the identity loss already forces band-invariant features, the task-only encoder will match ours, and the disentanglement structure is dead weight. Our prediction rests on the MISA and CH-SIMS evidence that training for a shared label lets the dominant band crowd out the weaker one: task-only encoders should do well within a band but worse across bands and with a missing band. The budget sweep settles this directly. If task-only wins or ties at every budget, we will report that the idea fails.
Why not just disentangle digitally after an ordinary optical readout? If the optical readout is a full-rank linear projection with enough pixels, a digital shared/specific head can in principle recover any split, and our hardware routing adds nothing. The argument for routing only applies when readouts are too few for that recovery. Once light from both bands is summed into one photodetector, the band identity of that light cannot be recovered digitally. We expect the two approaches to converge as the budget grows toward 16×16. The crossover point is itself a result we will report, and it tells a designer when the optical split is worth fabricating.
Why not just use digital binning and a low-power digital accelerator? At moderate resolutions this is probably the right engineering answer: Photonic advantage of optical encoders shows digital pipelines catch up above 8×8 readouts. That is why we claim nothing outside the low-readout regime. Energy accounting will count readouts, analog-to-digital conversions and digital multiply-accumulates for each system. We will not quote optical energy savings beyond what the cited papers measured.
Why not just use two cameras, one per band? Two cameras double the hardware, and they produce two unaligned codes. Cross-band matching then still needs digital alignment of those codes, which puts the full-resolution disentanglement cost back.
Proposed Experiments and Decisive Checks
The first check is controlled leakage, on a synthetic two-band dataset where the ground-truth factors are known. We render figures whose shape is shared across bands and whose texture pattern is band-specific. We measure how well a linear probe decodes the band-specific texture from $S$, and how well it decodes the cross-band texture from each $P$ zone. The prediction is near-chance decoding of NIR texture from $P_{vis}$ for our encoder, limited only by filter extinction, versus clearly above chance for the filter-free ablation. Decoding texture from $S$ should fall with $\alpha$. This check tells us how much of the separation the physics performs and how much the loss must still do.
The main experiment is SYSU-MM01 visible-to-NIR matching across the readout sweep. SYSU-MM01's infrared images are NIR, so they pass through the same simulated aperture. We report Rank-1 and mAP against all baselines. The decisive outcome is whether our encoder beats the task-only optical encoder at 3×3 to 8×8 equivalent budgets and converges with it above that.
The robustness experiment deletes one band at test time and measures the accuracy drop. It addresses the missing-modality weakness that CMAD highlights in digital models.
A noise stress test sweeps phase error and read noise to find where the advantage vanishes.
The tabletop prototype then checks one budget point in hardware. The go/no-go criterion is that the measured leakage and the ranking against the task-only encoder match simulation.
References
- MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality
- Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification
- Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks
- Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification
- Photonic advantage of optical encoders
- Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light
- Polarization diffractive networks: performing multiple linear transformations using a polarization-encoded diffractive optical network
- Automating multi-task learning on optical neural networks with weight sharing and physical rotation
- Tensorized Optical Multimodal Fusion Network
- Free-space optical encoder for computer vision
- CMAD: Correlation-Aware and Modalities-Aware Distillation for Multimodal Sentiment Analysis with Missing Modalities