Power Conservation as a Built-In Defense Against Noisy Labels in Photonic Networks

Deep networks trained on data with wrong labels first learn the real patterns and then slowly memorize the mistakes. One well-supported explanation is that the standard loss is unbounded: to fit a mislabeled image, the network pushes its output scores (logits) toward ever larger values, and the resulting large gradients bend the learned features. Clipping the size of the logit vector blocks this route and recovers much of the lost accuracy. A photonic neural network, one that does its matrix multiplications with light in interferometers, cannot push its outputs arbitrarily high in the first place. A passive optical mesh cannot emit more light than it receives, and photodetectors read out non-negative intensities. As a result, every output score of such a network lies in a bounded region set by the laser power and the detector gain, whatever the weights are.

This note proposes to test whether that physical ceiling works as a logit clipper that comes free with the hardware. It also proposes to turn the detector gain into a principled, noise-aware control that can be set without clean validation data. That second point is where the software version is known to be weak. If the idea holds, practitioners who train photonic networks through hardware-aware digital twins, meaning differentiable simulators of the chip, would learn two things: whether they need heavy noise-robust training at all, and how to set one physical dial so that it does the work of such training. The same experiments also produce a portable result for ordinary networks, namely whether a power-conserving output geometry resists memorization better than the standard cap on the length of the logit vector.

Core Claim

Consider a photonic classifier whose last stage is a passive interferometer mesh (lossless or lossy, so its transmission never exceeds one) followed by $K$ photodetectors, one per class. Let $P$ be the optical power entering the mesh and $g$ the detector gain. Each logit is then $z_k = g I_k$, where $I_k \ge 0$ is the intensity at port $k$ and $\sum_k I_k \le P$. The logits therefore live in a scaled simplex rather than in all of $\mathbb{R}^K$. The largest softmax confidence the network can express is reached when all light goes to one port:

$$p_{\max}(g) = \frac{e^{gP}}{e^{gP} + K - 1}.$$

Here $p_{\max}$ is the highest probability the network can put on any class. Because $p_{\max}$ is bounded below one, the cross-entropy loss on a mislabeled example stays bounded no matter how training proceeds. This is the property Mitigating Memorization of Noisy Labels by Clipping the Model Prediction enforces in software by rescaling the logit vector to have length at most $\tau$.

We make two claims. The first is that the physical bound delays or suppresses the memorization phase to a degree set by $gP$. The second is that a simple rule, choosing $g$ so that $p_{\max}(g)$ equals one minus an estimated noise rate, performs about as well as a LogitClip threshold tuned on clean validation data. The intuition behind the rule is that a network should never be allowed to be more confident than the labels are reliable. The rule needs only a noise-rate estimate, not a clean validation set.

There is also a geometric difference from LogitClip. The simplex constraint caps the margin between the top class and the others at $gP$ directly. LogitClip's length constraint allows large margins between logits of opposite sign. Whether this difference matters under instance-dependent human noise is an open empirical question, and we treat it as one.

Where This Sits

Three prior papers motivate the proposal by locating the mechanism of memorization. Early-Learning Regularization Prevents Memorization of Noisy Labels shows that "early learning then memorization" appears even in linear models trained by gradient descent. It also shows that gradients on mislabeled examples stay large while gradients on clean examples vanish, and it cancels those gradients with temporally averaged soft targets (ELR, ELR+). LogitClip attributes the same phenomenon to unbounded cross-entropy and caps the logit length. Does label smoothing mitigate label noise? shows that label smoothing acts as shrinkage in linear models and limits logit growth. All three change the software objective on unconstrained digital networks. None asks what happens when the hardware itself makes unbounded logits impossible.

Several photonic papers supply the physical components but constrain the design. Integrated multi-operand optical neurons for scalable and hardware-efficient deep learning uses the bounded cosine-squared response of a multi-operand interferometer as its activation, with 4-bit control, and reaches 85.89% on SVHN. Two laser papers provide active activations: Optical ReLU-like Activation Function Based on a Semiconductor Laser with Optical Injection gives a reconfigurable slope from 0.32 to 2.55, and 20-GHz bandwidth optical activation function based on a semiconductor laser gives ReLU-like behavior below 5 GHz, ELU-like behavior up to 20 GHz, and up to 3.1 dB of gain. That gain is a caveat for our claim. Once amplification enters, the "output never exceeds input" argument holds only up to the saturation power of the amplifying element, so the bound becomes $G_{\text{sat}}P$ rather than $P$. The experiments must track this. All of these papers evaluate only clean labels, and the research file found no prior work that trains optical networks on noisy labels.

Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations supplies the stress test. Human errors in CIFAR-10N depend on the individual image and are memorized faster than synthetic noise. Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels finds that the post-peak accuracy drop is small on real web noise. This cuts against us: if real noise produces little late-stage memorization, a mechanism that blocks late-stage memorization has less to gain. We report synthetic and human-noise results separately for this reason.

Research Program

We will build differentiable digital twins of three small photonic classifiers, trained offline as the photonic papers above do. The first is a multi-operand interferometer network with the measured cosine-squared activation and 4-bit phase quantization, following Feng et al. The second is a unitary mesh network with the injection-locked laser activation. We will run it in both the ReLU-like and the ELU-like regimes, with its gain and saturation modeled explicitly. The third is a mesh network with square-law detection only at the output, which serves as the purest passive case. In all three, the readout is $z_k = gI_k$, and $gP$ is swept over roughly a decade around the value where $p_{\max}$ equals the clean-label rate.

The analysis has three parts. First, we log per-example training loss and gradient norms for known-clean and known-mislabeled examples at each value of $gP$. This measures whether the persistent-gradient signature identified by ELR disappears under the physical cap, and at what $gP$ it returns. Second, we compare the simplex geometry against LogitClip's length constraint at matched maximum confidence. This separates the effect of how much confidence is capped from the effect of which shape of cap is used. Third, we test the noise-rate rule for $g$ using noise rates estimated from the training data alone, and measure how quickly accuracy degrades as the estimate is wrong by up to about ten percentage points. Only as a secondary question, we ask whether 4-bit quantization and phase noise add regularization beyond the power cap, and we compare this against injecting matched noise in software.

Baselines

The naive reference point is a digital multilayer network with the same number of trainable parameters as each optical twin. It uses ReLU activations, an unconstrained linear readout, and plain cross-entropy, and is trained with the same optimizer, epochs, data augmentation, and hyperparameter search budget. This model shows how much memorization occurs at this small scale at all. If it barely drops after its accuracy peak, the whole premise weakens, and we will report that result.

The strongest direct competitor is LogitClip applied to that same digital network. It is the closest software mechanism to our physical cap, and the file reports that it lifts cross-entropy on CIFAR-10 with instance-dependent noise from 68.36% to 86.74% (Mitigating Memorization of Noisy Labels by Clipping the Model Prediction). That result was obtained with ResNet/DenseNet-scale models and is not comparable in absolute terms to our small networks. We run LogitClip in two versions. The oracle version tunes $\tau$ on clean validation labels with the same search budget we give to $g$. The realistic version tunes $\tau$ on noisy validation labels. Beating LogitClip means one of two outcomes: the optical twin with the noise-rate rule matches the oracle version within one standard deviation over five seeds, or it clearly exceeds the realistic version. Both comparisons use the same data and noise draws.

The strongest overall noise-robust method in the file is ELR+. It reports 93.3% on CIFAR-10 with 80% symmetric noise, against 62.9% for cross-entropy, and 74.81% on Clothing1M (Early-Learning Regularization Prevents Memorization of Noisy Labels). We run single-network ELR, without the dual networks and Mixup, on the digital baseline and on the optical twins, and we also run it stacked with the optical cap. We do not expect a bare power cap to beat full ELR+. The relevant question is whether the cap captures most of ELR's gain at none of its per-example memory cost, and whether ELR adds anything once the cap is present. Label smoothing, with $\alpha$ chosen by the noise-rate rule, is the cheap shrinkage baseline. Lukasik et al. report that it moves CIFAR-100 accuracy under 20% noise from 57.06% to 60.70% with ResNet-32.

The ablation that isolates the key ingredient is an optics-free digital network. It uses unitary weight matrices, non-negative squared-magnitude outputs, and a fixed input norm, so the logits are confined to the same simplex, but it has no quantization, phase noise, or optical activation. If this network matches the optical twins, the benefit comes from the power constraint, not from anything specific to photonics, and we will say so. A second ablation removes the ceiling from the optical twin by setting $gP$ very large, which checks that the benefit disappears when the cap is loosened. For every existing noisy-label method, we found no reported result on Lacuna for optical or photonic networks.

Why not just…?

Why not just apply LogitClip in software? On a digital accelerator, one should, and our digital ablation will show whether the simplex geometry beats the length cap. The case for the optical route is narrower. In a photonic network the cap cannot be avoided, since it exists whether or not anyone designs for it, so the practical question is how to set it. LogitClip's weakness is choosing $\tau$ without clean data. If the noise-rate rule for $g$ does not match the oracle-tuned $\tau$, this alternative wins and the proposal reduces to a characterization study.

Why not just use ELR+? ELR+ is stronger on the benchmarks in the file. However, it stores a running prediction vector for every training example, trains two networks, and uses Mixup, and CIFAR-N reports that such methods lose 1–5% moving from synthetic to human noise at the same noise rate. The stacked experiment settles the comparison. If ELR on top of the cap adds large gains under human noise, then the cap alone is insufficient, and the hardware bound is best presented as a complement to ELR rather than a replacement.

Why not just use label smoothing? Label smoothing also bounds the optimal confidence. Lukasik et al. show it acts as shrinkage, but they report its gains only at moderate symmetric noise. It changes the targets, whereas the cap changes what the model can output. These differ on instance-dependent noise: smoothing still rewards the network for moving toward a wrong label, while the cap limits how far it can move. A matched-confidence comparison on CIFAR-10N will show whether the difference is real. If it is not, smoothing is the simpler recommendation.

Why not just add quantization or weight noise to a digital network? This is the natural reading of the hardware-imprecision gap, and we treat it only as a secondary hypothesis because it is easy to emulate in software. The noise-matched digital control answers it directly.

Proposed Experiments

Datasets are MNIST, Fashion-MNIST, and SVHN, on which the photonic papers above already report clean accuracy. We add symmetric noise at 20%, 40%, and 80%, and instance-dependent noise. For real human noise we use CIFAR-10N (the Aggregate, Random, and Worst sets, at roughly 9%, 17%, and 40% noise) and CIFAR-100N, fed to the optical networks as downsampled inputs. Absolute accuracy on these will be low, so we report the drop relative to training the same architecture on clean labels rather than headline accuracy. The metrics are peak and final test accuracy, the gap between them as a measure of memorization, the fraction of mislabeled training examples fit by the end of training, and per-group gradient norms. All results are averaged over five seeds.

The decisive check is a plot of the peak-to-final gap against $gP$ across all three optical architectures and the optics-free ablation. The claim is supported if the gap stays near zero below the value set by the noise-rate rule and rises beyond it, if the rule's final accuracy is within seed noise of oracle-tuned LogitClip, and if all of this holds on CIFAR-10N and not only on synthetic noise. The claim is refuted if the unconstrained digital baseline shows no memorization at this scale, or if the optical twins memorize regardless of $gP$. The second outcome would mean the network can overfit by redistributing light among output ports without ever needing large logits. For the laser-activation twin, we additionally check whether the bound follows $G_{\text{sat}}P$ as predicted, or whether active gain breaks it.