Relative Patch Rotation as a Regularizer for Calibrated Photometric Redshifts

Most galaxies in large sky surveys will never get a spectrum, so their distance, measured as redshift, has to be inferred from multi-band images. That inference is useful to cosmology only if the network returns an honest probability distribution over redshift rather than a single guess. When spectroscopic labels are scarce, the networks that produce these distributions tend to overfit to color cues. They become overconfident, and they fail catastrophically on galaxies whose colors are ambiguous. A standard remedy in computer vision is to train the network on a second, label-free task at the same time, so that the shared features stay rich. Asking the network which way an image was rotated is one of the most effective such tasks. The difficulty is that a galaxy has no "up." Asking which way a whole sky image was rotated is a question with no answer. Any network that answers it above chance must be reading instrument artifacts aligned with the detector's pixel grid.

We propose a version of rotation prediction that is well posed on the sky. We rotate a patch inside the galaxy relative to its surroundings, and ask the network to predict that relative rotation while it also learns the redshift distribution. The label is defined by how the galaxy's own structure is laid out, not by the camera frame. It therefore stays meaningful under the random whole-image rotations that astronomers already apply. The central bet is specific: this relative-rotation task will improve the calibration and outlier rate of redshift probability densities in the label-scarce regime, where plain augmentation and invariance penalties have been shown to do little.

Core Claim

The closest existing test is by Engel and Strube. They trained a ResNet50 to output a conditional density over redshift (a full predicted distribution per galaxy) on SDSS DR12 64×64 ugriz cutouts. To this they added a "soft physical constraint" that penalizes changes in the predicted distribution when the input is rotated or flipped. The result was a median absolute deviation of 1.12×10⁻² against 1.17×10⁻² for plain augmentation, at triple the training time (24.52 h versus 8.16 h). The unaugmented baseline kept the best outlier rate, 1.48%. That paper both motivates and constrains this proposal. It shows that telling the network "ignore rotation" adds almost nothing beyond augmentation. Any rotation-based regularizer must therefore make the network learn something, not merely be invariant.

The vision literature supplies the mechanism. RotNet trains a network to classify which of four rotations was applied to an image. Zhao and Itti show that rotation or pose prediction as a side task shifts filters from texture toward shape. CRAT combines an outer rotation with inner rotations of image halves, which breaks spatial symmetry and raised mini-ImageNet 1-shot accuracy from 64.82% to 68.44%. PatchRot predicts per-patch rotations in a Vision Transformer and adds a buffer gap so that edge continuity cannot give the answer away. PatchRot also reports that local rotation still teaches useful features when the whole object is rotationally ambiguous. All four methods were run on terrestrial classification. Each keeps a global-orientation component that is meaningless on the sky. None has been joined to a probabilistic regression target.

Our objective removes the global component entirely. Let $x$ be a cutout, $g$ a random whole-image rotation or flip applied as ordinary augmentation, and $T_{k}$ the operation that rotates a central inner patch by $k \cdot 90^\circ$ relative to its frame, with $k \in \{0,1,2,3\}$. We train a shared encoder with two heads:

$$\mathcal{L} = -\log p_\theta\big(z \mid g(x)\big) \;+\; \lambda \,\mathrm{CE}\big(k,\; q_\phi(k \mid T_k(g(x)))\big),$$

Here $p_\theta(z \mid \cdot)$ is the predicted redshift density and $z$ the spectroscopic redshift. $q_\phi$ is the relative-rotation classifier and CE is cross-entropy. $\lambda$ weights the auxiliary loss, and it is the only new hyperparameter. The label $k$ does not change under $g$, because rotating the whole image does not alter the relative angle between patch and surround. For that reason the pretext task and the standard augmentation are compatible rather than competing.

The second term is also applied to unlabeled cutouts, in the spirit of S4L. S4L showed that rotation prediction on both labeled and unlabeled images regularizes semi-supervised classification. We keep one training run with no separate pretraining stage.

Research Program

The first component is building the task so that it cannot be solved by shortcuts. The inner patch is centered on the galaxy's light, and its size is set as a fraction of a measured half-light radius, not in fixed pixels. Only 90° multiples are used, so no interpolation artifact marks the rotated region. Following PatchRot, a thin seam around the patch is replaced with noise matched to the local sky background. This removes the edge-mismatch cue. Galaxies too small to have internal structure at this resolution give an unsolvable task, and the network can only output a uniform guess, which costs no gradient bias. We record per-galaxy pretext accuracy as a function of angular size, both as a diagnostic and as a secondary scientific quantity.

The second component is a deliberate negative control: a global RotNet head trained on the same survey images. On a statistically isotropic sky it should sit at chance. If it rises well above chance, the network has found pixel-grid systematics such as bleed trails, anisotropic point-spread functions, or diffraction spikes. We expect that head to hurt redshift calibration. Showing this directly tests the "isotropy paradox" that has kept astronomers treating rotation only as noise to average out.

The third component is evaluation that treats the full redshift distribution as the product. We report median absolute deviation and catastrophic outlier fraction on the normalized residual. We also report the continuous ranked probability score, a proper scoring rule for predicted distributions. Calibration is measured with the probability integral transform, which checks where the true redshift falls within each predicted distribution and should be uniform. We add empirical coverage of the 68% and 95% intervals. Every metric is stratified by redshift and by angular size. Henghes et al. found that images help most at z < 0.3, where their mixed-input CNN reached MSE 0.0007, a 50% improvement over a random forest. That tells us where a structural regularizer should matter. We predict gains concentrated at low redshift and large angular size, shrinking toward zero for unresolved, high-redshift galaxies. A uniform gain across all bins would suggest the effect is generic regularization rather than structure learning.

The downstream motivation is honest but bounded. The research file stresses redshift-starved sources in gamma-ray astronomy: BL Lac blazars with featureless spectra (Albert et al., Aleksić et al.), and GRBs whose energetics require distance (Longo et al.). There, what matters is calibrated redshift densities that propagate correctly into extragalactic background light de-absorption. Our method applies only to resolved sources such as host galaxies and field galaxies. It does not help point-like nuclei, and we do not claim it does.

Baselines

The naive reference is the conditional-density ResNet50 of Engel and Strube, trained with and without random rotation and flip augmentation. We reproduce their split of 486,000 training and 121,000 test SDSS DR12 cutouts, so their reported numbers can serve as an anchor. Those numbers are MAD 1.17×10⁻² with augmentation, 1.48% outlier rate without augmentation, and 8.16 h training. Beating this reference at full labels means a lower MAD with no increase in outlier rate. Because we expect small differences at full labels, the more important test is at 1%, 5% and 20% label fractions. There, beating it means better interval coverage and CRPS with seeds' confidence intervals not overlapping.

The strongest existing competitor on this exact data is Engel and Strube's physically constrained model, including the rotation-invariance penalty, at a reported MAD of 1.12×10⁻² and 24.52 h training. We rerun it on our label-fraction subsets with the same backbone, epochs and hyperparameter search budget as our method. Our method must match or beat its MAD while staying near the augmentation baseline's wall-clock cost. The relative-rotation head adds one extra forward pass per labeled or unlabeled image, versus the multi-view evaluation of the penalty. The method must also show a calibration improvement that the penalty does not.

The strongest representation-learning competitor is two-stage contrastive pretraining with MoCo v2 on about 1.2 million unlabeled SDSS cutouts followed by fine-tuning, as in Hayat et al. and Hayat et al. 2021. Those papers report 2–4× label efficiency and bias Δz < 10⁻⁴. The file contains no MAD or calibration number for them on the Engel–Strube split. We will run it ourselves on the same unlabeled pool, with total GPU-hours matched to our joint run, and fine-tune with a density head. Beating it means equal or better CRPS and coverage at 1–5% labels for equal compute.

AstroCLIP (Parker et al.) is included as a frozen-embedding reference on the DESI Legacy Survey transfer set. The file reports only its spectrum-based redshift R² of 0.986, from the Multimodal Universe replication. No reported result was found on Lacuna for AstroCLIP image-based photo-z MAD. We therefore treat it as context, not as a head-to-head target.

Four ablations isolate the key ingredient. The first is the global RotNet head, the negative control described above. The second is a CRAT-style half-image head, which retains a camera-frame outer rotation. The third removes the seam masking, to test for edge shortcuts. The fourth is a compute-matched auxiliary head with random labels, which separates "any second head regularizes" from "this task teaches structure."

Why not just…?

Why not just use random rotation augmentation? It is already in every baseline and is not in competition with the proposal. The relative-rotation label is unchanged under whole-image rotation, so both are always applied together. The real question is whether augmentation alone leaves anything on the table. Engel and Strube's result is ambiguous here: invariance-style constraints added only 0.05×10⁻² in MAD. If our joint model also gains nothing over augmentation at every label fraction, the idea is refuted, and the label-fraction sweep is designed to show that.

Why not just pretrain contrastively and fine-tune? This is a serious competitor, and Hayat et al. showed it delivers label efficiency on the same survey. Our argument is not that it fails. It is that contrastive objectives explicitly remove orientation information and are tuned to global similarity, while redshift calibration may depend on internal structure that the pretext keeps. The compute-matched comparison at 1–5% labels settles this. If MoCo fine-tuning matches us on CRPS and coverage, the practical recommendation becomes "use the pretrained model," and we will report it that way.

Why not just add a masked-patch reconstruction head instead of rotation? This is the most plausible threat, since AstroCLIP's image encoder already uses masked modeling. Reconstruction on galaxy cutouts may spend capacity modeling smooth light profiles and sky noise, whereas rotation discrimination targets how structure is arranged. That argument is untested, so we include a masked-reconstruction head with an identical encoder, $\lambda$ search and compute. If it matches relative rotation, the contribution narrows to "joint auxiliary losses help photo-z calibration," with rotation not special.

Why not just hard-code rotation invariance? Dieleman et al. won the Galaxy Zoo challenge (RMSE 0.07492) by pooling 16 rotated and flipped views through shared layers. That multiplies the forward cost, and like the invariance penalty it tells the network what to ignore rather than what to learn. We include a 16-view test-time-pooled version of the augmentation baseline to check whether cheap invariance at inference closes the calibration gap on its own.

Proposed Experiments

The primary experiment trains all methods on SDSS DR12 at label fractions of 1%, 5%, 20% and 100%, with five seeds each. The remaining roughly one million images, plus the unlabeled pool, are available only to methods that use unlabeled data. Metrics are MAD, outlier fraction, CRPS, PIT uniformity and 68/95% coverage, stratified by redshift and angular size.

The decisive checks are three. First, at 5% labels, relative rotation must improve 95% coverage and CRPS over both augmentation and the invariance penalty, with gains concentrated in low-redshift, large-angular-size bins. Second, the global RotNet head should stay near chance, or its above-chance accuracy should coincide with degraded calibration, which would confirm that camera-frame rotation labels are harmful on the sky. Third, the random-label and masked-reconstruction heads must fall short of relative rotation. Otherwise the structural explanation fails.

A transfer test then fine-tunes the SDSS-trained models on DESI Legacy Survey cutouts from the Multimodal Universe with 5% labels. This checks whether the structure learned from one instrument's point-spread function carries over or was itself instrument-specific. We also report training wall-clock time for every run, since Engel and Strube showed that a 3× time cost can wipe out a small accuracy gain.