Networks that predict angles, such as a person's heading seen from a drone, a car's yaw, or a gripper's rotation, usually output the pair $(\cos\theta, \sin\theta)$ instead of the raw angle. This avoids the jump at 360°. Semi-supervised versions of these models, which learn from a few labeled images plus many unlabeled ones, then treat that pair as a single point. Their pseudo-labels, meaning the model's own guesses used as targets on unlabeled images, are single points too. When an image is ambiguous, for example a person seen from directly above who could be facing forward or backward, a point target can only average the two hypotheses into a vector near zero, or pick one and reinforce a possible mistake.
We propose a simple change. The teacher network's "soft label" for an angle should be a short vector of circular harmonics, $(\cos k\theta, \sin k\theta)$ averaged under the teacher's belief for $k = 1, \dots, K$. The student is trained to match these with ordinary squared error. Three facts make this more than a relabeling. First, the squared-error-optimal output of a harmonic head is the conditional average of those harmonics, so a plain regression head already learns a compressed predictive distribution if we stop normalizing it. Second, squared error on harmonics equals the squared distance between the corresponding smoothed densities on the circle. Third, averaging harmonic vectors across augmented views is exactly mixing the distributions. The front/back ambiguity that destroys a point target shows up cleanly in the second harmonic. The claim we test is that this harmonic soft label, at $K \approx 2$–4 outputs, carries the uncertainty that matters for semi-supervised orientation learning. We also ask whether it does so as well as or better than the binned soft labels a classification-minded reviewer would reach for, and better than point pseudo-labels.
Core Claim and Mechanism
Let $p(\theta \mid x)$ be the teacher's belief about the angle $\theta$ for image $x$. Its $k$-th circular moment, also called a trigonometric moment, is
$$m_k(x) = \mathbb{E}_{p(\theta\mid x)}\left[(\cos k\theta,\ \sin k\theta)\right] \in \mathbb{R}^2 .$$
Its length $|m_k|$ is between 0 and 1. A length near 1 means confident, and a length near 0 means spread out or multi-modal at that frequency. The student outputs $\hat m_k(x)$ for $k = 1..K$ and minimizes
$$\mathcal{L} = \sum_{k=1}^{K} \left\| \hat m_k(x) - m_k(x) \right\|^2 .$$
On labeled images the target is the moment of a point mass, $(\cos k\theta^*, \sin k\theta^*)$. On unlabeled images the target is the teacher's predicted moments, averaged over weak augmentations and dropout. By Parseval's identity (the Fourier-series fact that squared error in coefficients equals squared error between functions), this loss is the $L_2$ distance between the two densities truncated to $K$ frequencies.
The decoded angle is the peak of the truncated density, evaluated on a fine grid. The vector length $|\hat m_1|$, together with how $|\hat m_2|$ compares to it, gives a calibrated confidence and an "axial ambiguity" score. Both come for free, with no bins, no bin-width parameter, and none of the Bessel-function normalizers needed for von Mises likelihoods.
The ingredients come from different roles in prior work. The $(\cos\theta, \sin\theta)$ head is the standard fix for wrap-around in Improved Generalization of Heading Direction Estimation for Aerial Filming Using Semi-supervised Regression, in quadrotor modeling (Learning quadrotor dynamics using neural network for flight control) and in hand-object pose (Semi-supervised 3D Hand-Object Pose Estimation via Pose Dictionary Learning). All three normalize or treat the output as a point, and we deliberately do not.
The doubled-angle grasp head $(\cos 2\theta, \sin 2\theta)$ of 2.5D Image-based Robotic Grasping is, in our view, the $k=2$ harmonic used alone to encode a known symmetry. We keep both $k=1$ and $k=2$ so the network can learn per image how much symmetry is present.
On the distillation side, the motivating evidence is that soft distributions transfer more than point targets: GKD found soft outputs matter more than matching hidden features, and Noisy Self-Knowledge Distillation for Text Summarization used dropout at teacher inference to expose uncertainty. Both apply only on discrete label sets. DINO as a von Mises-Fisher mixture model is the closest bridge between distillation and directional statistics. It reads self-supervised distillation as fitting a mixture of hypersphere distributions over latent features. We instead distill a task-level physical angle, and we use moments rather than a parametric likelihood.
Research Program
We build a mean-teacher semi-supervised loop, in which the teacher is an exponential moving average of the student's weights. Each unlabeled image is passed through the teacher under several weak augmentations with dropout active. The resulting harmonic vectors are averaged, with angle-changing augmentations such as flips undone by applying the matching rotation of the harmonics: horizontal flip sends $\theta \to \pi - \theta$, which is a fixed linear map on each harmonic. The student sees a strong augmentation of the same image and regresses the averaged harmonics.
Because the targets are unnormalized, a confused teacher sends a near-zero target. The student is then pulled toward admitting uncertainty rather than toward a wrong confident angle. This is the mechanism we expect to reduce confirmation bias, the open problem that self-training reinforces its own early mistakes.
The analysis has three parts. First, we check calibration: on labeled validation data, does $|\hat m_1|$ predict angular error, and does it match the moment length of the empirical error distribution? Second, we check ambiguity: on images with true front/back confusion, does $|\hat m_2| \gg |\hat m_1|$ appear, and does the decoded density place two peaks? Third, we check collapse. Improved Generalization of Heading Direction Estimation for Aerial Filming Using Semi-supervised Regression reports that its temporal continuity loss can drive the model to a constant heading under poor weighting. We sweep the unlabeled-loss weight over two orders of magnitude for every method and report the fraction of runs whose prediction variance across the test set collapses.
Class imbalance is a known weak point of soft labels: Is Label Smoothing Truly Incompatible with Knowledge Distillation reports that uniform smoothing hurts long-tailed classes. Heading distributions are similarly peaked, with cars mostly parallel to the road. We therefore report error separately on rare angle ranges.
Baselines
The naive reference is a supervised-only $(\cos\theta, \sin\theta)$ regressor trained on the labeled subset with normalized output, using the same backbone. Every semi-supervised method must beat it to count.
The strongest published semi-supervised competitor on the drone benchmark is the temporal-continuity method of Improved Generalization of Heading Direction Estimation for Aerial Filming Using Semi-supervised Regression. There, a ground-pretrained network scoring 26.67% heading accuracy on aerial footage rose to 72.2% after semi-supervised fine-tuning. We reproduce it on the same MobileNet backbone, ground-level pretraining and labeled split, and tune its continuity weight with the same budget we give our method. Beating it means exceeding 72.2% under their protocol, or exceeding our reproduction by more than the seed-to-seed spread across five seeds. Because our method does not need video, we also report a variant that adds their continuity loss on top. This tells us whether the two signals are complementary.
The point-pseudo-label mean teacher is identical to our pipeline except that teacher outputs are normalized to unit length and only $k=1$ is used. This is the ablation that isolates our key ingredient: whether the teacher's uncertainty, carried as vector length and higher harmonics, matters.
The binned soft-label baseline discretizes the circle into 72 bins with wrap-around, like the classification half of MultiBin (3D Bounding Box Estimation Using Deep Learning and Geometry) and Bin & Delta (Monocular Object Orientation Estimation using Riemannian Regression and Classification Networks). It smooths labeled targets with a circular Gaussian and distills teacher histograms with temperature-scaled KL divergence. Bin count, smoothing width and temperature all get the same tuning budget as our $K$ and loss weight.
The parametric baseline predicts a single von Mises mean and concentration and distills by minimizing the KL divergence between teacher and student von Mises distributions. It shows whether a one-mode distribution is enough.
The internal ablations vary $K \in \{1, 2, 4, 8\}$, the number of averaged teacher views, and whether dropout is active in the teacher.
Why not just…?
Why not just discretize the angle into bins and use standard soft-label distillation? This is the real threat. A wrap-around histogram is just another basis for densities on the circle, so it can represent multi-modality too. We do not argue around it. The binned baseline gets an equal tuning budget, and the decisive comparison is accuracy and calibration at matched compute. Our expected advantages are specific and testable. Our head is a drop-in for existing trigonometric regressors, with 2–8 outputs instead of 72. It has no bin-width or temperature parameters. Its loss is a true distance between densities, not a KL that can blow up when a bin gets near-zero probability.
If bins match us on accuracy and calibration, the contribution narrows. It becomes the finding that low-order harmonics suffice, plus the cheaper head, and we will report it that way. If bins win, the harmonic idea fails, and that is informative too.
Why not just tune the temporal continuity loss of Wang et al. more carefully? Their own paper reports that the weight is fragile and can collapse the model to a constant heading. Continuity also requires video and says nothing about a single ambiguous frame. Our collapse sweep, run identically on both methods, directly measures whether harmonic distillation is less brittle. The combined variant tests whether continuity adds information on top.
Why not just predict a von Mises distribution, or a mixture of them, and distill by likelihood? A single von Mises distribution cannot represent front/back ambiguity, which is the case we care about. Mixtures need a chosen number of components and normalizers involving Bessel functions. DINO as a von Mises-Fisher mixture model found that approximating these normalizers mattered for stable training. The single von Mises baseline measures the cost of one mode. A two-component mixture is run on the ambiguous subset only, where its advantage should be largest.
Proposed Experiments
The primary benchmark is the drone filming dataset of Wang et al.: about 50 aerial videos with 6 labeled frames each, roughly 1.2% of frames. We use ground-level surveillance data (VIRAT, DukeMCMT) for pretraining as in that work, and report their heading accuracy metric at their 1% and full label fractions.
The secondary benchmark is PASCAL3D+ azimuth estimation, with 5% and 10% of training labels kept and the rest used unlabeled. We report median angular error and the fraction of predictions within 30°. The file contains no reported semi-supervised result on PASCAL3D+. The fully supervised Riemannian model of Monocular Object Orientation Estimation using Riemannian Regression and Classification Networks reports a median error of 10.10° for full 3D rotation. That number serves as a full-label reference point rather than a direct competitor, since we estimate azimuth only. We also tag test images from near-symmetric categories, such as boats and dining tables seen end-on, as an "ambiguous" subset.
The third benchmark is the Cornell grasp dataset with labels subsampled. Grasp angles are inherently two-fold symmetric, so the model should learn to suppress the first harmonic, $|\hat m_1| \approx 0$, without being told. This is a clean check of the mechanism. The fully supervised reference is 99.2% image-wise accuracy, reported by Real-Time, Highly Accurate Robotic Grasp Detection using Fully Convolutional Neural Network with Rotation Ensemble Module.
The decisive checks are ordered by how much they would change our belief. First, harmonic distillation must beat the normalized point-pseudo-label mean teacher on the ambiguous subsets. If it does not, carrying uncertainty does not help and the idea fails. Second, it must match or beat binned distillation at matched tuning. Third, it must collapse in fewer runs of the loss-weight sweep than temporal continuity. Fourth, $K=2$ must clearly outperform $K=1$ on PASCAL3D+ ambiguous categories, while $K>4$ adds little. That pattern is the signature that the useful soft knowledge in orientation is concentrated in the low harmonics.
References
- Improved Generalization of Heading Direction Estimation for Aerial Filming Using Semi-Supervised Regression
- Learning quadrotor dynamics using neural network for flight control
- Semi-Supervised 3D Hand-Object Pose Estimation Via Pose Dictionary Learning
- 2.5D Image-based Robotic Grasping
- GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
- Noisy Self-Knowledge Distillation for Text Summarization
- DINO as a von Mises-Fisher mixture model
- Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
- 3D Bounding Box Estimation Using Deep Learning and Geometry
- Monocular Object Orientation Estimation using Riemannian Regression and Classification Networks
- Real-Time, Highly Accurate Robotic Grasp Detection using Fully Convolutional Neural Network with Rotation Ensemble Module