Gravity-Preserving Distillation for Panoramic Robot Affordances

A mobile robot with one 360° camera sees the whole room at once, with no blind spots and no need to turn to find the counter, the drawer or the person handing it something. Almost no robot manipulation data exists in that format, though. The standard fix for scarce panoramic data comes from depth estimation. You cut the panorama into ordinary perspective views, run a strong perspective model on each view, and teach a panoramic "student" model to reproduce the stitched answers. To hide the seams between views, the best-known recipe also spins the sphere randomly during training. For robots this has a cost that has gone unnoticed: a random spin removes the one direction that matters most for manipulation, which is "up". Grasping, lifting, pouring and pulling open a drawer are defined relative to gravity. A student that learns from arbitrarily tilted panoramas is being taught that "up" does not matter.

This note proposes to invert the recipe. The teacher's view decomposition gets rotated. The student's input stays upright. We then spend the teacher's limited resolution where interactions actually happen. Human 360° viewing and egocentric interaction concentrate near the horizon, a pattern called equator bias. We treat this bias as a statement about camera height rather than about human attention. We measure where hand–object contacts fall in gravity-aligned human video, transport that distribution geometrically to the robot's lower camera height, and use the transported distribution to decide where the teacher looks at high resolution. The testable claim is specific. Gravity-preserving distillation should match random-rotation distillation on depth, and it should clearly beat it on gravity-dependent affordance predictions: where to grasp, and where the hand should move after grasping.

Background and the gap

The equirectangular projection (ERP) is the standard way of storing a panorama as a flat image. Longitude maps to the horizontal axis and latitude to the vertical axis. It stretches content more and more toward the top and bottom edges (the poles), which breaks the assumption that a feature looks the same wherever it appears in the image. The consequences are well documented. 360-Indoor shows that perspective detectors degrade on ERP. It also finds that COCO-pretrained detectors still beat spherical networks trained from scratch, because spherical data is too scarce. OmniDepth shows an unadapted pinhole depth model failing badly on panoramas and fixes this with filters whose shape changes with latitude. m-p YOLO and P2PDA handle distortion by splitting the panorama into sub-views or by domain adaptation. Together these papers motivate the proposal: perspective pretraining is valuable, and ERP distortion has to be handled explicitly.

The method we build on is Depth Anywhere. It distills Depth Anything, applied to the six faces of a cube wrapped around the sphere, into panoramic students (UniFuse, BiFuse++), and it applies random 3D sphere rotations before splitting the panorama into cube faces so that no seam stays in a fixed place. We reuse its distillation idea but reject its rotation of the student's input. We are not the only ones to notice this conflict. The research file's own gap analysis observes that random rotation "scrambles the natural latitude coordinate". Meanwhile, LAU-Net, MTAOIQA and Saliency in VR all obtain gains precisely by keeping latitude meaningful. Saliency in VR fits a Laplace-shaped latitude prior to human gaze. LAU-Net spends more network depth on equatorial bands. MTAOIQA samples viewports along the equator. Spherical World-Locking uses the head-mounted sensor's motion signal to place egocentric video in a gravity-aligned frame, where human interactions cluster near the horizontal plane.

These papers also constrain the design. The equator-bias literature itself warns that a rigid horizon prior suppresses valid targets well above or below eye level. MTAOIQA's authors concede that equatorial sampling can miss content at the poles. So the prior must never act as a hard mask on what the robot can see. It may only decide where the teacher looks at high resolution during training.

On the robot side, SWIM learns robot-agnostic affordances from human video: a 2D grasp point and a post-grasp waypoint. MotoVLA pretrains on 3D motion tracks from unlabeled human and robot video. RoboCasa scales household simulation. All three work only with narrow-field pinhole cameras. No paper in the file forms affordances over a panorama, and none resolves the conflict between rotation and gravity.

Core Claim

Our first claim is that gravity-dependent outputs cannot be recovered from a student trained on rotated inputs. A post-grasp waypoint says which way the hand moves once it holds an object: up out of a sink, or outward from a drawer. Once gravity is scrambled, the student has no reliable cue for this direction. So we keep every student input gravity-aligned, and we move the seams in the teacher instead. For each unlabeled panorama we sample several rotations $R$. For each rotation we rotate the sphere, split it into perspective views, run the teacher, rotate the per-pixel predictions back into the upright frame, and fuse them with weights that down-weight pixels near view borders. Seams land in a different place for each rotation, so fusion averages them out. The student never sees a tilted world. Its only augmentation is yaw, which in ERP is simply a horizontal circular shift and introduces no seam. We also add small pitch and roll jitter, matched to the error of the robot's orientation sensor.

Our second claim is that equator bias, once expressed in metres rather than degrees, tells us where to spend teacher resolution for a robot. Consider a contact point at height $h$ above the floor and horizontal distance $d$ from a camera at height $c$. It appears at latitude

$$\phi = \arctan\!\left(\frac{h - c}{d}\right).$$

Human interaction data fixes the joint distribution of $(h, d)$. Its latitude distribution at eye height $c_{\text{human}} \approx 1.6$ m is the familiar peaked band slightly below the horizon. Re-evaluating the same $(h,d)$ samples at the robot's camera height $c_{\text{robot}}$ yields a transported prior $p_{\text{robot}}(\phi)$. For a camera near counter height, this prior moves the interaction band up toward the equator and widens it, because closer surfaces sweep a larger range of angles.

We use $p_{\text{robot}}(\phi)$ in exactly two places. First, it sets the density of teacher views at each latitude. More, narrower perspective views go where $p_{\text{robot}}$ is high, which gives the teacher higher effective resolution on small handles and knobs. The total number of teacher calls per panorama stays fixed. Second, it weights the distillation loss:

$$\mathcal{L} = \sum_{\text{pixels } i} \big(\lambda + p_{\text{robot}}(\phi_i)\big)\,\cos\phi_i\,\ell\big(f_{\text{student}}(x)_i,\ \hat{y}_i\big).$$

Here $\hat{y}_i$ is the fused teacher pseudo-label at pixel $i$. The factor $\cos\phi_i$ corrects for ERP oversampling near the poles, because sphere area shrinks as $\cos\phi$. The term $\ell$ is the per-task loss. The floor $\lambda>0$ keeps polar regions, such as overhead cabinets and ceiling–wall edges, supervised. The prior is not applied at inference.

Research Program

The first step measures the human interaction prior. We take Aria Everyday Activities, which Spherical World-Locking already uses and which comes with the motion-sensor data needed for gravity alignment. We gravity-align the frames with the world-locking procedure, detect hand–object contacts, and lift each contact to 3D with a monocular depth teacher. This gives an empirical $(h,d)$ distribution. We then check whether its latitude marginal at eye height reproduces the Laplace-shaped profile reported by Saliency in VR. If it does not, the height-transport story loses its empirical anchor, and we will report that.

The second step builds the teachers and the pseudo-labels. The depth teacher is Depth Anything, the same teacher Depth Anywhere uses. The affordance teacher is a SWIM-style model trained on Epic-Kitchens and 100 Days of Hands that outputs grasp-point and post-grasp-waypoint heatmaps on perspective images. Unlabeled panoramas come from Matterport3D, Stanford2D3D and the 3,335 indoor images of 360-Indoor. We add a smaller set captured with a consumer 360° camera mounted at the camera height of a Hello Stretch robot in real kitchens.

The third step trains one student architecture under several conditions. The student is a UniFuse-style panoramic encoder that receives an explicit latitude channel, with two heads: depth, and the two affordance heatmaps. We train it once for each condition described under Baselines.

The fourth step deploys the student on a mobile manipulator. The panoramic grasp point and waypoint, back-projected with the student's depth, become 3D targets for a scripted grasp-and-move controller on a Stretch robot. This mirrors SWIM's affordance-to-action interface, but we do not use SWIM's world-model fine-tuning, so the comparison isolates perception. In simulation we attach a rendered panoramic camera to the RoboCasa robot. RoboCasa's rendering pipeline should support this, but that is an engineering assumption we have not verified. We use RoboCasa's atomic kitchen tasks and vary the camera height, which lets us test height transport directly.

Baselines

The naive reference point runs the perspective teachers directly on the full ERP image with no decomposition, and separately fine-tunes the same student on ERP with no distillation, using the small labeled set only. Prior results predict this reference will be poor: OmniDepth reports an Abs Rel of 0.3181 for an unadapted pinhole model against 0.0702 for RectNet on 360D. It anchors the bottom of every table.

The strongest existing method is Depth Anywhere's recipe itself: random 3D rotation, six-face cubemap distillation, and a 1:1 mix of ground truth and pseudo-labels, which its authors found optimal. We extend this recipe to the affordance heads unchanged. For depth, the numbers to match are those reported in Depth Anywhere: Abs Rel 0.082 for BiFuse++ in zero-shot transfer to Stanford2D3D, and 0.089 for UniFuse supervised with distillation on Matterport3D. We will rerun the recipe with our exact unlabeled pool, the same student, the same number of teacher calls per panorama, the same number of training steps and the same hyperparameter-search budget. That way any gap reflects the recipe and not scale.

Beating it means three things. First, depth must stay within noise of these numbers, a tie being the prediction. Second, grasp-point localization error and post-grasp waypoint direction error must be clearly lower, especially for vertical motions. Third, real-robot and simulated task success must be higher. For affordance accuracy on panoramas and for panoramic mobile-manipulation success, no reported result was found on Lacuna, so our reproduction of the Depth Anywhere recipe serves as the reference.

For object localization near interaction zones, we report per-latitude-band mAP on 360-Indoor. The context from the file is 33.6% mAP for FPN with a ResNet-101 backbone reported in 360-Indoor, and 34.29 mAP for m-p YOLO against 30.90 for plain YOLOv2 in Object Detection in Equirectangular Panorama.

The ablations isolate each ingredient. One variant keeps gravity-preserving pseudo-labels but uses uniform teacher-view density and no prior weighting, which isolates the upright-student effect. A second variant uses the untransported human-eye-height prior instead of $p_{\text{robot}}$, which isolates height transport. A third variant multiplies the student's inference output by the prior, as Saliency in VR does, which tests the hard-mask failure mode. A fourth variant removes the latitude input channel.

Why not just…?

Why not just keep Depth Anywhere's random rotations and add an "up" direction as an input? This is the most serious threat. If the student receives the rotation as an extra input, it could in principle un-rotate internally. We include this variant directly: a random-rotation student conditioned on the rotation matrix. If it matches our method on waypoint direction error and robot success, then the contribution reduces to "tell the network where gravity is". We would say so plainly. We expect it to lag in low-data regimes, because the student must learn the un-rotation from scarce labels. 360-Indoor's finding that spherical models starve without pretraining data suggests that inductive biases which cost nothing in data are worth the most here.

Why not just mount three or four pinhole cameras and run perspective models directly? A pinhole rig avoids distortion but has the seams built into the hardware. It also brings the calibration and self-occlusion problems that OmniDepth notes, and it loses the joint view of floor, walls and ceiling that OmniDepth credits for its depth gains. We include a simulated four-camera rig in RoboCasa with equal total pixel count, running the perspective teachers directly and fusing their outputs in 3D. If this rig matches the panoramic student on task success, the case for panoramic sensing on this task class weakens, and we will report it.

Why not just crop to an equatorial belt? This is cheap and matches equator bias. But 360-Indoor catalogues manipulation-relevant objects at latitudes from ±60° to ±90°, and our own height-transport argument predicts that the interaction band moves with camera height. A fixed belt is therefore wrong for at least one of the robot's own configurations. The belt-cropped variant appears in the per-band evaluation, and we expect it to fail on overhead-cabinet and low-drawer tasks.

Why not just fine-tune a large backbone on ERP and let it learn distortion? The file's evidence shows that unadapted transfer is weak. P2PDA starts from 28.50% mIoU before adaptation on DensePASS, and YOLOv2 on ERP trails m-p YOLO. Our naive baseline measures this directly with the same data budget.

Decisive checks

Three results decide whether the idea holds. First, relative to random-rotation distillation, the upright student should show a large gain on post-grasp waypoint direction and little or no gain on depth. That asymmetry is the signature of gravity mattering for affordances and not for geometry. If both improve equally, the effect is generic regularization. If neither improves, the rotation–gravity conflict does not matter in practice. Second, when we sweep the rendered camera height in RoboCasa, the teacher-view allocation that works best should follow the latitude shift predicted by the transport equation. If the human-height prior works just as well at every height, height transport adds nothing. Third, on the real Stretch robot, task success from panoramic affordances should exceed the random-rotation student under identical controllers and trial counts, and the gap should be largest on tasks with vertical motion. Each outcome is informative on its own, and a negative result on the first check would be a useful correction to how panoramic distillation is currently practised.