Geocentric Depth Encoding for Convolutional Networks

23 concepts21 papers1 proposal

When deep learning revolutionized 2D computer vision, it did so by exploiting the spatial structure of RGB images. Convolutional Neural Networks (CNNs) excel at detecting local patterns like edges, textures, and color gradients, which remain relatively stable across natural images. However, when consumer-grade sensors introduced depth data (RGB-D), researchers faced a fundamental representation problem.

A raw depth map is typically a single-channel image where each pixel represents the distance from the camera sensor to the nearest object surface. Simply treating this raw depth channel as a grayscale image or appending it as a fourth channel to an RGB input yields surprisingly poor results. As noted by Saining Xie et al. in Holistically-Nested Edge Detection, raw depth channels consistently underperform compared to geometric encodings that explicitly represent spatial relationships.

Why does raw depth fail? The issue lies in the difference between egocentric and geocentric coordinate systems. A raw depth map is strictly egocentric—it depends entirely on the camera's position and orientation. If a camera tilts downward, every single pixel value in the depth map changes, even though the physical structure of the room remains identical. Furthermore, standard CNN architectures are heavily optimized for three-channel inputs and rely on weights pre-trained on massive RGB datasets like ImageNet. As Deng-Ping Fan et al. point out in JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection, simple grayscale mapping fails to capture the complex geometric features and surface variations necessary for a shared backbone network to reason effectively.

The Anatomy of HHA Encoding

To bridge this gap, a landmark insight emerged from the computer vision community: instead of forcing a network to learn the laws of physics from raw distances, we can re-encode the single-channel depth map into a three-channel, world-anchored representation known as HHA (Horizontal disparity, Height above ground, and Angle with gravity).

This approach, pioneered by Pablo Arbelaez et al. in Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images, begins by estimating the gravity vector from the local surface normals within the depth map. This establishes a stable, geocentric coordinate frame. The raw depth map is then unpacked into three distinct channels:

  1. Horizontal Disparity: This represents the pixel-level parallax (inversely proportional to distance), capturing fine-grained local depth variations and boundaries.
  2. Height Above Ground: This assigns each pixel an absolute physical height relative to the estimated floor plane.
  3. Angle with Gravity: This computes the orientation of the local surface normal relative to the direction of gravity.

By transforming the data this way, the representation shifts from "how far is this pixel from the lens?" to "where is this point located in the physical world, and how is it oriented?"

Why Geocentric Encoding Empowers Neural Networks

The HHA representation works exceptionally well because it aligns the underlying physics of a 3D scene with the architectural biases of 2D convolutional networks.

First, it solves the semantic invariance problem. In an indoor scene, functional categories are intrinsically tied to gravity and height. A floor is always at height zero and perpendicular to gravity; a table surface sits at a predictable height parallel to the floor; walls and doors are strictly vertical. By explicitly encoding these properties, the network can immediately distinguish between a floor and a wall based on their orientation channel, even if they share identical color and texture. As Max Schwarz demonstrates in RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter, this structural context allows robotic systems to easily separate target objects from their supporting environments, such as shelves or tables.

Second, HHA makes geometric data look like a natural image to a CNN. The three channels exhibit continuous gradients, sharp edges, and regional textures that mimic the statistical properties of RGB channels. This allows models to exploit transfer learning. Instead of training a depth network from scratch on limited 3D data, researchers can feed the HHA image into a standard network pre-trained on ImageNet. Seungyong Lee et al. leverage this property in RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation, showing that geocentric representations emphasize geometric properties that are easily processed by standard convolutional layers.

Widespread Adoption Across Spatial Tasks

Because HHA successfully repurposes 2D architectures for 3D reasoning, it became a foundational technique across a wide variety of computer vision domains.

In object detection and proposal generation, Xiaozhi Chen et al. utilize HHA in 3D Object Proposals using Stereo Imagery for Accurate Object Class Detection to allow standard CNNs to process structural cues from stereo cameras. Similarly, Hongyuan Zhu et al. show in Discriminative Multi-modal Feature Fusion for RGBD Indoor Scene Recognition that this alignment with architectural biases facilitates highly effective multi-modal feature extraction without requiring explicit image segmentation.

The representation has also proven vital for fine-grained parsing and human action analysis. For instance, Siddhartha Chandra et al. apply HHA to real-time human limb segmentation in Accurate Human-Limb Segmentation in RGB-D Images for Intelligent Mobility Assistance Robots, while Gaia Caligiore et al. adopt it in Multisource Approaches to Italian Sign Language (LIS) Recognition: Insights from the MultiMedaLIS Dataset to preserve 3D spatial dynamics within standard video networks.

Even when depth sensors are absent at test time, geocentric encodings serve as powerful structural priors during training. In STDG: Semi-Teacher-Student Training Paradigram for Depth-guided One-stage Scene Graph Generation, Xukun Zhou uses an HHA-trained teacher network to guide a monocular RGB student network. Because HHA explicitly encodes the physical layout and ground plane, it provides a structured topological prior that helps the student model understand complex spatial relationships between objects without requiring depth data during inference.

By transforming raw, volatile distance measurements into a stable, world-anchored coordinate frame, geocentric encoding turns a difficult 3D reasoning problem into a familiar 2D pattern recognition task, unlocking the full potential of deep convolutional networks for spatial understanding.

Key Foundational Work

Accurate Human-Limb Segmentation in RGB-D Images for Intelligent Mobility Assistance Robots (2015) — contributed 1 concept elements to this direction.
Towards 3D object detection with bimodal deep Boltzmann machines over RGBD imagery (2015) — contributed 1 concept elements to this direction.
Holistically-Nested Edge Detection (2015) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 20 papers
  2. Version 1 · 2026-10-04 · 23 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hha_representation_75e8f13c,
  title = {HHA representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hha-representation/rdr_75e8f13c0ac747dc804cfdeb1cd5043a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_hha_representation_75e8f13c_v2,
  title = {HHA representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hha-representation/rdr_75e8f13c0ac747dc804cfdeb1cd5043a/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake