Standard cameras capture the world as a projection of colors, but robots and autonomous systems often rely on depth sensors to understand the three-dimensional structure of their environment. While raw depth data—typically a single-channel image where each pixel represents a distance—contains the necessary information, it is notoriously difficult for standard neural networks to process effectively.
Researchers across the field have converged on a powerful solution: transforming raw depth into a three-channel geocentric encoding, often referred to as HHA (Horizontal disparity, Height above ground, and Angle with gravity). This representation bridges the gap between raw distance measurements and the geometric reasoning required for complex tasks like object detection and semantic segmentation.
The Problem with Raw Distance
In a standard RGB image, a pixel's value is largely independent of the camera's height or tilt. In contrast, a raw depth map is highly sensitive to the camera's viewpoint. As Anran Wang and colleagues argue in MMSS: Multi-modal Sharable and Specific Feature Learning for RGB-D Object Recognition, raw distance values provide insufficient geometric context.
If a camera moves closer to a table, every pixel value in a raw depth map changes, even though the table's physical properties remain the same. Furthermore, Saining Xie and Zhuowen Tu noted in their work on Holistically-Nested Edge Detection that raw depth channels often underperform because they fail to explicitly represent the spatial relationships that define objects.
The HHA Solution: Encoding Geometry
To overcome these limitations, researchers like Siddhartha Chandra and Iasonas Kokkinos popularized a three-channel encoding that reinterprets depth through a geocentric lens. As detailed in Accurate Human-Limb Segmentation in RGB-D Images for Intelligent Mobility Assistance Robots, the HHA representation consists of:
- Horizontal Disparity: A value related to the inverse of depth, which helps the network focus on nearby objects where detail is highest.
- Height above Ground: The vertical distance of each point from the floor plane.
- Angle with Gravity: The angle of the local surface normal relative to the direction of gravity.
By transforming depth into these three channels, the data becomes "image-like" in structure but "geometric" in content. Rongrong Ji and Wei Liu found that this strategy enables the extraction of high-level features specifically tuned for spatial reasoning in 3D object detection.
Why Geocentric Encoding Works
The intuition behind HHA's success lies in two main areas: geometric invariance and transfer learning.
Geometric Invariance
In a raw depth map, a floor and a wall might have the same distance value. In HHA, they are immediately distinguishable: the floor has a "height above ground" of zero and an "angle with gravity" of zero, while a wall has increasing height and an angle of 90 degrees. Max Schwarz and Sven Behnke demonstrated in RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter that this representation is crucial for robots to distinguish between objects and the supporting environment, such as shelves or floors.
Enabling Transfer Learning
Most state-of-the-art computer vision models are pre-trained on massive datasets of color images (like ImageNet). These models expect three-channel inputs (RGB). By encoding depth into three channels (HHA), researchers can use these pre-trained weights directly on geometric data. Songyao Jiang and colleagues utilized this in Skeleton Based Sign Language Recognition, allowing their model to leverage visual patterns learned from millions of natural images to understand 3D body poses.
Applications Across Domains
The versatility of geocentric encoding is evident in its wide range of applications:
- Semantic Segmentation: Seungyong Lee used HHA in RDFNet to capture complementary features across multiple scales, significantly improving object recognition in cluttered indoor scenes. Similarly, Di Lin used it in the Cascaded Feature Network to help the network understand how visual characteristics change with distance.
- Object Affordances: Anh Nguyen and Darwin G. Caldwell applied HHA to detect object affordances, helping robots identify "graspable" regions by understanding the physical orientation of surfaces.
- Salient Object Detection: Hao Chen and Youfu Li showed that geocentric representations make models more robust to visual ambiguities, as seen in their cross-modal distillation framework.
By moving from "how far is this pixel?" to "how high is this point and how is it oriented?", geocentric encoding provides neural networks with the structural priors they need to perceive the world as a physical space rather than just a grid of numbers.