Introduction
In the field of computer vision, representing the human body typically falls into two categories: sparse skeletons or dense pixel-wise masks. Skeletons, consisting of a few dozen keypoints, are efficient for tracking movement but fail to capture body shape or volume. Conversely, pixel-wise parsing provides rich shape information but is computationally expensive to label and difficult to manipulate for graphics applications.
This paper introduces the Triplet Representation for Body (TRB), a compact 2D representation that bridges this gap. By combining skeleton keypoints with contour keypoints, TRB offers a flexible, easy-to-use framework that is rich enough to describe the human form while maintaining the simplicity of point-based systems.
The TRB Triplet and TRB-Net
The core innovation is the TRB Triplet. For every major joint (skeleton keypoint), the authors define two corresponding contour keypoints: one on the medial (inner) side and one on the lateral (outer) side of the limb. This creates a geometric constraint that links pose and shape.
To predict these triplets, the authors developed TRB-Net, which incorporates three primary technical innovations:
- X-structure (Xs): A multi-level message-passing scheme that facilitates communication between the "skeleton branch" and the "contour branch" of the network throughout the learning process.
- Directional Convolution (DC): Standard convolutions treat all spatial directions equally. DC updates pixels in a specific order (inside-out or outside-in) to explicitly model the spatial relationship between the central bone and the outer skin.
- Pairwise Mapping (PM): This module uses learned feature flows to warp heatmaps between neighboring points, ensuring that predicted joints and contours are geometrically consistent and preventing "broken" limb estimations.
Datasets and Evaluation
Because standard datasets did not include contour triplets, the authors augmented three popular benchmarks:
- MPII_trb: ~40k instances based on MPII Human Pose.
- LSP_trb: ~12k instances based on Leisure Sports Pose.
- COCO_trb: ~75k instances annotated from the COCO train-val set.
Key Insights and Results
The research yielded several significant findings:
- Symbiotic Learning: Training a model to identify contours actually improves its ability to locate joints. The limb boundary provides a strong geometric constraint that aids in precise joint localization.
- Accuracy Gains: TRB-Net reduced estimation error by 13.3% for skeletons and 15.1% for contours compared to standard multi-task baselines. On the LSP dataset, it achieved a state-of-the-art 94.5% [email protected].
- Intuitive Shape Editing: Because TRB is point-based, it can serve as a "control rig" for Generative Adversarial Networks (GANs). The authors demonstrated that by simply dragging contour points, they could make a person in a photo appear more muscular or slimmer while preserving the original pose and background.
Limitations and Future Work
While TRB captures shape better than traditional skeletons, it remains a 2D representation and can struggle with complex self-occlusions where limbs are hidden. Additionally, manual annotation of medial and lateral points can be subjective in cases of baggy clothing. Future research may extend TRB into 3D to bridge the gap between simple 3D skeletons and complex 3D human meshes like SMPL.
Summary updated