TRB: A Novel Triplet Representation for Understanding 2D Human Body

Haodong Duan, Kwan-Yee Lin, Sheng Jin, Wentao Liu, Chen Qian, Wanli Ouyang

Introduction

In the field of computer vision, representing the human body typically falls into two categories: sparse skeletons or dense pixel-wise masks. Skeletons, consisting of a few dozen keypoints, are efficient for tracking movement but fail to capture body shape or volume. Conversely, pixel-wise parsing provides rich shape information but is computationally expensive to label and difficult to manipulate for graphics applications.

This paper introduces the Triplet Representation for Body (TRB), a compact 2D representation that bridges this gap. By combining skeleton keypoints with contour keypoints, TRB offers a flexible, easy-to-use framework that is rich enough to describe the human form while maintaining the simplicity of point-based systems.

The TRB representation applied to various poses, showing the skeleton keypoints (red) and corresponding medial/lateral contour points (blue/green).

The TRB Triplet and TRB-Net

The core innovation is the TRB Triplet. For every major joint (skeleton keypoint), the authors define two corresponding contour keypoints: one on the medial (inner) side and one on the lateral (outer) side of the limb. This creates a geometric constraint that links pose and shape.

To predict these triplets, the authors developed TRB-Net, which incorporates three primary technical innovations:

  1. X-structure (Xs): A multi-level message-passing scheme that facilitates communication between the "skeleton branch" and the "contour branch" of the network throughout the learning process.
  2. Directional Convolution (DC): Standard convolutions treat all spatial directions equally. DC updates pixels in a specific order (inside-out or outside-in) to explicitly model the spatial relationship between the central bone and the outer skin.
  3. Pairwise Mapping (PM): This module uses learned feature flows to warp heatmaps between neighboring points, ensuring that predicted joints and contours are geometrically consistent and preventing "broken" limb estimations.
Architecture of TRB-Net featuring (a) the X-structure for branch fusion, (b) Directional Convolution components (Scattering and Gathering), and (c) the Warping/Refine unit for pairwise mapping.

Datasets and Evaluation

Because standard datasets did not include contour triplets, the authors augmented three popular benchmarks:

  • MPII_trb: ~40k instances based on MPII Human Pose.
  • LSP_trb: ~12k instances based on Leisure Sports Pose.
  • COCO_trb: ~75k instances annotated from the COCO train-val set.

Key Insights and Results

The research yielded several significant findings:

  • Symbiotic Learning: Training a model to identify contours actually improves its ability to locate joints. The limb boundary provides a strong geometric constraint that aids in precise joint localization.
  • Accuracy Gains: TRB-Net reduced estimation error by 13.3% for skeletons and 15.1% for contours compared to standard multi-task baselines. On the LSP dataset, it achieved a state-of-the-art 94.5% [email protected].
  • Intuitive Shape Editing: Because TRB is point-based, it can serve as a "control rig" for Generative Adversarial Networks (GANs). The authors demonstrated that by simply dragging contour points, they could make a person in a photo appear more muscular or slimmer while preserving the original pose and background.

Limitations and Future Work

While TRB captures shape better than traditional skeletons, it remains a 2D representation and can struggle with complex self-occlusions where limbs are hidden. Additionally, manual annotation of medial and lateral points can be subjective in cases of baggy clothing. Future research may extend TRB into 3D to bridge the gap between simple 3D skeletons and complex 3D human meshes like SMPL.

Summary updated

Abstract

Human pose and shape are two important components of 2D human body. However, how to efficiently represent both of them in images is still an open question. In this paper, we propose the Triplet Representation for Body (TRB) -- a compact 2D human body representation, with skeleton keypoints capturing human pose information and contour keypoints containing human shape information. TRB not only preserves the flexibility of skeleton keypoint representation, but also contains rich pose and human shape information. Therefore, it promises broader application areas, such as human shape editing and conditional image generation. We further introduce the challenging problem of TRB estimation, where joint learning of human pose and shape is required. We construct several large-scale TRB estimation datasets, based on popular 2D pose datasets: LSP, MPII, COCO. To effectively solve TRB estimation, we propose a two-branch network (TRB-net) with three novel techniques, namely X-structure (Xs), Directional Convolution (DC) and Pairwise Mapping (PM), to enforce multi-level message passing for joint feature learning. We evaluate our proposed TRB-net and several leading approaches on our proposed TRB datasets, and demonstrate the superiority of our method through extensive evaluations.
BibTeX
@inproceedings{duan2019trb,
  author = {Haodong Duan and Kwan-Yee Lin and Sheng Jin and Wentao Liu and Chen Qian and Wanli Ouyang},
  title = {{TRB: A Novel Triplet Representation for Understanding 2D Human Body}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00957},
}
Figure