Reconstructing a clothed person in 3D from one photo needs two things that have so far pulled against each other. The first is long-range context. How a shirt folds across the back depends on where the far arm is, and a network that only sees small image patches cannot know that. The second is robustness to unusual poses. Networks that pool context over the whole image tend to memorize where body parts usually sit in the picture, and they fall apart on poses they have not seen. We propose to keep long-range context and drop the pose-specific memorization. The idea is to compute the long-range context on a grid laid out in the body's own resting pose, not on the image grid. Image features are pulled back onto a front-and-back "canonical sheet" of a fitted body model in a standard T-pose, mixed globally there with cheap row-and-column attention, and then read back out at 3D query points. Two image pixels at distant positions then interact because they lie on related parts of the body, not because they happen to share an image row.
The claim can be tested directly. If image-space layout is the reason global encoders break on unusual poses, then the same attention operator should keep its gains on standard poses and lose much less accuracy on non-standard poses once it runs in canonical body coordinates. The decisive experiment swaps only the coordinate system in which an identical attention block runs.
The tension this resolves
Two lines of work in clothed-human reconstruction disagree about receptive fields. ICON predicts an implicit surface (a function that says, for any 3D point, whether it lies inside the person) using only local cues: distance to a fitted SMPL body (a standard parametric body mesh), the body normal, and predicted clothing normals at the projected pixel. ICON reports that global image encoders such as PIFuHD produce "melted" or detached limbs on non-fashion poses, and that local anchoring roughly halves their Chamfer error on CAPE-NFP. This finding motivates our worry about global context. On the other side, GTA argues that locality cannot model cloth tension set by distant limbs. It replaces the CNN with a Vision Transformer and predicts tri-plane features, and it is the first method below 0.8 cm Chamfer on CAPE-FP, at the cost of about two days of training on an RTX 3090 and a strong dependence on the SMPL fit. SIFU and HiLo add long-range reasoning in other ways. SIFU cross-attends to image features using SMPL-X normals as queries. HiLo runs attention over a low-resolution voxel grid of the body prior in posed 3D space. Both methods supply the body-conditioning techniques we reuse.
None of these methods asks in which coordinate system the long-range interaction should happen. GTA and SIFU mix information in image space or in planes tied to the camera. HiLo mixes in posed 3D space. In all three, which locations interact with which changes whenever the pose changes, so the network must learn a separate interaction pattern for each pose layout it sees. MeshMamba is the closest precedent for using canonical body coordinates. It orders mesh vertices by body-part UV charts and canonical template positions so that a state-space model can scan them. But it uses that ordering as the output representation, which restricts it to tight clothing. We use the canonical layout only to route context. Geometry is still predicted as a topology-free implicit field at arbitrary 3D points, so skirts and coats are not forced onto the body's connectivity, which is the failure mode that Point-Based Modeling of Human Clothing and Moulding Humans document for template-bound methods.
Core Claim
Long-range context for non-rigid clothing reconstruction becomes robust to pose when the interaction pattern is fixed in canonical body coordinates. A cheap factorized operator is then enough to provide that context. Concretely, we predict that a canonical-sheet attention block added to an ICON-style local model will match or approach GTA-level accuracy on fashion poses. We also predict it will keep the gap between fashion and non-fashion poses close to ICON's, whereas the identical block run in image space will show the larger gap that ICON attributes to global encoders.
Research Program
The system starts from an ICON-style pipeline. A SMPL-X body is fitted to the image, front and back clothed-normal maps are predicted, and an implicit function is queried at 3D points using local features. We add one component, the canonical context branch.
To build the sheet, we render the fitted body's canonical T-pose orthographically from the front and the back. This gives two $H_c \times W_c$ grids (for example $128 \times 128$), where each cell stores a body-surface point. Through the fitted pose, each cell maps to a posed 3D point and then to an image pixel. We bilinearly sample the image encoder's features there, with a visibility flag for cells that are occluded in the photo. Clothing pixels that fall outside the body's projection, such as a flared skirt hem, would otherwise be lost. Each such pixel is splatted to the canonical cell of its nearest body point in 2D, together with a learned embedding of its 2D offset. The sheet therefore carries garment evidence beyond the skin.
To mix globally, we apply recurrent criss-cross attention from CCNet on the concatenated front and back sheets. Each cell attends to its own row and column, and two passes connect every cell to every other at roughly $O(N\sqrt{N})$ cost instead of $O(N^2)$. On a T-pose sheet, rows run across shoulders and arms and columns run down the torso and legs. These are the directions along which cloth tension and drape propagate, so the factorization becomes a meaningful anatomical prior instead of an arbitrary image axis. To let front and back exchange information, we add a third attention path between front cell $(i,j)$ and its mirrored back cell. This path is how the occluded back side gets context, which is the role GTA's dense cross-attention plays.
To read out, a 3D query point $x$ takes its $k$ nearest posed-body surface points. It looks up their canonical cells and forms a context vector as a distance-weighted average,
$$c(x) = \sum_{m=1}^{k} w_m(x)\, S\big(u_m(x)\big),$$
where $S$ is the mixed canonical sheet, $u_m(x)$ is the canonical cell of the $m$-th nearest body point, and $w_m(x)$ are softmax weights over negative distances. With $k>1$, a point between the legs on a dress draws from both legs instead of snapping to one. The context vector $c(x)$ is concatenated with ICON's local features before the implicit decoder.
The design inherits one risk: everything rests on the SMPL-X fit. If the fit is wrong, features land on the wrong canonical cells. We keep ICON's loop that alternates normal prediction and body refinement, and we treat fit noise as a first-class evaluation axis rather than a footnote.
Baselines
The naive reference point is ICON itself, trained by us on the same THuman2.0 training split with the same normal predictor and SMPL-X fitter. It defines what local-only features achieve and, more importantly, how much accuracy is lost between CAPE-FP and CAPE-NFP. Beating it means lower Chamfer and normal error on CAPE-FP with no larger relative degradation on CAPE-NFP. For context, ECON reports 0.926 cm Chamfer on CAPE's challenging poses against 0.971 cm for ICON. ECON is a second reference built on explicit 2.5D normal integration, and we run it from its released form as a non-attention competitor.
The strongest existing methods are GTA and SIFU. GTA reports Chamfer below 0.8 cm on CAPE-FP. SIFU reports 0.6 cm Chamfer on THuman2.0, compared with 0.73 cm for GTA and 0.95 cm for ICON, which makes SIFU the strongest reported number in the file on THuman2.0. We found no reported result on Lacuna for GTA or SIFU broken out on CAPE-NFP, so that comparison must come from our own reruns. For fairness, all methods use the same training scans and the same SMPL-X initializations at test time. They also share a matched training budget, which we fix at GTA's reported two RTX 3090 days, with an equal hyperparameter search of a few runs each. Beating SIFU means matching its THuman2.0 Chamfer within noise while showing a clearly smaller fashion-to-non-fashion gap. A pure accuracy win is not required, and we say so in advance.
HiLo is included as the posed-space attention competitor. It reports about a 9.54% Chamfer improvement over ICON and ECON on CAPE. We expect our method to at least tie it on accuracy, and to be ahead under perturbed fits and in memory use, because it avoids HiLo's voxel grid.
The ablation that isolates the key ingredient is the image-space twin. It is the identical criss-cross block with the same channels, the same two recurrent passes and the same parameter count, but run on the image feature map instead of the canonical sheet, and read out at the query point's projected pixel. A second ablation keeps the canonical sheet but replaces attention with a convolution of the same size. This separates "canonical coordinates" from "long-range mixing." A third sets $k=1$ in the readout to measure what soft multi-point retrieval buys on loose garments.
Why not just…?
Why not just train a global image encoder with heavy pose augmentation? This is the most serious threat. If GTA's fragility comes from too few pose layouts in training, augmentation could close the gap without changing coordinates. Augmentation can vary in-plane rotation, scale and cropping, but it cannot create new articulations from a fixed set of scans. Even so, we will train the image-space twin and a GTA-style encoder with aggressive in-plane rotation and crop augmentation. If either one matches our CAPE-NFP gap, the canonical-coordinate claim is wrong and we will report that.
Why not just widen the receptive field of the CNN with dilation or oversized kernels? Dilated stacks dilute distant dependencies and blur boundaries, as Guanbin Li et al. discuss. Oversized kernels such as those in ParCNetV2 do give a global field in one layer. However, they are explicitly position-dependent in image coordinates, which is exactly the property we suspect causes pose overfitting. The image-space twin ablation covers this family. A ParCNetV2-style large-kernel variant on the image map will be included as a further check.
Why not just use HiLo-style attention over the body in posed 3D space? Posed space is body-aware, but its layout still changes with pose: a raised arm moves its voxels. It also pays for a 3D grid, which forces low resolution. If HiLo's degradation under perturbed SMPL-X fits and non-fashion poses matches ours, the canonical sheet adds nothing beyond body conditioning. The pose-gap and fit-noise experiments compare the two directly.
Why not just serialize body vertices and use a state-space model as MeshMamba does? MeshMamba's ordering is tied to the output mesh, and its authors restrict it to tight clothing. Scanning a flattened grid also introduces the directional bias that EccoMamba works to remove. Criss-cross attention has no scan order. Still, a Mamba scan over our canonical sheet is a reasonable drop-in, and we will run it as a variant of the mixer, not as a competing hypothesis.
Proposed Experiments
All models train on THuman2.0. Evaluation uses CAPE-FP and CAPE-NFP for Chamfer distance and normal error, THuman2.0 test scans for comparison with SIFU's and GTA's reported numbers, and Renderpeople for loose garments, reporting normal error against ECON's 0.0478.
The first decisive check is the pose gap. For each method we report the ratio of CAPE-NFP to CAPE-FP Chamfer. The hypothesis predicts that the canonical model sits near ICON's ratio and the image-space twin sits near GTA's, with similar CAPE-FP accuracy for the two attention variants.
The second check is in-plane rotation. Test images rotated by 30, 60 and 90 degrees leave the canonical sheet essentially unchanged once the body is refit, but they reshuffle image rows and columns. Methods that rely on image layout should degrade steeply, while ours should track the degradation of the SMPL-X fitter alone.
The third check is fit noise. We perturb SMPL-X pose parameters around the fitted values by increasing amounts, with and without ICON's refinement loop, and plot Chamfer against perturbation size for ours, ICON, SIFU and HiLo. This is where the method could fail, because pulling features through a wrong fit misplaces them. A steeper curve than ICON's would mean the canonical branch trades layout fragility for fit fragility. That would be a negative result worth publishing.
The fourth check is loose clothing. On the Renderpeople subjects wearing dresses and coats, we compare $k=1$ with $k>1$ readout and the off-body splatting on versus off. This tests whether body-anchored context still helps where the garment leaves the body, the regime where MeshMamba stops.
Finally, we report training time, peak memory and parameters against GTA's two-day budget, to confirm that the factorized canonical mixing is cheap enough to matter.
References
- ICON: Implicit Clothed humans Obtained from Normals
- Global-correlated 3D-decoupling Transformer for Clothed Avatar Reconstruction
- SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction
- HiLo: Detailed and Robust 3D Clothed Human Reconstruction with High-and Low-Frequency Information of Parametric Models
- MeshMamba: State Space Models for Articulated 3D Mesh Generation and Reconstruction
- Point-Based Modeling of Human Clothing
- Moulding Humans: Non-Parametric 3D Human Shape Estimation From Single Images
- CCNet: Criss-Cross Attention for Semantic Segmentation
- ECON: Explicit Clothed humans Optimized via Normal integration
- Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining
- ParCNetV2: Oversized Kernel with Enhanced Attention
- EccoMamba: Enhanced Cross-hierarchical Continuity Orthogonal Mamba for Medical Image Segmentation