Image generators still draw hands with six fingers, fused digits and impossible joint bends. The best current fixes, HandRefiner and RHanDS, fit a 3D hand model to the broken hand, render it as a depth image, and repaint the region under that guidance. They raise two problems. First, the repainting network uses whatever it sees in the broken image, so some of the original deformity can leak back into the output. Second, pushing the 3D guidance harder fixes the shape but turns skin into plastic. We argue these two problems are one problem: the accuracy-interpretability trade-off, seen inside a hand generator.
We propose a hand inpainter that sees the hand's structure only through a small set of human-readable variables, which we call concepts. Examples are how many fingers are present, how far each joint is bent (in coarse bins), and which finger is in front of which. It sees everything else (skin, lighting, art style) through a residual channel, a second input pathway that carries whatever the concepts leave out. During training we actively strip this residual of structural information. The claim is that a residual that provably cannot encode pose removes the texture-versus-structure tension. Structure is then controlled entirely by auditable concepts, while texture keeps full capacity. Editing a concept, for example setting "finger count = 5" or "index finger straight", should then reliably change the output, which neither the depth-map pipelines nor existing generative concept bottlenecks have been shown to do for articulated anatomy.
Core Claim
Generative concept bottleneck models face a known dilemma, and both answers in the literature are untested on anatomy. Concept Bottleneck Generative Models (CBGM) inserts a concept layer into GANs, VAEs and diffusion models and keeps image quality with an "unknown concept" residual. That residual is trained only to be orthogonal to the known concepts. CoCo-Bot argues that such residuals are cheating channels. It removes them, steering a frozen StyleGAN2 with energy gradients over concepts instead. Both papers work only with global attributes such as "smiling" or bird parts.
Hands make the dilemma sharp. Orthogonality to a handful of binary concepts does not stop a residual from carrying the full 3D pose of a mangled hand, which is the "structural leakage" that RHanDS documents. Removing the residual entirely risks the plastic-texture regime that HandRefiner found at high control strength.
Our claim is testable. If the residual is trained so that a probe cannot recover kinematic state from it, then:
- concept interventions on hands become faithful;
- the persistence of the original deformity drops;
- texture quality matches the unconstrained case;
- there is no control-strength knob to tune.
Research Program
Concepts. The concept vocabulary is derived automatically from fitted parametric hand models (MANO), so no new hand annotation is needed. It has four parts:
- per-finger presence, which gives the finger count;
- per-joint flexion, discretized into a few bins of anatomically valid range;
- pairwise depth order for adjacent fingers;
- a contact/fusion flag between neighbouring fingers.
The flexion bins and depth order are what make the representation readable and editable. A raw MANO axis-angle vector is precise but not something a user reasons with. The depth-order concept follows HandRefiner's finding that 2D keypoints fail precisely because they lack depth ordering.
Training data. FreiHAND supplies ground-truth MANO fits. HaGRID supplies scale and diversity, using pseudo-labels from a mesh reconstructor. The pseudo-labeler bias that the file flags for CoCo-Bot applies to these pseudo-labels too, so we report results separately on FreiHAND, where the labels are real.
Model. The inpainter is a latent diffusion inpainting model. Structure enters only through CBGM-style concept embeddings, using the prescient-design/CBGM layers. A residual encoder reads the masked malformed hand and its surroundings. The objective is
$$\mathcal{L} = \mathcal{L}_{\text{denoise}} + \lambda_{o}\,\mathcal{L}_{\text{orth}} - \lambda_{a}\,\mathcal{L}_{\text{probe}}(r \to k).$$
The three terms are as follows:
- $\mathcal{L}_{\text{denoise}}$ is the standard diffusion loss.
- $\mathcal{L}_{\text{orth}}$ is CBGM's orthogonality loss between the residual $r$ and the concept embeddings.
- $\mathcal{L}_{\text{probe}}(r \to k)$ is the loss of a probe network that tries to predict the full continuous kinematic state $k$ (MANO pose and finger count) from $r$. It enters with a negative sign through gradient reversal, so the residual is pushed to make prediction fail.
The key difference from CBGM is that leakage is penalized against the whole kinematic state, not just against the concept labels.
Evaluation of leakage. After training, we measure leakage with a fresh probe trained from scratch on the frozen residual. This guards against the encoder merely fooling the specific adversary it was trained against.
Inference. A concept predictor reads the malformed hand and proposes concept values, which are anatomically constrained (for example, five fingers). The user or an automatic rule can inspect and override these values. The inpainter then repaints from the concepts plus the scrubbed residual. A failed fit therefore shows up as a visible, editable concept value rather than a silently wrong depth map. This addresses the failure HandRefiner reports for small and occluded hands.
Baselines
Do nothing. The first reference point is the unrefined generated hand. HandRefiner reports a mean per-joint position error (MPJPE) of 17.722 and FID of 77.60 on its HAGRID benchmark before refinement.
HandRefiner. The strongest structural competitor is HandRefiner: MediaPipe detection, Mesh Graphormer fit, and a depth-ControlNet inpainter. It reports FID 74.12 and MPJPE 7.878 on that benchmark. We rerun it with its adaptive control strength search, using the same base inpainting model, the same crops, and a matched hyperparameter search budget.
RHanDS. The strongest texture-preserving competitor is RHanDS. It reports FID 73.63. No reported MPJPE for RHanDS was found on Lacuna, so we measure it ourselves. RHanDS needs a paired reference hand, so we evaluate it on the subset where one exists and report that subset separately.
Ablations isolating the key ingredient. All three use our architecture with identical concepts and compute:
- CBGM-orthogonality-only: the probe term is removed ($\lambda_a=0$).
- No-residual (CoCo-Bot-like): structure and texture both come from concepts plus unmasked surrounding context.
- Unconstrained residual: neither leakage term is used.
What counts as winning. We beat HandRefiner if we match its MPJPE and FID within noise while lowering deformity persistence and raising intervention fidelity. We also need to stay off the plastic-texture curve, meaning no FID penalty as structural adherence rises. Beating the CBGM-orthogonality ablation on leakage and persistence is the result that validates the core claim.
Why not just…?
Why not just edit the MANO parameters inside HandRefiner or RHanDS? MANO is already a kinematic representation, and RHanDS lets users adjust the rig. This is the most serious threat. Our answer is empirical. A depth-conditioned inpainter that also sees the malformed pixels can override the depth guidance, and HandRefiner's need for high control strength suggests it does. We test this directly by editing MANO pose in HandRefiner and measuring how often the edit appears in the output, compared with the same edit applied as concept clamping in our model.
To separate the representation from the scrubbing, we also bolt our structure-blind residual onto HandRefiner's depth pathway. If that hybrid matches our model, the honest conclusion is that the scrubbed residual is the contribution, and discrete concepts add only readability. We would report it that way.
Why not just tune control strength, as HandRefiner already does? Tuning control strength trades along the same curve we are trying to remove, and it costs 3.5x latency. We plot FID against structural adherence across control strengths for HandRefiner, and across $\lambda_a$ for our model. The claim fails if our model's curve does not dominate.
Why not use HandEval as a reward and run preference optimization? HandEval raises SDXL's HandScore from 2.31 to 3.66 as a reward signal. Like the curriculum in HG-DPO, though, it optimizes a single scalar and gives no handle for a specific correction. We include a HandEval-guided SDXL baseline for HandScore only. We use HandEval's defect taxonomy as an independent judge rather than as a training signal, so that we are not graded by our own optimizer.
Proposed Experiments
Data. Evaluation uses three sets:
- HandRefiner's HAGRID refinement benchmark, for comparability.
- HandPair's degraded hands. These are perturbations of real hands, so the true hand is known and recovery can be scored against it.
- HandPair's 100 frontier-model images with human ratings.
We stratify all results by hand size in the frame and by hand-object occlusion, where mesh fitting is known to fail.
Accuracy metrics.
- FID on hand crops.
- MPJPE against ground truth on HandPair. On HAGRID we use a reconstructor that is not used inside our pipeline, to avoid circularity.
- HandScore, alongside HandEval's reported correlation with human ratings of 0.6693.
Interpretability metrics.
- Intervention fidelity: the fraction of clamped concepts that an independent detector reads back correctly. CBGM reports steerability of 25.6% against 8.7% for a conditional GAN on CelebA, but no hand figure exists.
- Residual leakage: the accuracy of a fresh probe predicting finger count and flexion from the residual.
- Deformity persistence: the rate at which HandEval assigns the original defect class to the repaired hand.
Decisive checks. Three results would decide the claim:
- Persistence and leakage should fall together as $\lambda_a$ rises, while FID stays flat. If FID climbs, the dilemma is real for hands, and CoCo-Bot's no-residual stance gains support.
- On small or occluded hands, correcting a wrong predicted concept should recover MPJPE that HandRefiner loses to its failed mesh fit.
- The HandRefiner-plus-scrubbed-residual hybrid tells us whether the concept vocabulary itself earns its place.
References
- Concept Bottleneck Generative Models
- CoCo-Bot: Energy-based Composable Concept Bottlenecks for Interpretable Generative Models
- RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance
- HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting
- FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images
- HaGRID -- HAnd Gesture Recognition Image Dataset
- HandEval: Taking the First Step Towards Hand Quality Evaluation in Generated Images
- Boost Your Human Image Generation Model via Direct Preference Optimization