SK-Adapter: Skeleton-Based Structural Control for Native 3D Generation

Anbang Wang, Yuzhuo Ao, Shangzhe Wu, Chi-Keung TangarXiv 2026 · arXiv

TL;DR

SK-Adapter introduces a lightweight, parameter-efficient structural adapter that enables precise skeletal control for native 3D generation without sacrificing the generative priors of the underlying foundation model. By injecting 3D skeleton tokens directly into a frozen 3D flow transformer via cross-attention, it achieves state-of-the-art structural alignment and high-fidelity asset generation in under 15 seconds. The authors also release Objaverse-TMS, a newly curated dataset of 24k high-quality text-mesh-skeleton triplets to power future research.

What Problem Does This Paper Address?

While native 3D generative models (like Trellis) can synthesize high-fidelity 3D assets from text or images in seconds, they lack a mechanism for precise structural control. Prompts like "bend the knee 60 degrees" are too ambiguous for text encoders, and image prompts only provide view-specific cues.

Prior attempts at structural control typically rely on a "2D lifting" strategy—projecting 3D skeletons into 2D planes to condition multi-view diffusion models before reconstructing them back into 3D. However, compressing 3D structures into 2D introduces severe spatial ambiguity, depth flattening, and self-occlusion issues. Multi-stage reconstruction pipelines also tend to degrade texture quality and introduce geometric artifacts. SK-Adapter aims to bypass this lossy 2D projection bottleneck by treating the 3D skeleton as a first-class control signal directly within native 3D space.

Fig. 1: SK-Adapter efficiently generates 3D assets in native 3D domain from given skeletons with its lightweight and effective designs, which also supports skeleton adap- tation and flexible skeleton-based editing.

Methods

SK-Adapter uses a Parameter-Efficient Fine-Tuning (PEFT) strategy, keeping the pre-trained 3D generation backbone (Trellis) completely frozen while training only a lightweight adapter network consisting of two main components:

  1. Topology-Aware Encoding: Skeletons are more than just a collection of joint coordinates; their hierarchical connectivity matters. The authors employ Graph Relative Positional Encoding (GRPE) to capture both the spatial coordinates of the joints and their topological relationships (such as topological distance and edge relations like parent, child, sibling, or end-effector). This maps the skeleton into a set of sparse spatial tokens.
  2. Skeletal Cross-Attention Mechanism: These skeletal tokens are injected into each block of the frozen 3D flow transformer using novel cross-attention layers. This allows spatial voxels to dynamically "attend" to the most relevant skeletal joints.

To ensure training stability and preserve the base model's generative priors, the output of the cross-attention layer passes through a zero-initialized linear layer before being added back via a residual connection. At the start of training, the adapter outputs a "null signal," preventing the model from destabilizing its pre-trained distribution.

Fig. 2: Overview of the SK-Adapter framework. The GRPE module encodes the 3D skeleton’s joints and topology into sparse tokens. These tokens are injected into a frozen pre-trained backbone via trainable cross-attention layers.

Additionally, the authors extend this framework to Flexible 3D Editing by adapting the Repaint inpainting algorithm. Given a modified skeleton and a masked bounding box, the model can regenerate content strictly within the masked area while keeping the background intact, enabling tuning-free local additions or re-posing.

Datasets

A major bottleneck in structure-guided 3D generation has been the lack of high-quality datasets containing text, 3D meshes, and skeletal rigs simultaneously. To bridge this gap, the authors introduce Objaverse-TMS, a large-scale dataset consisting of 24k text-mesh-skeleton triplets.

Unlike previous datasets that relied on flawed automated rigging algorithms, Objaverse-TMS is curated from the intersection of Anymate and CAP3D (subsets of Objaverse-v1 and Objaverse-XL), ensuring expert-annotated, physically plausible skeletal rigs that remain perfectly aligned with their corresponding 3D meshes.

Fig. 3: Distribution of skeleton joint counts in Objaverse-TMS.

Key Insights and Ideas

  • Congruence of Control Space: The core insight is that precise structural control requires strict spatial alignment between the guidance signal and the generative latents. Operating entirely within the native 3D domain eliminates the multi-view inconsistency and depth ambiguity plaguing 2D-projection methods.
  • Topology Matters: Simply conditioning a network on 3D joint coordinates is insufficient for complex structures. Explicitly encoding graph topology (e.g., distinguishing a leaf-node "end-effector" like a fingertip from an internal joint) prevents structural collapse and ensures the generated mesh faithfully wraps around the skeleton.
  • Non-Invasive Adaptation: Zero-initialized residual connections allow a massive 3D foundation model to absorb strict geometric constraints gradually without suffering from catastrophic forgetting of its rich texture and shape priors.

Key Results

The authors evaluated SK-Adapter against state-of-the-art baselines including SKDream (a 2D-lifting method) and SpaceControl (a training-free spatial guidance method) using a balanced test set (TMS-eval) spanning humanoids, animals, and objects.

  • Structural Alignment: SK-Adapter achieved a state-of-the-art ReRigging Score of 0.2228 (where lower is better, and the ground-truth oracle reference is 0.2073), significantly outperforming SKDream (0.2818) and SpaceControl (0.2740).
  • Visual Fidelity: It achieved the highest human-aligned perceptual quality with a PickScore of 21.01 and drastically reduced the KD-DINO metric to 0.7778 (compared to 1.3809 for SKDream), proving that freezing the backbone preserves high-frequency details and coherent textures.
  • Efficiency: SK-Adapter generates a complete 3D asset in less than 15 seconds, matching the speed of training-free native methods and easily beating lifting-based pipelines (which take ~40 seconds per sample).

Limitations and Future Work

Despite its strong performance, SK-Adapter remains intrinsically bounded by the capabilities of its pre-trained 3D base model. It occasionally struggles with fine-grained geometric and textural details in region-sensitive areas like human or animal faces, which can appear distorted.

Furthermore, when the conditioning skeleton becomes excessively complex or self-intersecting—such as the intricate topology of human fingers—the structural guidance can become ambiguous, leading to messy local geometries. The authors suggest that future work could integrate more advanced, higher-resolution 3D foundation models and scale up training to larger datasets to mitigate these issues.

Summary updated

Abstract

Native 3D generative models have achieved remarkable fidelity and speed, yet they suffer from a critical limitation: inability to prescribe precise structural articulations, where precise structural control within the native 3D space remains underexplored. This paper proposes SK-Adapter, a simple yet efficient and effective framework that unlocks precise skeletal manipulation for native 3D generation. Moving beyond text or image prompts, which can be ambiguous for precise structure, we treat the 3D skeleton as a first-class control signal. SK-Adapter is a lightweight structural adapter network that encodes joint coordinates and topology into learnable tokens, which are injected into the frozen 3D generation backbone via cross-attention. This design allows the model to not only effectively ``attend'' to specific 3D structural constraints but also preserve its original generative priors. To bridge the data gap, we contribute the Objaverse-TMS dataset, a large-scale dataset of 24k text-mesh-skeleton pairs. Extensive experiments confirm that our method achieves robust structural control while preserving the geometry and texture quality of the foundation model, significantly outperforming existing baselines. Furthermore, we extend this capability to local 3D editing, enabling region-specific editing of existing assets with skeletal guidance, which is unattainable by previous methods. Project page: https://sk-adapter.github.io/
BibTeX
@misc{wang2026sk,
  author = {Anbang Wang and Yuzhuo Ao and Shangzhe Wu and Chi-Keung Tang},
  title = {{SK-Adapter: Skeleton-Based Structural Control for Native 3D Generation}},
  year = {2026},
  eprint = {2603.14152},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.14152},
}
Figure