CAMO: Category-Agnostic 3D Motion Transfer from Monocular 2D Videos

Taeyeon Kim, Youngju Na, Jumin Lee, Minhyuk Sung, Sung-eui YoonSubmission to ICLR 2026 · Rejected

TL;DR

CAMO is a category-agnostic framework that transfers articulated motion from a single 2D monocular video directly to any 3D character mesh. By combining articulated 3D Gaussian splatting with a novel morphology-adaptive parameterization, it bypasses the need for category-specific templates (like SMPL) or expensive 3D supervision, achieving high-fidelity animation in under 10 minutes.

What Problem Does This Paper Address?

Animating 3D characters usually requires high-quality 3D motion data, which is expensive to capture. While we have a surplus of 2D video, transferring motion from a flat video to a 3D model is a nightmare of "shape-pose ambiguity." Is the character's leg short because of its anatomy, or because it's pointing toward the camera?

Existing solutions often fall into two traps:

  1. The Template Trap: They rely on category-specific priors (like the SMPL model for humans). If you want to animate a creature that doesn't fit a pre-defined template—like a dragon or a stylized alien—these methods fail.
  2. The Cascade Trap: They use a "reconstruct-then-retarget" pipeline. Errors in the initial 3D reconstruction of the video source snowball, leading to distorted or physically impossible motions in the final 3D asset.

Methods

CAMO (Category-Agnostic MOtion transfer) introduces a direct optimization strategy that aligns a 3D target mesh to 2D video observations. Its architecture rests on three pillars:

  • Articulated 3D Gaussian Splatting (Articulated-GS): Instead of traditional meshes, the system represents the character using 3D Gaussians attached to a kinematic skeleton. This allows for differentiable rendering, meaning the system can "see" the difference between its current pose and the video frame and backpropagate that error to fix the pose.
  • Morphology-Adaptive Parameterization: This is the "secret sauce." The model doesn't just optimize pose; it treats bone lengths, global scale, and local offsets as learnable parameters. This allows the 3D asset to "stretch" its proportions to match the source subject in the video while maintaining its original topological identity.
  • Dense Semantic Correspondences: To solve the ambiguity of overlapping limbs, CAMO uses a pre-trained vision foundation model to establish "semantic glue." It matches pixels in the 2D video to specific vertices on the 3D mesh (e.g., "this pixel is a left paw, and this vertex is a left paw"), providing a strong geometric guide that prevents the model from getting confused during complex movements.

Datasets

The authors evaluated CAMO across several diverse datasets to prove its "category-agnostic" claims:

  • DeformingThings4D (DT4D): Used for quadrupeds and "Others" (non-standard animals like birds, whales, and dinosaurs).
  • Mixamo: A standard library for humanoid character animations.
  • Real-World Videos: Casual monocular videos from the DAVIS dataset and public repositories like Pexels, featuring animals in the wild with complex backgrounds and occlusions.

Key Insights and Ideas

  • Disentanglement is Key: By explicitly decoupling global scale, bone lengths, and local offsets from the time-varying pose, the authors mathematically prove that the model can uniquely identify the correct motion. This prevents the optimization from "cheating" by distorting the shape to fit a bad pose.
  • Direct Image-Space Supervision: By bypassing intermediate 3D reconstruction of the source video, CAMO avoids the cumulative errors that plague two-stage pipelines. It treats the 2D video as the "ground truth" observation.
  • Foundation Models as Anchors: Leveraging pre-trained, geometry-aware features (like those from DINOv2 or specialized extractors) allows the system to understand character parts without ever being explicitly trained on that specific animal category.

Key Results

  • Superior Accuracy: CAMO outperformed state-of-the-art baselines (like Transfer4D and NPR+) across all categories. In some challenging animal categories, it reduced error metrics (PMD and FID) by up to 85%.
  • Efficiency: The entire optimization process completes in under 10 minutes on a single NVIDIA RTX 4090 GPU.
  • Robustness: The system is remarkably resilient to "noisy" starts. Even when the initial camera orientation was off by 180 degrees (facing the wrong way), CAMO was able to recover and produce accurate motion transfer.
  • Versatility: The paper demonstrates successful "cross-category" transfers, such as applying a rabbit's hopping motion to a deer or even transferring animal motions to humanoid characters.

Limitations and Future Work

  • Rigid Kinematics: Currently, CAMO is limited to articulated skeletons. It struggles with "squishy" non-rigid dynamics like soft-tissue deformation, jiggling fat, or secondary motions like hair and tail sway.
  • Physics Blindness: The model prioritizes visual alignment over physical plausibility. It doesn't "know" about gravity or floor contact, which can sometimes lead to foot sliding or lack of weight.
  • Sequence Length: While it handles standard clips well, very long sequences can saturate the capacity of the underlying MLP, leading to "temporal drift" where the 3D model slowly loses alignment with the source.

The authors suggest that future versions could integrate monocular depth predictors or physically grounded constraints (like Jacobian-space consistency) to further improve realism in complex, "in-the-wild" scenarios.

Summary updated

Abstract

Motion transfer from 2D videos to 3D assets is a challenging problem, due to inherent pose ambiguities and diverse object shapes, often requiring category-specific parametric templates. We propose CAMO, a category-agnostic framework that transfers motion to diverse target meshes directly from monocular 2D videos without relying on predefined templates or explicit 3D supervision. The core of CAMO is a morphology-parameterized articulated 3D Gaussian splatting model combined with dense semantic correspondences to jointly adapt shape and pose through optimization. This approach effectively alleviates shape-pose ambiguities, enabling visually faithful motion transfer for diverse categories. Experimental results demonstrate superior motion accuracy, efficiency, and visual coherence compared to existing methods, significantly advancing motion transfer in varied object categories and casual video scenarios.
BibTeX
@misc{kim2026camo,
  author = {Taeyeon Kim and Youngju Na and Jumin Lee and Minhyuk Sung and Sung-eui Yoon},
  title = {{CAMO: Category-Agnostic 3D Motion Transfer from Monocular 2D Videos}},
  year = {2026},
  eprint = {2601.02716},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.02716},
}
Figure