TL;DR
MoPort is a view-aware 2D interface that allows pretrained 3D Motion Language Models (MoLMs) to process monocular 2D video directly. By using paired-view alignment and camera-modulated encoding, it maps ambiguous 2D observations from any viewpoint into a consistent 3D semantic token space, outperforming traditional 3D reconstruction pipelines.
What Problem Does This Paper Address?
Motion Language Models (MoLMs) have shown impressive capabilities in describing human actions, but they have a major deployment hurdle: they are almost exclusively trained on high-quality 3D Motion Capture (MoCap) data. In the real world, we mostly have monocular 2D video.
This creates two significant gaps:
- Dimensional Gap: 2D joint sequences lack the depth information present in 3D training data.
- Viewpoint Ambiguity: The same 3D motion looks drastically different depending on the camera angle. An arm reaching forward might look like it's reaching upward if the camera is positioned high and looking down.
Existing solutions often try to "lift" 2D poses to 3D first, but these reconstruction pipelines are computationally heavy and prone to errors that propagate into the language model.
Methods
The authors introduce MoPort, a "plug-and-play" interface that maps 2D joint sequences directly into the latent and token spaces of a frozen, pretrained MoLM. It consists of two primary novel components:
- Paired View Alignment (PVA): During training, the model sees two different 2D projections of the same 3D motion. It is supervised by a "3D teacher" (a pretrained 3D encoder) to ensure that both views map to the same 3D latent representation. A specific "view loss" explicitly pulls these paired latents together to enforce cross-view consistency.
- Camera-Modulated Encoding (CME): To resolve depth ambiguity, the model needs to know how it's looking at the person. CME includes a camera head that estimates extrinsics from the 2D sequence. These estimates are used to transform 2D joint rays into a shared reference frame and to modulate the encoder's features using Feature-wise Linear Modulation (FiLM).
Datasets
The model was trained and evaluated on:
- HumanML3D & KIT-ML: Standard benchmarks for motion-language tasks, using synthetic 2D projections for training.
- Fit3D: A real-world RGB video dataset used to validate the model's performance on actual camera footage rather than just synthetic projections.
Key Insights and Ideas
- Semantic Consistency over Geometric Accuracy: MoPort doesn't try to perfectly reconstruct 3D coordinates. Instead, it focuses on mapping 2D observations to the semantic tokens the language model already understands. This "shortcut" is more robust for captioning than full 3D lifting.
- Viewpoint Awareness is the Secret Sauce: Simply adding camera information (CME) and enforcing consistency (PVA) allows the model to handle "Out-of-Distribution" (OOD) viewpoints—angles it never saw during training—with remarkably little performance degradation.
- Portability: Because MoPort maps to the native token space of the MoLM, it can be swapped into different architectures (like MotionGPT or MG-MotionLLM) without needing to retrain the heavy language model components.
Key Results
MoPort significantly outperforms baselines that rely on 3D estimation or simpler 2D-to-3D mapping.
- HumanML3D Performance: It achieves an R@1 of 0.5674, recovering 95.8% of the performance of a model using ground-truth 3D data.
- Robustness: On OOD viewpoints, MoPort's performance only drops by 3.1%, whereas previous state-of-the-art 2D interfaces dropped by over 22%.
- Real-World Validation: On the Fit3D dataset, MoPort achieved 80.36% accuracy, even outperforming the ground-truth 3D baseline in some cases, likely due to its better handling of real-world camera perspectives.
Limitations and Future Work
Despite its strengths, MoPort has a few notable limitations:
- Intrinsic Knowledge: It currently assumes camera intrinsics (like focal length) are known.
- Synthetic Training Bias: While it works on real video, it is still primarily trained on synthetic projections of MoCap data.
- Complex Ground Interactions: The model still struggles with motions that involve significant interaction with the ground (e.g., lying down or complex falls).
Future research directions include jointly estimating camera intrinsics and training on larger-scale, "in-the-wild" real-world video datasets to further close the domain gap.
Summary updated