MoPort: A View-Aware 2D Interface for Motion Language Models

Lixing Tan, Qing Xia, Yuting Guo, Shuai Li, Aimin HaoarXiv 2026 · arXiv

TL;DR

MoPort is a view-aware 2D interface that allows pretrained 3D Motion Language Models (MoLMs) to process monocular 2D video directly. By using paired-view alignment and camera-modulated encoding, it maps ambiguous 2D observations from any viewpoint into a consistent 3D semantic token space, outperforming traditional 3D reconstruction pipelines.

What Problem Does This Paper Address?

Motion Language Models (MoLMs) have shown impressive capabilities in describing human actions, but they have a major deployment hurdle: they are almost exclusively trained on high-quality 3D Motion Capture (MoCap) data. In the real world, we mostly have monocular 2D video.

This creates two significant gaps:

  1. Dimensional Gap: 2D joint sequences lack the depth information present in 3D training data.
  2. Viewpoint Ambiguity: The same 3D motion looks drastically different depending on the camera angle. An arm reaching forward might look like it's reaching upward if the camera is positioned high and looking down.

Existing solutions often try to "lift" 2D poses to 3D first, but these reconstruction pipelines are computationally heavy and prone to errors that propagate into the language model.

Methods

The authors introduce MoPort, a "plug-and-play" interface that maps 2D joint sequences directly into the latent and token spaces of a frozen, pretrained MoLM. It consists of two primary novel components:

  • Paired View Alignment (PVA): During training, the model sees two different 2D projections of the same 3D motion. It is supervised by a "3D teacher" (a pretrained 3D encoder) to ensure that both views map to the same 3D latent representation. A specific "view loss" explicitly pulls these paired latents together to enforce cross-view consistency.
  • Camera-Modulated Encoding (CME): To resolve depth ambiguity, the model needs to know how it's looking at the person. CME includes a camera head that estimates extrinsics from the 2D sequence. These estimates are used to transform 2D joint rays into a shared reference frame and to modulate the encoder's features using Feature-wise Linear Modulation (FiLM).
Figure 2. MoPort framework. (a) A shared MoPort encodes different 2D projections of one 3D motion and aligns them with latent and token targets from a pretrained 3D encoder. The view loss pulls the two latents together, and sampled cameras supervise the camera head. (b) The camera head estimates the extrinsics as a, and Gram-Schmidt gives bR, which rotates the rays into the 3D reference frame. An MLP maps the standardized R6D representation c of bR to multi-scale FiLM parameters. (c) At inference, YOLOv8x [11] and ViTPose-H [37] extract 2D tracks, which MoPort maps to the native tokens of a pretrained MoLM. Dashed arrows indicate training supervision.

Datasets

The model was trained and evaluated on:

  • HumanML3D & KIT-ML: Standard benchmarks for motion-language tasks, using synthetic 2D projections for training.
  • Fit3D: A real-world RGB video dataset used to validate the model's performance on actual camera footage rather than just synthetic projections.

Key Insights and Ideas

  • Semantic Consistency over Geometric Accuracy: MoPort doesn't try to perfectly reconstruct 3D coordinates. Instead, it focuses on mapping 2D observations to the semantic tokens the language model already understands. This "shortcut" is more robust for captioning than full 3D lifting.
  • Viewpoint Awareness is the Secret Sauce: Simply adding camera information (CME) and enforcing consistency (PVA) allows the model to handle "Out-of-Distribution" (OOD) viewpoints—angles it never saw during training—with remarkably little performance degradation.
  • Portability: Because MoPort maps to the native token space of the MoLM, it can be swapped into different architectures (like MotionGPT or MG-MotionLLM) without needing to retrain the heavy language model components.

Key Results

MoPort significantly outperforms baselines that rely on 3D estimation or simpler 2D-to-3D mapping.

  • HumanML3D Performance: It achieves an R@1 of 0.5674, recovering 95.8% of the performance of a model using ground-truth 3D data.
  • Robustness: On OOD viewpoints, MoPort's performance only drops by 3.1%, whereas previous state-of-the-art 2D interfaces dropped by over 22%.
  • Real-World Validation: On the Fit3D dataset, MoPort achieved 80.36% accuracy, even outperforming the ground-truth 3D baseline in some cases, likely due to its better handling of real-world camera perspectives.
Qualitative comparison showing MoPort correctly identifying a "stair descent" across multiple viewpoints where other models fail.

Limitations and Future Work

Despite its strengths, MoPort has a few notable limitations:

  • Intrinsic Knowledge: It currently assumes camera intrinsics (like focal length) are known.
  • Synthetic Training Bias: While it works on real video, it is still primarily trained on synthetic projections of MoCap data.
  • Complex Ground Interactions: The model still struggles with motions that involve significant interaction with the ground (e.g., lying down or complex falls).

Future research directions include jointly estimating camera intrinsics and training on larger-scale, "in-the-wild" real-world video datasets to further close the domain gap.

Summary updated

Abstract

Motion language models (MoLMs) encode 3D human motion as discrete tokens for language-based understanding. Deploying them on monocular video raises two problems. Pose detection yields 2D joint sequences, leaving a dimensional gap from the 3D motion used in training. Moreover, the 2D observation changes with the camera viewpoint, while the depth lost in projection makes the 2D pose ambiguous across viewpoints. We propose MoPort, a view-aware 2D interface that enables MoLMs to take 2D observations as input. We introduce Paired View Alignment (PVA), which gives two views of the same motion a shared 3D teacher target and directly constrains the consistency of their latents, so that 2D observations from different viewpoints are mapped to a consistent semantic representation. Camera-Modulated Encoding (CME) estimates the camera extrinsics from the 2D sequence and uses them to modulate encoding, providing the 2D latents with geometric information about the viewpoint. MoPort achieves R@1 of 0.5674 on HumanML3D and 0.3286 on KIT-ML, outperforming all evaluated baselines. On HumanML3D, its R@1 decreases only from 0.5763 on in-distribution viewpoints to 0.5585 on out-of-distribution viewpoints. We validate portability on three downstream MoLMs. Furthermore, on the real-world RGB video dataset Fit3D, MoPort achieves 80.36\% accuracy. Code will be released upon acceptance.
BibTeX
@misc{tan2026moport,
  author = {Lixing Tan and Qing Xia and Yuting Guo and Shuai Li and Aimin Hao},
  title = {{MoPort: A View-Aware 2D Interface for Motion Language Models}},
  year = {2026},
  eprint = {2611.01541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01541},
}
Figure