TL;DR
CoCode-World introduces a shared 3D proxy framework that enables generative world models to produce consistent multiplayer video. By generating multiple player views jointly within a single video transformer branch, the system ensures that shared entities—like characters and objects—maintain the same appearance and identity across different camera perspectives and occlusions.
What Problem Does This Paper Address?
Generative world models aim to create "playable" environments, but scaling them to multiplayer settings is surprisingly difficult. In a shared world, each player has a unique camera view. If you generate these views independently, you run into "hallucination" mismatches: a character might be wearing a red jacket in Player A’s view but a blue one in Player B’s.
While shared 3D geometry can help with spatial layout, it doesn't guarantee visual consistency. Existing methods often use complex "cross-agent attention" modules to sync views, which can be hard to train and scale. The authors tackle the challenge of coupling precise spatial control (where things are) with appearance consistency (what things look like) across multiple views.
Methods
The core of CoCode-World is the separation of structural control from visual synthesis.
- Shared 3D Proxies: Instead of feeding the model raw game maps, the authors use "proxies"—coarse, textureless 3D representations of the scene. These proxies specify the layout, motion, and occlusions of entities based on the game engine's state.
- Joint Generation in a Shared Context: This is the "aha" architectural move. Rather than using separate generation branches for each player, the authors concatenate the two views side-by-side into a single wide canvas.
- Native Attention for Communication: By placing both views in one context, the model’s native self-attention mechanism naturally handles cross-view communication. A target region in Player A's view can "attend" to the appearance of a character in Player B's view (or the reference image) to ensure they match.
- LoRA Adaptation: The researchers used a massive pretrained video model (MiniMax-H3) and kept it frozen, training only lightweight Low-Rank Adaptation (LoRA) layers. These layers learn the mapping from the colorful but textureless proxies to high-fidelity RGB video.
Datasets
The authors introduced a new two-player GTA V dataset to train and evaluate their model:
- Training Data: 7,846 paired clips (~5 seconds each) captured from a private GTA V server. It includes synchronized RGB video, camera parameters, and the underlying 3D world state.
- Benchmark: A set of 150 cases covering four categories: exploration, vehicle riding, fighting, and environment interaction. To test generalization, they used GPT Image 2.5 to stylize reference images, creating "unseen" characters and scene styles (e.g., turning GTA into a LEGO-like world).
Key Insights and Ideas
- Concatenation is King: The simplest way to get two views to talk to each other is to put them in the same room. Using a single-branch joint generation approach outperformed more complex dual-branch architectures with explicit cross-attention.
- The "Goldilocks" Proxy: More detail isn't always better. The authors found that providing full character skeletons (limb-level detail) often made the generated motion look rigid and unnatural. Coarse character geometry combined with detailed environment lighting worked best, as it allowed the video model's internal "prior" to handle natural human articulation.
- Zero-Shot Style Transfer: Because the model learns to follow the structure of the proxy rather than the specific pixels of the training data, it can render the same gameplay in entirely different visual styles (e.g., watercolor, paper cutout) just by changing the RGB reference image.
Key Results
CoCode-World achieved state-of-the-art results in cross-view identity and video quality:
- Identity Consistency: On a 0–5 scale judged by a large language model (Qwen), CoCode-World scored 3.847, significantly beating Seedance 2.5 (3.552) and SolarRenderer (3.696).
- Visual Quality: It outperformed baselines in DOVER (visual quality) and LAION (aesthetic) scores, showing that joint training improves the overall look of the video, not just the consistency.
- Qualitative Success: As seen in Figure 4, while baselines often "lose" characters or change their clothes between views, CoCode-World maintains stable identities even in complex night scenes or farm environments.
Limitations and Future Work
- The "N-Player" Problem: While the method works for two players, adding more players increases the token sequence length. Because attention complexity is quadratic, scaling to a 16-player battle royale would require much more efficient memory management.
- Real-Time Latency: The current model uses 50 denoising steps. While it can be distilled down to 8 steps with good quality, true real-time "playable" performance (1-2 steps) remains a challenging frontier.
- Passive Rendering: The current system is a renderer; it doesn't "know" how to update the world state based on player actions. Integrating this with a learned world-logic engine is a key next step.
Summary updated