When several vehicles share what they see, the receiving vehicle has to place each partner's observations into its own coordinate frame. If a partner's reported position or heading is slightly wrong, everything it sends lands in the wrong place. A heading error of one degree shifts an object 50 meters away by almost a meter. Robust collaborative perception methods correct this by assuming that any disagreement between agents comes from pose error. Dynamic-scene reconstruction methods make the opposite assumption: they read any disagreement across views as object motion. In real traffic both happen at once. A passing truck and a partner with a drifting compass produce the same symptom, which is a 3D structure that appears in different places to different observers. No current pipeline asks which of the two caused a given discrepancy.
We propose to answer that question explicitly, and to use open-vocabulary 3D features as the instrument for doing so. A pose error has a distinctive signature: it is a single rigid transform applied to everything one agent observes, and its rotational part grows with range. Object motion is per-object, and every agent that sees the moving object at the same time sees the same motion. Arranged as a grid of agents by objects, the discrepancies therefore split into an agent-shared part and an object-shared part. Splitting them requires static landmarks of many kinds: poles, building corners, signs, parked cars. Closed-set vehicle detectors cannot supply these. Language-aligned 3D features can, because they let the receiver recognize "the same lamppost" across agents without a fixed label set.
Core Claim
Inter-agent pose correction and static/dynamic decomposition should be solved jointly, as one robust factorization over open-vocabulary 3D instances. Solving either one alone corrupts the other. Pose-correction methods such as RoCo build local graphs of detected vehicle boxes, match them across agents, and refine poses by graph optimization. Their anchors are closed-set vehicles, which are exactly the objects most likely to be moving, so every moving car pulls the pose estimate toward a wrong answer. Decomposition methods such as Split4D separate static from dynamic 3D Gaussians during optimization but assume known poses. Under pose noise they will label a misaligned building facade as dynamic. We claim that a joint estimator anchored on open-vocabulary static structure recovers poses more accurately than either kind of method in scenes with moving traffic. We further claim that it yields a cleaner shared map that can be queried with language.
The Estimator
Each agent $a$ transmits compact 3D Gaussian primitives (explicit ellipsoids with position, shape, color and features), following the bandwidth-efficient packaging of VOGS-CP. We add per-Gaussian instance and language features in the style of COS3D. COS3D trains a sharp instance feature from Segment Anything masks, then learns a quick mapping from instance space to the language space of CLIP, the image–text embedding model. Grouping a partner's Gaussians by instance gives a set of instance centroids $x_{a,o}$ for agent $a$ and instance $o$, expressed with the agent's reported pose. Instances are associated across agents by combining language-feature similarity with local geometric layout. This is RoCo's matching step, but its vocabulary covers poles, trees and walls, not only vehicles.
The receiver then solves
$$ \min_{\{\Delta T_a\},\{y_o\},\{m_o\}} \sum_{a,o} \rho\big(\lVert \Delta T_a\, x_{a,o} - (y_o + m_o\,\tau_a)\rVert\big) + \lambda \sum_o \lVert m_o \rVert_2 , $$
where $\Delta T_a$ is the rigid correction to agent $a$'s pose, $y_o$ is instance $o$'s consensus position at the receiver's timestamp, $m_o$ is its velocity, $\tau_a$ is agent $a$'s transmission delay, $\rho$ is a robust Welsch loss borrowed from EnerGS, and $\lambda$ sets the strength of the sparsity term. That term is a group-sparsity penalty: it encodes the prior that most instances are static, so a nonzero velocity must be justified by data. The open-vocabulary features enter in one more place. The language prediction "movable thing" (car, person, cyclist) lowers $\lambda$ for that instance but never sets $m_o$ to zero or to non-zero. This matters because a parked car is an excellent anchor and a passing car is a poison one, and the label alone cannot tell them apart.
The problem is identifiable whenever each agent sees at least three non-collinear static instances and each moving instance is seen by at least two agents. A static instance can then only explain the residual through $\Delta T_a$. Distant static instances constrain rotation most strongly, which is exactly where Ren locates the most damaging errors.
Research Program
The work has three stages.
The first stage builds the factorization above on top of COS3D-style instance and language fields lifted from each agent's cameras and LiDAR. It is solved by a few Gauss–Newton steps per frame, warm-started from the previous frame to keep runtime within the budget set by learned aligners. We report wall-clock time against RoCo's iterative loop and against V2X-ViT's roughly 57 ms per frame.
The second stage feeds the attribution back into the geometric prior. EnerGS partitions space into occupied, free and unknown regions, and its free-space barrier pushes Gaussians out of empty air. EnerGS itself notes that moving agents violate this free-space assumption. We build the energy field only after pose correction, and only from rays that end on instances attributed as static. Gaussians attributed as dynamic are exempt from the barrier and move with their estimated $m_o$. We study this as a downstream consequence of correct attribution, not as a separate contribution.
The third stage analyzes how attribution errors propagate into open-vocabulary query accuracy. We expect two failure modes. Ghost duplicates of the kind documented by MRCNet should split language mass across copies. Frozen trails left by moving objects, the transient-densification artifact studied by RobustSplat, should create false positives for queries such as "car".
Baselines
The do-nothing reference fuses partner Gaussians with the reported noisy poses and the static scene assumption, exactly as VOGS-CP does with known extrinsics. Run under injected noise, it shows how much is lost. A clean-pose oracle bounds the attainable gain from the other side.
The strongest existing pose-correction competitor is RoCo. On V2XSet it reports 90.0% AP under 0.8 m translation noise, and on DAIR-V2X it reports [email protected] gains of about 1.7–2.3% over CoAlign and V2X-ViT. RoCo is the right comparison because it is the only method in the file that, like ours, explicitly re-estimates relative pose from object correspondences. We will run it on the same detector backbone, the same noise draws and the same latency settings. We will also give it our open-vocabulary instance centroids as an additional input, so that any remaining gap is attributable to the motion term and not to anchor vocabulary. Beating RoCo means a lower rotation and translation error, and a higher detection AP, specifically in the subset of frames containing moving vehicles, at heading noise of 1° and above.
The strongest learned-alignment competitor is V2X-ViT. It reports a 21.2% AP gain over single-vehicle perception on V2XSet and only a 5–10% drop under 0.2 m pose error with 100 ms delay. NEAT reports an 8.85% [email protected] gain over F-Cooper on OPV2V. Both are trained by their released recipes on the same splits.
For decomposition, Split4D is run with poses from the do-nothing baseline and, separately, with poses from RoCo. This tests whether a sequential pipeline (align, then decompose) matches joint estimation.
Three ablations isolate the key ingredient. The first removes the motion term by fixing all $m_o=0$, which reduces our method to open-vocabulary RoCo. The second replaces the motion term with hard semantic gating, discarding every instance whose language label is movable. The third removes the open-vocabulary anchors, keeping vehicles only.
Why not just…?
Why not just mask out everything the language model calls "vehicle" or "person" and align on the rest? This is the most serious threat, and it is ablation two above. It fails in principle when static landmarks are sparse, as on open highways, and parked vehicles are the only well-distributed anchors. It also leaves moving traffic unattributed and so does nothing for the dynamic map. Experiments will settle it directly: if semantic gating matches the joint estimator on pose error across noise levels and scene types, we will report that the motion term is unnecessary for alignment and narrow the claim to map quality.
Why not just train with injected pose noise? Ren reports that naive augmentation makes collaborative models conservative, so that they learn to ignore partner inputs. We include a noise-augmented VOGS-CP variant and measure its gap to single-agent perception to confirm this in the Gaussian-sharing setting.
Why not just run classical registration or feature matching between agents? HAMMER aligns robots with place recognition and local structure-from-motion, but only once, indoors and with large visual overlap. Intersections and opposite-lane views often lack that overlap. Registration of raw point clouds, for its part, has no notion of motion and will align to a passing truck. We run point-cloud registration as a baseline and break its error down by overlap and by the fraction of moving points.
Why not just align first, then run a dynamic reconstruction method? The sequential pipeline is the Split4D-after-RoCo baseline. It fails if moving vehicles bias the first-stage pose, and that bias is exactly what the frames-with-moving-vehicles split measures.
Proposed Experiments
Pose-error and detection experiments use V2XSet and OPV2V. We inject the Gaussian translation offsets of 0.2–0.8 m that V2XSet already models, and we add heading noise from 0° to 2°, since rotation is the regime Ren identifies as most damaging. Results are reported separately for frames with and without moving vehicles near overlapping regions. Motion ground truth comes from the per-frame box trajectories in the simulated sequences. This lets us score motion attribution directly, as precision and recall of instances labeled moving, which no baseline reports.
Real-world transfer is tested on DAIR-V2X and V2V4Real. Ground-truth poses there are themselves imperfect, so we report detection AP. DAIR-V2X gives late-fusion and early-fusion references of 41.90 and 50.03 AP on its synchronized benchmark. For NEAT on V2V4Real, no numerical result was found on Lacuna.
Map and language quality are evaluated on Semantic-OPV2V by querying each semantic class name through the language field zero-shot, without training on the occupancy labels, and computing mIoU and IoU. VOGS-CP reports +3.28 mIoU and +22.41 IoU over collaborative baselines but no absolute figure in the file. We will therefore rerun VOGS-CP under noise ourselves rather than quote it.
The decisive check is a crossing curve. Plot pose error against the fraction of moving instances in the shared view. RoCo with open-vocabulary anchors should degrade as that fraction rises. The joint estimator should stay flat. Semantic gating should stay flat only when static landmarks are plentiful. If this pattern does not appear, the central claim is wrong. A secondary check repeats the same curve for open-vocabulary query mIoU, to confirm that better attribution actually reaches the semantic map.
References
- RoCo: Robust Cooperative Perception By Iterative Object Matching and Pose Adjustment
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Vision-Only Gaussian Splatting for Collaborative Semantic Occupancy Prediction
- COS3D: Collaborative Open-Vocabulary 3D Segmentation
- EnerGS: Energy-Based Gaussian Splatting with Partial Geometric Priors
- Learning to Communicate and Correct Pose Errors
- Multi-Agent Collaborative Perception via Motion-Aware Robust Communication Network
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer
- Align Before Collaborate: Mitigating Feature Misalignment for Robust Multi-agent Perception
- HAMMER: Heterogeneous, Multi-Robot Semantic Gaussian Splatting