A 3D scene graph turns a scanned room into a list of objects (nodes) and labeled relations between them (edges), such as "chair standing on floor" or "picture hanging on wall." The natural way to predict these graphs is to combine two inputs. Camera images carry rich appearance and meaning. Point clouds carry metric shape and pose. Joint training of such systems has a known failure: the network learns mostly from whichever input is easiest to exploit and leaves the other one undertrained. In scene graphs the easy input is appearance. Once you know the subject is a "picture" and the object is a "wall," the predicate is nearly determined, so image features plus category co-occurrence can satisfy most of the loss. The point-cloud branch then never learns the relative orientation needed for predicates that do depend on pose, such as "lying on" versus "standing on," "hanging on" versus "leaning against," or "attached to" versus "part of."
This note proposes a different fix. Instead of reweighting the losses of the existing task, which is the standard response to modality dominance, we give the weak modality a second task that the strong modality cannot shortcut. The geometry branch predicts the continuous relative rotation between every linked pair of objects. The rotation is written in the 6D form of On the Continuity of Rotation Representations in Neural Networks: the network outputs two 3D vectors, and Gram–Schmidt orthogonalization turns them into a rotation matrix. Unlike quaternions or Euler angles, this mapping has no jumps, so it can be regressed without sudden error spikes. The claim we test is that this edge-level geometric target corrects appearance dominance more effectively and more specifically than gradient-rebalancing methods do.
Core Claim
In camera-plus-point-cloud 3D scene graph prediction, appearance features dominate learning, and the cost falls on pose-dependent predicates. We claim that attaching a continuous 6D relative-rotation regression head to the geometry branch, supervised at every candidate edge, does three things. First, it increases how much the fused model relies on geometry, measured by the accuracy drop when that branch is removed at test time. Second, it improves recall on pose-dependent predicates more than G-Blend or MMPareto at matched compute. Third, its benefit is larger inside the fused model than inside a geometry-only model.
The third prediction separates our explanation, which is about rebalancing the two inputs, from the more boring one that rotation supervision is simply a good auxiliary loss for point encoders. If the gain is the same with or without images present, the dominance story is wrong. In that case we would report the result as "rotation auxiliaries help 3D scene graphs," which is a smaller contribution.
The four research directions play different roles here. Modality dominance is the problem. Structured scene graphs are the setting where the problem has a specific and measurable shape: a shortcut through category co-occurrence, similar to the language-prior shortcut Agrawal et al. documented in visual question answering. Cross-modal fusion is the architecture whose balance we study. The 6D representation is the tool that makes a dense rotation target practical to optimize.
Relation to Prior Work
Prior work falls into three groups.
Work that motivates the gap. Several papers give image and language signals to 3D scene graph models, but none of them address the competition between the two inputs.
- VL-SAT uses 2D image patches and CLIP text as a training-only oracle that guides a point-cloud model through cross-attention, then discards the oracle at test time.
- CCL-3DSGG distills CLIP into a 3D student through contrastive alignment.
- GMPSSG processes RGB-D sequences with Gaussian-mixture prototypes.
The first two sidestep competition between inputs by never fusing at inference. The third fuses without asking whether one stream starves the other. We study genuine test-time fusion and measure the starvation directly.
Work that supplies the remedies we must beat. These methods all rebalance gradients on copies of the same classification label:
- Gradient Blending reweights per-modality losses by how much each one overfits.
- MMPareto resolves conflicts between unimodal and multimodal gradients and rescales update magnitudes.
- Modality dropout blanks one input at random.
- MEFormer shares one decoder across LiDAR-only, camera-only, and fused streams for detection.
None of these change what the weak modality is asked to learn. That is the variable we change.
Work that supplies or constrains the method. - Zhou et al. supply the representation. They also show that quaternions reach near-180° worst-case error and train about 5× slower on ShapeNet pose estimation, though they evaluate only isolated objects and kinematic chains, never relations between objects. - A Planar-Symmetric SO(3) Representation for Learning Grasp Detection is a constraint. Continuity alone does not handle symmetric objects, so our loss must take a minimum over symmetry-equivalent targets. - PointFusion is a further constraint. It found that batch normalization harms metric regression, so the rotation head avoids normalizing away coordinate scale. - SGPN and LEO define the graph backbones and the 3DSSG protocol. Both keep edges purely categorical.
Research Program
Model. The base model has a point-cloud encoder per object and per edge, in the style of SGPN and SGFN. It adds a 2D branch that projects each object's points into the 3RScan RGB-D frames, pools image features there as PointFusion does, and fuses them with point features through cross-attention as in SGFormer. A graph network then predicts object and predicate labels.
Rotation head. On top of this we add a head that reads only the geometry branch's edge feature. For each candidate edge $(i,j)$ it outputs a 6D vector, which Gram–Schmidt converts to $\hat R_{ij}$. The target is the relative rotation $R_{ij} = R_i^\top R_j$ between the two objects' oriented frames. The loss is
$$\mathcal{L}_{\text{rot}} = \min_{S \in \mathcal{G}_j} \; d\big(\hat R_{ij},\, R_i^\top R_j S\big),$$
where $d$ is geodesic angle on rotations and $\mathcal{G}_j$ is the set of rotations under which object $j$'s frame is ambiguous. That set is the box symmetry group by default, and it is collapsed to rotation about the vertical axis for objects annotated as rotationally symmetric. The total objective adds $\lambda \mathcal{L}_{\text{rot}}$ to the standard scene graph losses, with one weight $\lambda$ tuned on validation.
Source of rotation targets. Targets come from per-instance oriented boxes in the 3RScan annotations. This is an assumption we must verify, because the research file does not describe the annotation format. If boxes are yaw-only, we fall back to frames fitted by principal component analysis to each instance's points, combined with the same symmetry-aware minimum. We then report results separately on the subset where the fitted frame is stable across the dataset's rescans of the same scene.
Measuring dominance. We measure dominance before and after every intervention, using three quantities:
- the drop in each metric when one branch's features are zeroed at test time;
- linear-probe predicate accuracy on each frozen branch;
- the overfitting-to-generalization ratio of Wang et al., tracked per branch during training.
We also fit a category-prior baseline that predicts the predicate from (subject class, object class) training frequencies alone. Agreement between the fused model and this prior serves as a direct shortcut index.
Pose-dependent predicate split. We fix in advance which predicates count as pose-dependent, using training labels only. For each of the 26 predicates we estimate how much ground-truth relative rotation adds information about the predicate beyond the class pair. The top third becomes the "pose-dependent" split. This keeps the main evaluation from being chosen after seeing results.
Baselines
Naive reference points. The first is the fused model trained with plain summed losses and no balancing. This is the configuration in which we expect dominance to appear. Next to it we report the geometry-only and image-only versions of the same network and the category-prior predictor. Together these show whether naive fusion even beats its best single input, the pattern Wang et al. found on Kinetics, where the joint model reached 71.7% against 72.6% for RGB alone.
Rebalancing methods. These are the strongest competitors on the specific question of dominance, because they are the published remedies for it.
- G-Blend: we attach unimodal predicate heads to each branch and reweight them by per-branch overfitting-to-generalization ratio.
- MMPareto: applied to the same unimodal heads, with its conflict test and magnitude rescaling. It reports 75.13% on CREMA-D against 71.10% for uniform weighting and 72.01% for G-Blend.
- Modality dropout: we drop the image branch at a tuned rate.
All three methods and our rotation head get the same backbone, data, number of epochs, and a matched hyperparameter search of 20 trials each on the validation split. Beating them means at least two things on 3DSSG PredCls and SGCls, both by margins larger than the spread across three random seeds. First, higher mean recall on the pose-dependent split. Second, a larger drop when the geometry branch is removed at test time, which is evidence that geometry is actually being used.
Strongest published scene graph systems. Absolute performance is anchored to published results.
- VL-SAT reports overall triplet mA@50 rising from 58.37% to 65.09% over SGFN.
- GMPSSG reports 43.0% triplet accuracy in the 20-object-class, 8-predicate setting, and 29.6% mean predicate recall in the 160-class, 26-predicate setting.
- LEO reports PredCls ngcR@20 of 73.3% on top of 3DHetSGP.
We do not expect to beat these systems by adding a head to a simpler backbone. The fair test is to add our rotation head to the VL-SAT training recipe with image features kept at test time, and separately to LEO's line-graph model. A gain over each system's published number under its own protocol counts as transfer.
Ablations that isolate the key ingredient. Four controls isolate what the rotation head contributes:
- Representation: the same head with quaternions under an antipodal-invariant loss, and with yaw-only (cos, sin).
- Target versus input: feeding the target relative rotation as an edge input feature instead of predicting it.
- Rotation versus any regression: a shuffled-target control that keeps the extra loss but destroys its content.
- Branch placement: the rotation head attached to the fused feature instead of the geometry-only feature. If this works equally well, the "give the weak branch its own task" mechanism is not what is operating.
Why not just…?
Why not just apply G-Blend or MMPareto to a fused scene graph model? This is the main threat, and the experiment is designed around it. Both methods can only push the geometry branch toward the same predicate labels that category co-occurrence already explains. We therefore expect them to raise geometry-only probe accuracy mostly on easy, pose-independent predicates. MMPareto's own analysis assumes the multimodal objective is the easier one, which may hold here or may not. If either method matches our pose-dependent recall and our geometry reliance, our claim fails, and we will report it that way. The two methods can also be combined with the rotation head, so we run those combinations to see whether the effects add.
Why not just feed relative pose as an input feature? Supplying ground-truth or estimated relative rotation as an edge input gives the predicate classifier the information directly. It needs an upstream pose estimator at test time, and it does not train the geometry encoder at all. The target-versus-input ablation answers this. If input features with an estimated pose match the auxiliary target, the simpler design wins.
Why not just use yaw, since indoor objects are mostly upright? This is a real possibility. Most 3RScan furniture is gravity-aligned, and a (cos, sin) yaw encoding is already continuous. The benefit of full 6D should then appear only on tilted, lying, or hanging configurations, so we report the yaw-versus-6D gap separately on edges whose target rotation is more than 20° off the vertical axis. If the gap is negligible there too, we adopt yaw and say so. The broader claim about which task the weak branch learns does not depend on the exact representation. The quaternion comparison is retained because Zhou et al. show quaternion regression has large worst-case errors, and we want to confirm this in the relational setting rather than assume it.
Why not just discard images at test time, as VL-SAT does? That avoids dominance by never fusing, but it gives up appearance cues at inference. Our image-only and geometry-only references, together with VL-SAT under a matched backbone, show whether fusing at test time is worth having once dominance is controlled.
Proposed Experiments
All experiments use 3DSSG on 3RScan in both standard settings, with PredCls and SGCls under R@K, mR@K, and mA@K. Every result is also reported on the pre-registered split of pose-dependent and pose-independent predicates, together with geodesic error on edge rotations.
First decisive check: interaction test. This is a 2×2 design crossing fused versus geometry-only with and without the rotation head, three seeds per cell. The claim requires a positive interaction. The head must raise pose-dependent mR@K more in the fused model than in the geometry-only model, and the gain must come with a larger drop when the geometry branch is removed.
Second decisive check: shortcut reduction. The fused model with the rotation head should agree less with the category-prior predictor on pose-dependent edges, while its accuracy on those same edges rises.
Third decisive check: robustness to degraded images. We darken frames, drop views, and perturb camera-to-scan calibration by small rotations at test time. This follows the stress-test spirit of MEFormer, which reports keeping 42.5% mAP when LiDAR is removed, compared with 0.2% for BEVFusion. A model whose geometry branch was actually trained should degrade more gracefully than the rebalancing baselines.
Transfer test. We add the head to the VL-SAT recipe and to LEO and compare against their published numbers quoted above.
Out-of-domain check. ScanNet appears in VL-SAT's out-of-domain evaluation, but the file reports no number for it. We would evaluate there without a published reference figure; no reported result was found on Lacuna for ScanNet scene graph accuracy.
References
- On the Continuity of Rotation Representations in Neural Networks
- Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
- VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud
- CLIP-Driven Open-Vocabulary 3D Scene Graph Generation via Cross-Modality Contrastive Learning
- Learning Gaussian Mixture-distributed Prototypes for 3D Scene Graph Generation from RGB-D Sequences
- What Makes Training Multi-Modal Classification Networks Hard?
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance
- ModDrop: adaptive multi-modal gesture recognition
- Robust Multimodal 3D Object Detection via Modality-Agnostic Decoding and Proximity-based Modality Ensemble
- A Planar-Symmetric SO(3) Representation for Learning Grasp Detection
- PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
- Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions
- Edge-Centric Relational Reasoning for 3D Scene Graph Prediction
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation