A compact language model that answers questions about a 3D room cannot read every relationship between every pair of objects, because the number of pairs grows with the square of the object count and each edge costs tokens. The current fix is a fixed rule: keep each object's two nearest neighbours and discard the rest. A second problem compounds the first. These models often do not look at the scene at all. A recent audit found that text-only "blind" models match 3D language models on standard benchmarks. On a filtered benchmark that removes questions answerable from text alone, the best models lose more than 60% of their accuracy. We propose to make the pruning step solve both problems together. A small selector, conditioned on the question, chooses a fixed budget of scene-graph edges to show the language model. It is trained with reinforcement learning on a reward that pays for two things. The chosen edges must be sufficient, so the model answers correctly from them. They must also be necessary, so the answer falls apart when those edges are swapped for others.
The bet is that "necessity" is the missing training signal. A selector rewarded only for correct answers can learn to pass along any edges while the language model answers from linguistic priors. A selector rewarded for edges whose removal hurts the answer can only collect reward when the answer actually depends on the geometry. If this works, a compact model would fit within the same token budget as today's fixed-neighbour pruning. Its answers would also depend on the scene in a way we can check by ablating the selected edges.
Core Claim
Under a fixed edge-token budget, a question-conditioned edge selector trained on a sufficiency-plus-necessity reward will do three things better than fixed nearest-neighbour pruning, cheap question-matching heuristics, and loss reweighting against blind-model shortcuts. It will raise accuracy on shortcut-filtered 3D question answering, raise consistency under viewpoint rotation, and make the model's answers measurably dependent on the edges it was shown. Removing the necessity term should erase most of the gain on the filtered benchmark while leaving accuracy on the unfiltered benchmark roughly intact. This is the signature that the term, and not budget or learning capacity, does the work.
What Prior Work Establishes
Three lines of work motivate the proposal.
3DGraphLLM feeds 3D semantic scene graphs into a language model, with object nodes and relation features from a pretrained relation extractor mapped into token space. It improves grounding over non-relational baselines (+6.4% [email protected] on ScanRefer, +7.5% [email protected] on Multi3DRefer) and gives up to five times faster grounding inference. Its pruning, however, is fixed: k=2 nearest neighbours plus non-maximum suppression and distance filters. The authors chose this cutoff for resource reasons and name edge-token reduction as an open problem. We keep its architecture and token interface and replace only the rule that decides which edges enter the context.
Selective Quad Attention (SQUAT) shows in 2D scene-graph generation that a learned edge-selection module beats keeping all candidates. Removing it costs nearly 9% mR@100 on Visual Genome. Together with Jung et al., it argues that soft attention lets thousands of small background weights accumulate into noise. SQUAT supplies the case for learned hard selection. It differs from our setting in three ways: it keeps a global top-ρ% regardless of any query, it is trained with supervised relatedness labels, and it never feeds a language model.
Do 3D LLMs Really Understand 3D Spatial Relationships? supplies the diagnosis and the benchmark. It introduces Real-3DQA, which filters out questions a blind model solves, and a Viewpoint Rotation Score that checks answer consistency across four observer rotations. It also introduces 3D-aware Reweighted Fine-Tuning (3DR-FT), which upweights training examples the blind model finds surprising. 3DR-FT changes which examples the model learns from but leaves its input dense and unstructured. Our approach changes what the model sees, and penalizes answers that do not depend on it. The two are complementary, and we test whether they stack.
Two further papers supply the training method and constrain the design. Embodied-R1 and AutoTriton show that compact models (3B and 8B) specialized with Group Relative Policy Optimization (GRPO) on checkable rewards beat supervised fine-tuning. GRPO is a reinforcement-learning method that compares several sampled outputs for the same input and pushes toward the better ones. Embodied-R1 attributes its gain to the "multi-solution dilemma": supervised targets penalize valid alternatives. Edge selection has the same structure, since many different subgraphs can support one answer and no edge-level labels exist for question answering.
Specializing Smaller Language Models towards Multi-Step Reasoning constrains us in a different way. It shows that specializing a compact model costs general ability, measured as sharp BigBench Hard drops. We therefore keep the language model's weights mostly frozen during the reinforcement-learning stage.
Research Program
We build on the 3DGraphLLM pipeline and its compact backbone. The pipeline uses ScanNet and 3RScan scenes, a relation feature for each object pair, and a two-stage recipe that trains on ground-truth segmentation and then on predicted segmentation.
Candidate pool. For each scene and question, we build a generous candidate pool. It contains each object's ten nearest neighbours plus all pairs involving objects whose text-image similarity to question words exceeds a threshold. This deliberately admits long-range pairs, such as an observer-referenced object across the room, which k=2 discards.
Selector. The selector is a small network that scores each candidate edge. Its inputs are the pair's relation feature, relative geometry (distance and direction in the situation frame given by the question), and a pooled embedding of the question and situation text. It samples a subgraph $S$ of exactly $B$ edges without replacement. $B$ is matched to the token cost of 3DGraphLLM's k=2 graph, so every comparison runs at equal context length.
Reward. For a question $q$ with answer $y$, the language model's probability of the correct answer given the selected edges is $p(y \mid S, q)$. We also form a replacement subgraph $\tilde{S}$ by swapping every edge in $S$ for a different edge drawn from the candidate pool, keeping the count $B$ and the node set's spatial spread matched. The reward is
$$R(S) = p(y \mid S, q) - \alpha \, p(y \mid \tilde{S}, q) - \beta \, \max\big(0,\; p(y \mid \tilde{S}, q) - p_{\text{blind}}(y \mid q)\big),$$
The coefficients $\alpha$ and $\beta$ are fixed weights. $p_{\text{blind}}$ is the probability from a text-only copy of the model.
The first term pays for sufficiency. The second pays for necessity: if the model answers equally well from the wrong edges, the selected ones were not doing anything. The third term stops a degenerate fix in which the model becomes uniformly unconfident whenever edges change. It only penalizes replacement-graph confidence above what text alone would give.
Training. We sample a group of subgraphs per question and update the selector with GRPO, as in Embodied-R1 and AutoTriton. Training has three stages. First, the language model is fine-tuned with supervision on the original 3DGraphLLM recipe, but with randomly budgeted subgraphs drawn from the pool, so it learns to read variable edge sets. Second, the language model is frozen while the selector is trained with GRPO. Third, an optional short phase alternates the selector with low-rank adapter updates to the language model, using the same reward. Freezing in the second stage keeps the specialization tax measurable and small.
Analysis. We measure what the selector learns. One check is the fraction of selected edges outside the k=2 neighbourhood, broken down by question type (direction, counting, reachability). Another is how selections shift when only the situation text changes, with the scene held constant.
Baselines
Naive reference: 3DGraphLLM unchanged. The do-nothing reference is 3DGraphLLM exactly as published, with k=2 pruning, retrained by us on the same splits with the same backbone. We find no reported 3DGraphLLM number on Real-3DQA in Lacuna, and only the qualitative claim of improvement on SQA3D, so we will measure it ourselves. Beating it means higher Real-3DQA exact match and Viewpoint Rotation Score at identical edge-token count.
Larger fixed budget. The same model with k raised until the edge tokens are three times the budget. This tells us how much of any gain is just "more edges". We would count our method as beating it if it matches or exceeds this baseline on Real-3DQA at one third of the edge tokens.
Strongest existing anti-shortcut method: 3DR-FT. We apply 3DR-FT to the k=2 3DGraphLLM model with the same data and the same number of gradient steps as our full pipeline. The comparison point from the literature: 3DR-FT raised LEO on Real-3DQA from 19.1% to 29.3%, reported in Ma et al.. The same paper also reports LEO's untuned Real-3DQA score as 14.3% exact match, so the 19.1% starting point presumably reflects a different training setup, which is one more reason we re-run everything ourselves. The same paper reports Viewpoint Rotation Scores of 6–18% for most models. Beating 3DR-FT means a higher Real-3DQA score and a higher Viewpoint Rotation Score than 3DR-FT on our backbone, with the same tuning sweep size given to both.
Supervised edge selector. A SQUAT-style selector, following SQUAT, is trained with cross-entropy on 3RScan ground-truth relation labels to keep the top-B edges, but without question conditioning. This isolates whether learned relatedness alone, the thing SQUAT showed helps, explains the result.
Key ablations. Our own method trained with the sufficiency term only ($\alpha=\beta=0$) is the ablation that isolates the proposal's central ingredient. A random-B-edges-from-the-pool control establishes the floor.
All systems share the backbone, the segmentation inputs, the edge budget (except the larger-budget baseline), and a tuning budget of an equal number of configurations.
Why not just…?
Why not just feed every edge, or use a much bigger model with a long context? Token cost is the binding constraint that 3DGraphLLM reports, and SQUAT and Jung et al. show that soft attention over many distractor pairs adds noise rather than ignoring it. Still, "more edges" is the obvious explanation for any gain. The larger-fixed-budget baseline tests it directly. If tripling k matches our selector on Real-3DQA, the selector buys only efficiency, and we will report it that way.
Why not just pick edges around objects named in the question? This is the most serious threat, because it is query-conditioned, free, and probably captures much of what a learned selector would. We address it by making it part of our candidate pool and running it as a standalone baseline: take the question-matched nodes plus their nearest neighbours up to budget B. Our claim does not rest on query conditioning itself. It rests on the necessity reward making the answer depend on the edges. The prediction is therefore specific. The heuristic should do well on unfiltered SQA3D but gain little on Real-3DQA and the Viewpoint Rotation Score, because nothing stops the language model from ignoring the edges it receives. If the heuristic matches us on Real-3DQA, the necessity mechanism is not needed and the paper becomes a negative result about it.
Why not just apply 3DR-FT? 3DR-FT already gave the largest reported gain on Real-3DQA (19.1% to 29.3% for LEO), and it needs no new module. But reweighting changes which examples matter, not what evidence the model reads or whether its answers depend on that evidence. We run 3DR-FT alone, our method alone, and both combined. If 3DR-FT alone equals the combination, our contribution reduces to efficiency.
Why not supervise the selector with ground-truth relations instead of reinforcement learning? Relation labels say which pairs are related, not which pairs a given question needs. And several subgraphs can support the same answer, the multi-solution problem Embodied-R1 found to hurt supervised training. The supervised SQUAT-style baseline measures how much this matters here.
Proposed Experiments
Datasets and metrics. Training uses SQA3D and ScanQA on ScanNet scenes, with 3RScan for the ground-truth-segmentation stage. Primary evaluation is Real-3DQA exact match and Viewpoint Rotation Score. Secondary evaluation is SQA3D exact match. SQA3D reports a blind baseline of 43.65% and 47.20% for ScanQA with auxiliary tasks, which frames how much of SQA3D is solvable without vision. To confirm that grounding does not regress under the reduced budget, we evaluate ScanRefer [email protected] and Multi3DRefer [email protected], the tasks where 3DGraphLLM reports its relational gains.
Faithfulness test. We add an edge-necessity metric computed at test time: the accuracy drop when the selected edges are replaced by budget-matched random pool edges. A model that reads its graph should drop sharply. A shortcut model should not.
Decisive checks. The first decisive check is the ablation contrast. If the necessity term is responsible, the sufficiency-only selector should roughly match the full method on SQA3D but fall clearly behind on Real-3DQA, the Viewpoint Rotation Score, and the edge-necessity drop. Equal results across the board would falsify the core claim.
The second check is robustness to upstream noise. We repeat all evaluations on predicted rather than ground-truth segmentations. Learned selectors may suffer more than fixed rules from broken object proposals, which the file flags as an open weakness of both relation scoring and 3DGraphLLM.
The third check is the specialization tax. We compare the frozen-language-model variant against the adapter-tuned variant on a text-only general reasoning suite such as BigBench Hard. If adapter tuning buys accuracy on Real-3DQA at a large general cost, the frozen variant is the result we recommend.
References
- 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding
- Devil's on the Edges: Selective Quad Attention for Scene Graph Generation
- Do 3D Large Language Models Really Understand 3D Spatial Relationships?
- Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation
- AUTOTRITON: Automatic Triton Programming with Reinforcement Learning in LLMs
- Specializing Smaller Language Models towards Multi-Step Reasoning
- SQA3D: Situated Question Answering in 3D Scenes