To label a 3D scene, a common approach takes masks from a 2D segmentation model run on every camera frame, projects them onto the 3D points, and lets the frames vote. An error such as a shadow mistaken for an edge usually appears in only one view, so agreement across many views filters it out. The weakness is that every frame votes on everything, with equal weight, against the whole scene. A cabinet handle covers a few hundred points. Hundreds of frames that see the handle only as a blur at the edge of the image, or a run of near-identical frames that share the same glare, can outvote the few frames that actually resolve it. Recent part-level systems report that performance collapses on exactly these tiny parts.
We propose to aggregate the votes the way a journal's editorial board handles submissions. An editor sends each decision only to reviewers who are qualified to judge it. The editor discounts reviewers with a poor track record and does not count two reviewers from the same lab as independent. A decision about a sub-question is made within the scope of its parent decision rather than against the whole journal. Applied to multi-view masks, each candidate part is judged only by views that see its parent object clearly. Each view's weight comes from how well it agreed with the consensus on the parent-level decision. Near-duplicate views are merged into one vote. Child-part scores are normalized within the parent region, so the bulk of walls and tabletops cannot dilute them. The 2D models stay frozen and no training is involved. Only the aggregation rule changes, so any gain can be attributed to it.
Core Claim
Two things suppress small parts in current training-free 3D segmentation, and both live in the aggregation step rather than in the 2D models. The first is unscoped normalization: votes for a part compete with the scene, not with its parent. The second is unweighted, correlated voting: every frame counts the same, and adjacent frames count as independent evidence. A hierarchical aggregator fixes both by scoping votes to the parent region, weighting views by their reliability on parent-level decisions, and deduplicating correlated views. With identical 2D masks, it should improve small-part segmentation on SceneFun3D and on part-annotated ScanNet++ and MultiScan, and it should extend to nested queries ("the button on the remote on the coffee table"). The gain should be concentrated in the smallest size bins.
Where This Sits
The motivating evidence comes from part-level 3D work. PS3 builds superpoints (small groups of 3D points treated as one unit) from Semantic-SAM masks, tracks masks across frames with SAM2, and merges superpoints by breadth-first search over consensus rates. Its authors state that it fails on tiny parts such as wall hooks and knobs. ScanNet++ documents the same collapse for small components. We keep PS3's masklets and pivot-view idea but replace its single global merging threshold with scoped, weighted decisions.
The closest existing work is Fun3DU. Its pipeline works in four steps. An LLM splits an instruction into a parent object and a functional child. OWLv2 finds the parent, and a selector keeps the 50 best views. Molmo and SAM segment the child in those views. A flat multi-view heatmap lifts the result to 3D. Fun3DU therefore already does editorial assignment at the view-selection stage. Its hierarchy is fixed at two levels, and the final vote is uniform and unscoped. Our contribution is the aggregation rule itself. That rule is recursive to arbitrary depth, and its view weights carry over from parent decisions to child decisions.
Flat consensus methods supply the reference behavior. MaskClustering treats every 2D mask as a peer node and merges masks by a view-consensus rate. SAI3D votes over superpoint affinities with progressive thresholds whose cost grows linearly in the number of frames. Weighting by visibility, as in iSegMan's use of transmittance, discounts occluded points but does not estimate whether a view's segmenter is reliable.
From the governance side, Voting or Consensus? supplies the constraint behind our deduplication step: majority voting fails when voters share systematic biases. Embodied LLM Agents Learn to Cooperate in Organized Teams shows that designated hierarchy cuts redundant communication among agents. It does so only in a symbolic household simulator, so it motivates our approach without validating it for perception. The IEEE editorial-board pages in the file (e.g.) describe the tiered roles and serve only as structural inspiration. We do not claim they supply evidence.
Research Program
The aggregator works over a tree of regions. The root is the scene. A parent node is a 3D region, such as a cabinet obtained by consensus at the level above. Its children are candidate part masks lifted from 2D. Building the aggregator involves three components.
The first component is assignment. For a parent region $P$, a view is qualified only if $P$ projects into it with enough pixel area and lies near the center of the frame. This generalizes Fun3DU's selector to every level of the tree.
The second component is conflict-of-interest deduplication. Qualified views are clustered by camera pose and image overlap, and each cluster $c$ casts a single averaged vote. This targets the correlated-glare failure.
The third component is the reviewer's track record. Each cluster's weight $w_c$ is its agreement, measured by intersection over union, with the leave-one-out consensus mask of $P$ at the parent level. A cluster that segmented the cabinet badly is trusted less on the handle.
The score of point $p$ inside $P$ for a child part is
$$ s(p) = \frac{\sum_c w_c\, m_c(p)}{\sum_c w_c\, v_c(p)}, $$
where $m_c(p) \in [0,1]$ is cluster $c$'s mask vote at $p$, and $v_c(p)$ is its visibility of $p$ (depth-tested, as in iSegMan-style weighting). The sums run only over qualified clusters. Because the denominator counts only views that actually see $p$ inside $P$, a small part is judged against its parent, not against the scene.
A child region is accepted by thresholding $s$ inside $P$. It then becomes a parent for the next level, which is how nested queries are handled. When a child's score is split, meaning its high- and low-weight clusters disagree, the system re-assigns the decision: it adds the next-best unqualified views or zoomed crops and re-queries the 2D segmenter on those crops only. This is the "appeal" step. It is the only part that adds 2D model calls, and we measure its cost.
The tree comes from the existing parent-child parse for SceneFun3D instructions, extended to multiple levels by the same LLM prompt. On ScanNet++ and MultiScan it comes from the object-part pairs those datasets already annotate. Because every component is a fixed rule, each one can be switched off independently.
Baselines
The naive reference is uniform voting. It uses the same 2D masks, projected and averaged over all views that see each point, with no scoping, no weights and no deduplication. This shows how much of any gain comes from aggregation at all.
On SceneFun3D, the strongest competitor is Fun3DU, which reports 15.2 mIoU on SceneFun3D according to Functionality Understanding and Segmentation in 3D Scenes. It is the strongest competitor because it already has parent-child assignment. Any gain over it must therefore come from scoped, weighted aggregation, not from merely having a hierarchy. We will rerun Fun3DU and replace only its final heatmap step with our aggregator. The LLM, OWLv2, Molmo and SAM outputs stay identical and cached. Thresholds for both methods are tuned on the same held-out scenes with the same number of trials. Beating it means a higher mIoU and AP25 on the evaluation split, with the gain concentrated in the smallest third of functional elements.
On part segmentation, the strongest competitor is PS3. It reports 33.2 AP50 and 41.9 AP25 on the ScanNet++ part subset, and 8.6 AP, 20.1 AP50 and 32.4 AP25 on MultiScan (PS3). We keep PS3's superpoints and masklets and replace only the breadth-first merging with scoped weighted scoring. Beating it means higher AP50 on both datasets, with recall on tiny parts reported separately.
We also run MaskClustering's view-consensus rate as an alternative flat aggregator over the same masks. Because the hierarchy is the proposal's key ingredient, we ablate each component alone. The three single-component variants are scoping-only, reliability weights only, and deduplication only. We also test the full aggregator with uniform weights, and with weights drawn at random from the same distribution. Runtime and the number of 2D model calls are reported for every variant, since SAI3D-style linear cost is already a known bottleneck.
Why not just…?
Why not just normalize votes by object size, or lower the threshold for small parts? This is the most serious threat. Scoping is itself a form of size normalization, and it may account for most of the gain. A scene-level rescaling by projected area, applied without any tree, is therefore included as a baseline, and the scoping-only ablation tests it directly. If scoping-only matches the full system, we will report that simple parent-relative normalization is the finding and that reliability weighting adds nothing. That would still be a useful, cheap result. We expect weights and deduplication to matter mainly where adjacent views share errors, so we will report gains separately on scenes with high and low specular content.
Why not just learn per-view reliability with a standard unsupervised annotator-agreement model over all votes? Flat reliability estimation treats each view as having one accuracy across the whole scene. A frame can be excellent on a large cabinet and useless on its handle because of resolution. Our weights are inherited within a parent region and gated by qualification. We test this directly with a variant that estimates a single global reliability per view and uses it everywhere. If that variant performs as well, the hierarchical part of the claim fails.
Why not just use Fun3DU with a better 2D part segmenter? A better segmenter helps every aggregator equally. Our comparisons hold the masks fixed precisely to separate the two effects. As a check, we swap in the stronger mask source PS3 uses and test whether the aggregation gain persists.
Why not use an LLM critic that audits failures and rewrites the pipeline, as in the Criticize-Reflect loop? In the organized-teams study (Guo et al.), dynamic leadership raised communication cost substantially. That evidence comes from symbolic tasks with no geometry. Our appeal step is a cheap, geometric version of re-assignment. We leave language-model auditing out until the rule-based version is understood.
Proposed Experiments
The decisive test uses SceneFun3D, which has 14,867 functional elements, together with the ScanNet++ and MultiScan part subsets. Results are stratified by part size measured in 3D points. The prediction is specific: relative to uniform voting and to Fun3DU's heatmap, the gain should grow as parts get smaller, and it should be near zero for large parts. A flat improvement across all sizes would suggest that the method is merely re-tuning thresholds.
The second test targets correlated views. On scenes ranked by specular content and by frame density, deduplication should help more where frames are densest. We will also inject synthetic correlated errors, pasting the same false mask into $k$ consecutive frames. The aggregator's error should stay roughly flat in $k$, while uniform voting's error should rise.
The third test covers nested queries. No benchmark for them was found in the file. We will build a subset from SceneFun3D instructions whose target has a two-level context, compose those levels into three-level queries, and verify the annotations by hand. On this subset we compare recursive aggregation against Fun3DU with the intermediate level collapsed.
Finally, we report the cost per scene: wall-clock time, number of 2D calls added by the appeal step, and points processed. The question is whether scoping reduces work by evaluating each child only within its parent's qualified views.
References
- PS3: Part Level Instance Segmentation in 3D
- ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes
- Functionality Understanding and Segmentation in 3D Scenes
- MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation
- SAI3D: Segment any Instance in 3D Scenes
- iSegMan: Interactive Segment-and-Manipulate 3D Gaussians
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- Embodied LLM Agents Learn to Cooperate in Organized Teams
- IEEE Transactions on Information Theory publication information