Disagreement-Guided Pooling for Joint RGB-Depth Discriminators

When a generative model produces an image together with its depth map, the two must agree: a sharp edge in the picture should sit on a real step in depth, not float over a flat wall. The usual way to enforce this is a joint discriminator. This is a critic network that sees the RGB image and the depth map stacked as one multi-channel input and learns to reject pairs whose edges do not line up. Such critics shrink their input step by step, through strided convolutions or pooling, before reaching a verdict. For a chair leg, a cable, a thin shelf edge or a distant lamp post that covers a few pixels, that shrinking blurs away exactly the evidence of misalignment before the critic can use it. The generator is then free to get small structures wrong without penalty.

We propose to change how the joint critic downsamples. Each pooling step weights locations by a learned importance score, as in LIP: Local Importance-Based Pooling. Here the score is also fed an explicit, cheap signal of cross-modal disagreement: places where image gradients and depth gradients disagree. Evidence of misalignment therefore survives downsampling and still reaches the deep layers, which carry enough context to tell an object boundary from surface texture. The central claim is testable before any GAN is trained. A disagreement-guided critic should detect injected misalignments on small structures far better than a uniformly downsampling critic of equal size. That advantage should then appear as better edge-depth alignment on thin structures in generated and inpainted RGB-D data.

What exists and where the gap is

The case for joint critics is established. Independent per-modality critics let depth boundaries drift relative to texture edges and produce ghosting, as observed by Dhamo et al. and Hachiuma et al.. These works motivate the problem. The methods we build on all use uniform strided critics. Consistent Multimodal Generation via A Unified GAN Framework adds a Consistency Discriminator over concatenated RGB, depth and normals to a StyleGAN3 generator. Joint Inpainting of RGB and Depth Images by GAN with a Late Fusion Approach uses 4-channel global and local critics for RGB-D inpainting. Densely Connected Pyramid Dehazing Network judges transmission and radiance jointly. Self-supervised End-to-end ToF Imaging pairs predicted depth with an RGB template in a WGAN critic. None of these four touches the downsampling inside the critic. The Late Fusion paper explicitly reports boundary blurring when holes contain small or dense structures.

The closest neighbor is Edge Guided GANs with Contrastive Learning. It fights the vanishing of small classes such as poles, but it does so in the generator, by amplifying channels by semantic class, and it works on 2D edges rather than depth. Our proposal acts in the critic, needs no class labels, and targets continuous depth.

On the downsampling side, LIP supplies the method: softmax-weighted pooling with a learned importance logit. LIP also supplies a constraint, since its authors report that importance weights can collapse early in training without normalization tricks. LIP was only tested in single-modality classification and detection backbones. MSR-RCNN and the multi-branch building detector motivate the problem from another angle: small targets vanish under ordinary pyramids, and full-resolution branches recover them at real memory cost. Root Gap Correction supplies a cautionary result for the "skip connections fix detail" assumption. Its U-Net still misses sub-pixel gaps because early downsampling erases thin strands before the skips can carry them.

Core Claim and Mechanism

At downsampling stage $s$, the critic pools features $x$ over each local window $\Omega_i$ with weights $\exp(F(x_j, m_j))$, normalized over the window, as in LIP. Here $F$ is a small 1×1 convolutional network. The extra input $m$ is a mismatch map computed from the input pyramid at that stage's scale:

$$m = \big|\, \hat g(I) - \hat g(D) \,\big|,$$

In this map, $\hat g(\cdot)$ is the locally normalized gradient magnitude of the RGB image $I$ or depth map $D$. Normalization makes the two modalities comparable. The map $m$ does not decide realism; it only biases what survives pooling. The deep layers still judge whether a given disagreement is legitimate, for example a painted texture on a flat wall, which real data contains, or an illegitimate one, such as a depth step with no visible edge.

This split is the crux of the design. A hand-crafted mismatch penalty cannot make the legitimacy judgment. A uniform critic can make it only for structures large enough to survive pooling. We stabilize $F$ with the affine instance normalization that LIP reports is needed.

The generator side is held fixed as a U-Net or StyleGAN3-style network, so that any gain is attributable to the critic.

Research Program

The work has three stages.

The first stage is a critic-only sensitivity probe. We take real RGB-D pairs from Stanford2D3D and SceneNet RGB-D, segment structures by their minimum width using connected components of depth discontinuities, and inject controlled misalignments. These are texture edges shifted by $k$ pixels relative to depth, or depth steps removed under intact texture. Each critic variant is trained as a binary real-versus-perturbed classifier. We report detection AUC as a function of structure width and shift size. This probe isolates the hypothesis cheaply: uniform strided critics should lose sensitivity sharply below some width, while the disagreement-guided critic should degrade slowly.

The second stage is joint RGB-D inpainting on SceneNet RGB-D, which has ground truth. We reproduce the Late Fusion setup and swap only the critics.

The third stage is unconditional RGB-D-normal generation on Stanford2D3D. We reproduce the MultimodalGAN setup and swap only the Consistency Discriminator.

Every stage gets a size-stratified alignment metric that the existing benchmarks lack. In inpainting, it is the Chamfer distance between RGB edges and depth discontinuities inside the hole, binned by structure width. In generation, it is the distribution distance between real and generated values of that edge-discontinuity offset, again binned by width. Global metrics such as Depth-FID can hide small-structure failure, so these binned metrics are the primary outcome.

Baselines

The naive reference point is decoupled per-modality critics. For inpainting, this is the "No Fusion" configuration in the Late Fusion paper, which reports $l_1$ depth error 14.3 and RMSE 93.4. It shows what happens with no joint judgment at all.

The strongest existing method for inpainting is the Late Fusion joint RGB-D inpainting network with 4-channel global and local critics, run under its reported protocol. The paper reports $l_1$ depth error 6.83, depth RMSE 52.2 and RGB SSIM 0.987 on SceneNet RGB-D. For generation, the strongest existing method is MultimodalGAN with its Consistency Discriminator, which reports Depth-FID 24.8 on Stanford2D3D. These are the strongest competitors because they are the only systems in the literature that already enforce texture-depth alignment through a joint critic. Our method is a drop-in change to their critic.

For fairness, we reproduce each baseline ourselves on identical splits, hole masks and generator. We match critic parameter count to within 5% and report critic FLOPs. Every variant gets the same hyperparameter search budget over learning rate and adversarial weight, with three seeds. Reproduced numbers are reported alongside the published ones.

Beating the baselines means two things. Thin-structure alignment error must improve in the narrowest width bins, beyond seed variance. The global numbers ($l_1$ depth, RMSE, SSIM, Depth-FID) must not get worse. A gain only in global metrics would not support the claim.

The key ablations isolate the ingredient. The first is plain LIP pooling with a learned logit and no mismatch input, which asks whether adaptive pooling alone suffices. The second uses the mismatch map with a fixed, non-learned weighting, which asks whether learning matters. The third is a uniform critic that receives $m$ as an extra input channel, which asks whether the cue helps because it steers pooling or merely because it is extra information.

Why not just…?

Why not just add a Sobel or Canny edge-alignment loss between RGB and depth? Such a loss penalizes all disagreement, including legitimate texture on flat surfaces, so it pushes the generator to erase texture or invent depth. The literature points the same way. The Late Fusion paper's "Early Fusion" variant, which concatenates inputs and relies on reconstruction loss, reaches $l_1$ depth error 11.0 against 6.83 for joint adversarial evaluation with late fusion. The dehazing network pairs gradient losses with a joint critic rather than replacing the critic. We will still run this loss as a baseline with a tuned weight. If it matches our thin-structure alignment without hurting texture statistics, measured as FID on RGB, the critic machinery is unjustified.

Why not just use a shallow full-resolution patch critic, or a local critic on crops, so nothing is downsampled? This is the real threat. A shallow critic sees small structures at full resolution but has a small receptive field, so it should struggle to decide whether a disagreement is a legitimate texture edge. Full-resolution branches also cost the memory and latency that the multi-branch building detector reports. The sensitivity probe settles this directly. We include a FLOPs-matched shallow patch critic and a crop-based local critic in the style of the Late Fusion setup. If either matches our width-binned AUC and downstream alignment, we will report that pooling design is not the bottleneck.

Why not rely on generator skip connections to preserve detail? Skips help the generator produce detail, but they do nothing to make the critic penalize misaligned detail. Root Gap Correction also shows that skips cannot carry what early downsampling has already destroyed. Our baselines all use skip-equipped generators, so any remaining thin-structure error is attributable to the critic.

Decisive Checks

The idea stands or falls on three outcomes. First, in the sensitivity probe, the disagreement-guided critic must keep detection AUC substantially higher than both the uniform critic and plain LIP for structures a few pixels wide, at matched parameters. Second, on SceneNet inpainting and Stanford2D3D generation, that critic-level sensitivity must translate into lower width-binned alignment error, without regressions in the published global metrics. Third, the mismatch-as-input-channel ablation must underperform mismatch-steered pooling; otherwise the contribution is the cue, not the pooling.

If the probe shows no size-dependent advantage, the hypothesis that critic downsampling drives small-structure misalignment is false. That would be a useful negative result for every joint-critic system above.