Recovering a 3D scene and the camera poses that observed it is a chicken-and-egg problem: if the initial pose guesses are wrong, the reconstructed geometry comes out wrong too, and vice versa. Modern systems solve both at once by differentiable rendering — rendering a candidate scene from candidate cameras and using the pixel error to update both — but the dominant signal, photometric RGB error, is confounded by lighting, albedo, and a well-documented failure mode where the scene "cheats" by deforming itself to fit bad poses. At the same time, a separate literature on physical adversarial camouflage paints 3D objects with multi-view-optimized textures that reliably blind detectors and, plausibly, these analysis-by-synthesis loops themselves. Nobody has asked the obvious question at the intersection: is a rendering-based pose estimator that also trusts object silhouettes — the pure 2D outline of an object, which carries geometry and orientation but is blind to paint — intrinsically resistant to texture camouflage, and where does that resistance break?
This proposal builds a joint pose–geometry optimizer whose geometric anchor is a differentiable silhouette loss with exact analytical boundary derivatives, and then attacks it. The core bet is twofold. First, silhouettes constrain exactly what photometric error cannot: the outer boundary of the object, independently of surface appearance, so they should widen the basin of attraction of joint refinement and suppress the degenerate "needle-Gaussian" solutions that pure photometric loss invites. Second, because every existing physical camouflage method operates on texture and objectness — not on outlines — a silhouette-anchored loop should survive attacks that destroy photometric-only refinement, while a deliberately optimized geometric perturbation should break it, exposing and quantifying a genuinely new attack surface.
Core Claim
The claim has a constructive part and a robustness part. Constructively: adding a silhouette-consistency term, whose gradients are computed as exact boundary integrals rather than blurred edge approximations, to a joint camera-pose/scene-geometry optimizer will (a) extend the range of initial pose errors over which the optimizer converges, and (b) reduce degenerate geometry, compared to the same optimizer with photometric loss plus a heuristic shape regularizer. In terms of robustness: under multi-view physical camouflage of the kind that collapses detector performance, the silhouette-anchored loop will retain pose accuracy where photometric-only refinement drifts, and the attack that defeats it must perturb geometry, not texture — a strictly more detectable and expensive class of attack.
Concretely, we optimize
$$\mathcal{L}(g, \Theta, V) \;=\; \mathcal{L}_{\text{photo}}(g, \Theta) \;+\; \lambda \, \mathcal{L}_{\text{sil}}(g, V),$$
where $g$ is the camera extrinsics parameterized on the $\mathfrak{se}(3)$ tangent space so rotations stay singularity-free, $\Theta$ are background scene parameters (3D Gaussians), $V$ are the vertices of a mesh prior for the tracked object, $\mathcal{L}_{\text{photo}}$ is standard RGB reprojection error, and $\mathcal{L}_{\text{sil}}$ is the mismatch between the rendered object outline and a 2D segmentation mask. The silhouette term is the transfer of an existing exact technique to a new regime: Analytical Derivatives for Differentiable Renderer: 3D Pose Estimation by Silhouette Consistency computes closed-form derivatives of silhouette error by integrating over pixel areas with Liang-Barsky boundary clipping, but applied it only to a fixed low-dimensional SMPL body model in a benign setting. We attach that derivative machinery to camera poses and free mesh vertices inside a joint loop, on the reasoning that silhouettes depend only on boundary geometry and camera extrinsics, so their gradients cannot be corrupted by anything painted inside the boundary.
Research Program
Three things get built. The first is the anchored optimizer itself. The forward pass renders the background with a standard (non-differentiable, hardware-style) splatting or Z-buffer step for speed, while the backward pass routes silhouette-boundary gradients through the analytical clipping formulation and photometric gradients through the existing differentiable splatting kernels — the same decoupling philosophy as Differentiable Surface Rendering via Non-Differentiable Sampling, which keeps the fast discrete forward pass and repairs only the backward pass. The object is represented by a mesh with an optional deformation mode, which is what makes this joint pose–geometry optimization rather than pose-only refinement; End-to-End 6-DoF Object Pose Estimation Through Differentiable Rasterization and DSC-PoseNet: Learning 6DoF Object Pose Estimation via Dual-Scale Consistency both refine pose against silhouettes but freeze the CAD geometry as an immutable prior, and neither considers an adversary.
The second component is the attack suite. We take camouflage textures produced by established multi-view adversarial pipelines — the triplanar-mapped textures of ACTIVE: Towards Highly Transferable 3D Physical Camouflage for Universal and Robust Vehicle Evasion and the gradient-calibrated textures of Physical Adversarial Camouflage Through Gradient Calibration and Regularization — and apply them to the tracked object in multi-view sequences, measuring what they do to the refinement loop rather than to a detector. This is the gap: those two works sample camera views as fixed external parameters and never pose the camera as a refineable variable, while Adversarial Attacks on Monocular Pose Estimation attacks only feed-forward pose networks with per-image 2D noise, never a test-time analysis-by-synthesis loop and never a 3D-consistent perturbation. We also build an adaptive adversary in the spirit of MeshAdv: Adversarial Meshes for Visual Recognition, which deforms vertices but assumes known viewpoints and never targets a pose optimizer: our variant optimizes a small vertex deformation specifically to spoof the silhouette anchor and bias the estimated pose, which tells us how much geometric change the defense costs an attacker to defeat.
The third component is an analysis of multi-view gradient conflict in pose space. Physical Adversarial Camouflage Through Gradient Calibration and Regularization showed that texture gradients from opposing viewpoints cancel each other and fixed it with Gram-Schmidt decorrelation at roughly 22% per-iteration overhead; multi-view silhouette and pose optimizers such as HeroGS: Hierarchical Guidance for Robust 3D Gaussian Splatting under Sparse Views simply average gradients across views. We will measure whether per-view $\mathfrak{se}(3)$ gradients conflict (via pairwise cosine similarity) and whether decorrelation widens the convergence basin, treating this as a measured diagnostic rather than a headline claim.
Baselines
The "do nothing" reference point is the classical decoupled pipeline: run structure-from-motion (COLMAP-style) for poses, freeze them, and optimize the scene with fixed-pose 3D Gaussian Splatting. This is the default practice and the natural null hypothesis; it requires no tuning budget beyond its own defaults, and its expected failure mode — catastrophic error accumulation when initial poses are wrong — is documented in Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization. For the adversarial evaluation the corresponding reference point is an unattacked run of each method, so every reported degradation is relative to the method's own clean performance.
The strongest existing competitor is the photometric-only joint optimizer of Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization, which embeds analytical pose gradients directly in the splatting CUDA kernels, reports a PSNR of 31.24 dB on Tanks and Temples (versus 26.34 dB for Nope-NeRF), converges from initial rotation errors up to 15 degrees, and runs its LLFF optimization in 10 minutes versus 300 minutes for BARF. We will reimplement its loop faithfully (same Gaussian initialization, same iteration count, same compute budget per scene) and give our method exactly the same settings, tuning $\lambda$ only on a held-out validation split of scenes with equal tuning budget for every competitor's own hyperparameters — including their anisotropy loss weight. Beating it means three things, in order of importance: convergence (final pose error within tolerance) at initial rotation noise strictly beyond its 15-degree limit, higher PSNR at equal noise, and materially smaller pose-error increase under camouflage.
The key ablation replaces the silhouette anchor with the heuristic shape regularizer that the strongest baseline already uses — the anisotropy loss penalizing Gaussian eccentricity from Look Gauss, No Pose — keeping everything else identical. This isolates whether the gains come from the silhouette information or merely from having any geometric prior; if a well-tuned anisotropy loss matches the anchor, the constructive claim collapses to "another regularizer." A second ablation swaps the analytical boundary derivatives for the soft membership-function silhouette of Neural 3D Mesh Renderer, isolating whether exactness matters at our resolutions. On the attack side, the baseline adversaries are the camouflage textures of Physical Adversarial Camouflage Through Gradient Calibration and Regularization (which drove DETR average precision from 97.56% to 6.28%) and per-view projected-gradient-descent pixel attacks on rendered frames, the strongest cheap adversary a reviewer would demand.
Why not just…?
Why not just run COLMAP first and freeze the poses? Because the failure this proposal targets is precisely the regime where that pipeline breaks: Look Gauss, No Pose documents that classical SfM is slow and frequently fails outright under repetitive texture, low texture, or motion blur, and that pose errors propagate into severe reconstruction artifacts. Its joint refinement reaching 31.24 dB on Tanks and Temples against 26.34 dB for the prior pose-free method is direct evidence that joint beats decoupled when initialization is imperfect — and our question is whether the silhouette anchor pushes the tolerable-initialization frontier further out. The decoupled pipeline stays in as the null baseline, so if it wins at small noise levels, that will show honestly.
Why not just keep the photometric loss and add a stronger shape regularizer? This is the most serious alternative, and we do not argue around it — we run it. The anisotropy loss in Look Gauss, No Pose already suppresses the needle-Gaussian degeneracy, so the burden is on us to show the silhouette anchor adds information, not just penalty. Our argument is mechanism-level: a regularizer penalizes the symptom (eccentric splats) while a silhouette term adds an independent constraint tied to boundary geometry and camera pose, which is exactly the quantity that pose errors corrupt; the same logic is why DSC-PoseNet reached a 54.7% ADD score on LINEMOD using only 2D-box-derived silhouette supervision where RGB-only baselines like AAE (32.6%) and DPOD (40.5%) fell short. But the decisive word belongs to the head-to-head ablation at matched tuning budget, including the regime the regularizer was never designed for: camouflage-corrupted photometric signal, where a penalty on splat shape cannot restore lost information.
Why not just use the soft alpha channel of the splat renderer as the silhouette, instead of analytical boundary integrals? Possibly one should, and we treat this as an open empirical question rather than a foregone conclusion. The precedent is that exactness mattered where it was tested: on UP-3D, the analytical derivatives of Analytical Derivatives for Differentiable Renderer cut per-vertex error from 172.2 mm (soft N3MR) to 142.8 mm and were 22% faster per iteration at 128×128. Whether that gap persists at the higher resolutions and boundary-dominated regimes of pose refinement is unknown; our soft-silhouette ablation answers it, and if the soft version matches, we adopt it and the paper's contribution shifts cleanly to the anchoring and robustness claims, which stand independently of which renderer computes the boundary gradient.
Why not just attack with standard 2D pixel perturbations (PGD) per view? For attacking a refinement loop, this is a genuine threat, not a strawman, and we include it as an attack baseline. The evidence that pure 2D noise under-delivers is that multi-view reconstruction itself filters uncoordinated pixel noise — Gaussian Splatting Under Attack: Investigating Adversarial Noise in 3D Objects found that object-masked, 3D-consistent noise survives 3DGS optimization (dropping CLIP top-1 accuracy from 95.4% to 12.5% on training views) while uncoordinated noise does not — and 2D attacks on feed-forward pose nets, though devastating (an 18.46× rotation-error increase on KITTI sequence 09 in Adversarial Attacks on Monocular Pose Estimation), have never been tested against an iterative analysis-by-synthesis loop that can re-render and self-correct. If adaptive per-view PGD derails our loop as effectively as 3D-consistent camouflage, the defense claim narrows to physical-world attacks, and we will report it that way.
Proposed Experiments
Experiment 1 tests the basin of attraction. On Tanks and Temples and LLFF, we inject controlled initial rotation noise from 5 to 45 degrees (plus translation jitter) and measure final PSNR and pose rotation/translation error for the decoupled pipeline, the photometric-only baseline (whose published convergence bound is 15 degrees and whose published Tanks and Temples PSNR is 31.24 dB, both from Look Gauss, No Pose), the anisotropy-loss ablation, and the silhouette-anchored method. We also run Replica; no reported result was found on Lacuna for any of these methods on Replica, so all Replica numbers will be freshly measured for every method under identical settings. Success is convergence beyond 15 degrees with strictly better pose error than every ablation at matched compute.
Experiment 2 tests degeneracy directly: we log Gaussian eccentricity distributions, floater counts, and per-scene anisotropy-loss values, predicting that the silhouette anchor prevents the flattening behavior rather than merely penalizing it after it appears.
Experiment 3 is the camouflage evaluation. Using multi-view synthetic driving sequences of a vehicle with a mesh prior, textured with camouflage from the NGC/LPGD pipeline and an ACTIVE-style variant, we track the object's pose across the sequence with each refinement method and report the relative increase in pose error under attack versus clean — with the feed-forward degradation factor of 18.46× from Adversarial Attacks on Monocular Pose Estimation as the contextual scale of how badly pose estimation can break. Detector average precision is logged as manipulation check (the cited camouflage pipelines report driving DETR from 97.56% to 6.28% and YOLOv3 from 99.57% to 7.29%), but the headline metric is pose error, which no cited work reports for any refinement loop. The prediction: photometric-only refinement drifts badly because the corrupted texture poisons the RGB loss, while the silhouette anchor is largely unaffected, since the outline is unchanged by paint.
Experiment 4 attacks the anchor itself. Following the MeshAdv recipe of optimizing vertex displacements under a smoothness penalty, but now targeted at the silhouette loss and evaluated against our optimizer, we measure the minimum vertex-displacement budget (and the resulting silhouette IoU change) required to bias the estimated pose by, say, 10 degrees. This quantifies the price of defeating the defense: if a successful attack requires silhouette distortions large enough to be visible in the mask (IoU drop above a stated threshold), the defense has forced the adversary into a detectable regime — the honest, useful form of a robustness claim.
Decisive Checks
The proposal fails constructively if, at matched tuning, the anisotropy-loss ablation converges as far and as accurately as the silhouette anchor; that outcome would reduce the contribution to the robustness study, which can stand alone. The robustness claim fails in its strong form if adaptive per-view PGD on rendered frames derails the anchored loop as effectively as 3D-consistent camouflage — in which case the honest conclusion is that the defense covers physical, 3D-consistent attacks only, and Experiment 4's quantification of the geometric-attack price becomes the paper's central result. Either way, the evaluation produces the first numbers on how test-time differentiable pose refinement behaves under multi-view adversarial optimization, a question none of the cited literature touches.
References
- Analytical derivatives for differentiable renderer: 3D pose estimation by silhouette consistency
- Differentiable Surface Rendering via Non-Differentiable Sampling
- End-to-End 6-DoF Object Pose Estimation Through Differentiable Rasterization
- DSC-PoseNet: Learning 6DoF Object Pose Estimation via Dual-Scale Consistency
- ACTIVE: Towards Highly Transferable 3D Physical Camouflage for Universal and Robust Vehicle Evasion
- Physical Adversarial Camouflage Through Gradient Calibration and Regularization
- Adversarial Attacks on Monocular Pose Estimation
- MeshAdv: Adversarial Meshes for Visual Recognition.
- HeroGS: Hierarchical Guidance for Robust 3D Gaussian Splatting under Sparse Views
- Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization
- Neural 3D Mesh Renderer
- Gaussian Splatting Under Attack: Investigating Adversarial Noise in 3D Objects