Fixed Feedback for Camera Poses: Feedback Alignment as a Way Out of Pose Local Minima

When a 3D scene model and its camera positions are fitted together from photographs, each camera pose is nudged in whatever direction reduces the mismatch between the rendered and the observed image. That direction is the exact gradient of the mismatch. Near a wrong answer, the exact gradient often points nowhere useful: high-frequency texture in the scene model creates small pockets where it vanishes, and the optimizer stalls far from the true pose. This proposal borrows a trick from feedback alignment, a family of learning rules that update a network with a fixed projection of the output error instead of the exact chain-rule gradient. The pose of each camera would be updated by multiplying its image error by a fixed "feedback" matrix, while the scene model keeps training normally. The optimizer would then stop where that fixed projection of the error is zero, not where the true gradient is zero, so the pockets that trap gradient descent are no longer stopping points.

The bet has two parts. The first is that a well-chosen fixed feedback operator gives the pose a wider basin of attraction, meaning it converges from a larger range of starting guesses. The second, and the part with no counterpart in classical vision, is that the scene model, trained jointly, will reshape itself so that this fixed feedback becomes a good descent direction. Feedback-alignment theory calls this the alignment phase. If both hold, pose recovery from poor initializations improves without frequency schedules or many random restarts, and pose updates no longer require a backward pass through the renderer.

Core Claim

Let camera $i$ have pose $\xi_i \in \mathfrak{se}(3)$, a six-number description of a rigid motion, and let $r_i$ be the vector of per-pixel photometric residuals between the rendered and observed image. Let $J_i = \partial r_i / \partial \xi_i$ be the exact Jacobian through the scene model. Standard joint optimization such as BARF updates $\Delta \xi_i = -\eta J_i^\top r_i$. We instead use

$$\Delta \xi_i = -\eta\, B_i\, r_i ,$$

where $B_i$ is a $6 \times N$ feedback operator that is fixed, or refreshed only rarely, and is not computed through the current scene network. This update is a descent direction whenever

$$r_i^\top J_i B_i^\top r_i > 0 ,$$

that is, whenever the feedback step lies within $90^\circ$ of the true gradient. This is the same criterion feedback-alignment analyses track for hidden layers (Explaining the Learning Dynamics of Direct Feedback Alignment, Scaling Direct Feedback Learning with Jacobian Alignment Guarantees). Pose iterates stop only where $B_i r_i = 0$. A spurious stationary point of the true loss, where $J_i^\top r_i = 0$ but $r_i \neq 0$, is generically not such a point.

The specific new claim is a dynamical one. Under joint training, the scene model's pose Jacobian $J_i$ rotates toward $B_i$, so the descent condition becomes easier to satisfy over time, exactly as forward weights align to fixed feedback in Align, then memorise. If this alignment is real, the scene representation itself becomes "pose-friendly" in a way that annealing or feature metrics do not produce.

The research file shows no prior work applying direct feedback to camera poses or to any analysis-by-synthesis loop. That absence motivates the work, but the idea also has a classical cousin that I treat as the main threat. The inverse-compositional style of image alignment precomputes a Jacobian on a fixed template. So the question is not whether fixed Jacobians can work, but whether the joint alignment effect adds anything beyond them.

Research Program

The work compares three choices of $B_i$, and they separate the hypotheses.

The first is a purely random Gaussian $B_i$, the direct analogue of Nøkland's DFA. It succeeds only if the scene genuinely aligns to the feedback, which makes it the cleanest test of the alignment claim.

The second is an observation-derived $B_i$. It is built from spatial gradients of a blurred observed image, multiplied by the analytic warp Jacobian of perspective projection under a coarse depth: either constant depth, or a monocular depth prior as used in NoPe-NeRF. This operator never touches the scene network. It is the inverse-compositional baseline in feedback-alignment clothing.

The third is a matched $B_i$ in the spirit of Feedback-Weight Matching, which builds feedback matrices from the SVD of pretrained weights so alignment holds from the first step. Here the analogue is a low-rank SVD projection of the true $J_i$ taken at initialization or every $K$ steps and then frozen in between. The low-rank truncation is meant to discard the high-frequency directions that create pockets.

The scene model is the BARF coordinate network trained by ordinary backpropagation on the photometric loss. A secondary arm trains its hidden layers with DFA as well, to see whether fully local credit assignment is tolerable. Two caveats bound that arm. Feedback Alignment Meets Low-Rank Manifolds reports that residual paths still distort alignment. The scalability study found feedback-alignment rules struggle on harder vision tasks. I therefore expect this arm to cost rendering quality and treat it as exploratory.

Throughout training I log the per-camera angle between $B_i r_i$ and $J_i^\top r_i$, computing $J_i$ only for measurement. That curve is the direct readout of whether alignment happens.

Baselines

The naive reference is joint NeRF-and-pose optimization with exact gradients and full positional encoding. BARF reports that this fails on LLFF from identity initialization.

The strongest direct competitor is BARF itself, which widens the basin by turning on positional-encoding frequencies gradually (BARF). On NeRF Synthetic with perturbations averaging $14.9^\circ$ of rotation, BARF reports final rotation error under $0.2^\circ$ and 27.50 PSNR. On ScanNet, the file quotes BARF at 0.117 m absolute trajectory error (ATE) and 31.41 dB, as reported in the NoPe-NeRF paper.

The strongest prior-assisted competitor on ScanNet is NoPe-NeRF, which anchors geometry with monocular depth and reports 0.030 m ATE and 31.86 dB PSNR. It is compared only against our variants that also use monocular depth, so that both sides have the same prior.

All methods share the BARF codebase, ray budget, iteration count and learning-rate tuning budget. Every method gets the same number of hyperparameter trials, including the BARF schedule endpoints.

Three ablations isolate the key ingredient:

  • Fixed exact Jacobian. Swap $B_i$ for the true Jacobian frozen at initialization with no low-rank truncation. This separates "fixed" from "truncated."
  • Noise-matched gradient. Replace $B_i r_i$ with $J_i^\top r_i$ plus isotropic noise tuned to the same average angle. This separates structured feedback from mere perturbation.
  • Frozen scene. Freeze the scene network after a short warm-up. This removes any possibility of alignment, so any gain that survives cannot come from the alignment effect.

Beating BARF means one of two things on NeRF Synthetic: a strictly higher convergence rate, defined as final rotation error below $1^\circ$, at perturbation magnitudes beyond $14.9^\circ$ without losing PSNR; or matching BARF's accuracy with no frequency schedule. Beating NoPe-NeRF means ATE below 0.030 m on ScanNet.

Why not just…?

Why not just use coarse-to-fine frequency annealing? BARF already widens the basin and is the bar to clear. The difference is mechanical. Annealing blurs the residual itself and thereby discards fine detail early. Fixed feedback keeps the full-resolution residual and smooths only the direction in which it is applied. The two are also compatible, so I run feedback with and without the BARF schedule. If annealing alone matches every feedback variant at every perturbation magnitude, the proposal has failed and I will report that.

Why not just add gradient noise or restart from many initial poses? NeMo needs 144 initial pose seeds to handle local minima (NeMo). This shows that restarts work but are expensive. The noise-matched ablation tests directly whether structure in $B_i$ matters. If isotropic noise at the same angle recovers the same basin, the feedback-alignment framing adds nothing.

Why not just use classical inverse-compositional alignment, or frozen foundation features as in AlignPose? This is the real threat. The observation-derived $B_i$ is essentially an inverse-compositional Jacobian. AlignPose shows frozen DINOv2 feature metrics give wide basins, reaching 89.5% average recall on T-LESS. The experiment that settles this is the random-$B$ arm together with the frozen-scene ablation. If only the observation-derived operator works, and it works equally well with a frozen scene, then the contribution reduces to "inverse-compositional updates inside NeRF." That is a useful engineering result, but not evidence for learned alignment. A feature-metric residual is orthogonal to the change in update rule and could be layered on later.

Why not just initialize from a feed-forward pose model? 3R-GS reports that MASt3R-style initial poses lack sub-pixel precision and still leave joint optimization trapped. Better initialization therefore shrinks the problem without removing it. I include ScanNet runs started from such coarse poses to check that the gain persists when the initialization is already close.

Proposed Experiments

The decisive experiment is a basin-width curve on NeRF Synthetic. Initial rotation and translation perturbations are swept from small up to well beyond BARF's $14.9^\circ$ setting, with 20 seeds per magnitude. For each method I plot convergence rate and report final PSNR. The feedback approach succeeds if its curve sits to the right of BARF's.

On LLFF I test recovery from identity initialization. The file has no reported quantitative number for this setting, so success is judged by agreement with COLMAP poses. On ScanNet I report ATE, rotation error and PSNR against BARF and NoPe-NeRF as specified above.

The mechanistic check is the alignment-angle trajectory. The random-$B$ variant should start near $90^\circ$ and then fall well below it as training proceeds, and the frozen-scene control should not show this decline. If that pattern appears together with a widened basin, the scene has learned to make fixed feedback useful. If it does not appear, the honest conclusion is that only structured, observation-derived feedback helps, and the classical reading wins.

A secondary measurement is peak memory and wall-clock time when pose updates run in parallel with scene updates. This is reported for completeness, not as the main claim.