Gravity-Anchored Sensorless Freehand 3D Ultrasound

A handheld ultrasound probe produces a stream of flat 2D images. To turn them into a 3D volume, you must know where each image sat in space: three rotation angles and three translations per frame. "Sensorless" methods try to infer this pose from the images alone, and they drift. The best image-only tracker in our file still ends a prostate sweep about 17 mm from where it should be. Yet a cheap accelerometer clipped to the probe reports, at every frame, which way is down. That gives two of the three rotation angles (tilt forward/back and tilt sideways) in absolute terms, with no drift.

We propose to rebuild sensorless reconstruction around this fact. Gravity fixes tilt. The images are then asked to recover only what gravity cannot see: rotation about the vertical axis and the three translations. The unknown angle at which the sensor was clipped onto the probe is estimated jointly rather than calibrated by hand.

The reason to expect more than a small gain is specific. Image-based out-of-plane motion estimation confuses two different motions. When the probe tilts, and when it slides perpendicular to the image plane, the speckle pattern (the grainy texture in ultrasound images) changes in similar ways. Errors in tilt therefore leak into translation estimates and accumulate as drift. Removing tilt from the unknowns should break this confusion. The claim is that the translation error falls, not merely the rotation error.

Core Claim

Each frame's gravity reading, expressed in the probe's own axes, pins pitch and roll up to accelerometer noise. Ultrasound sweeps are slow, nearly static motions, which is exactly the regime where separating gravity from the probe's own acceleration works. Inertial positioning already relies on this: the survey Deep Learning for Inertial Positioning: A Survey describes gravity-aligned frames as the standard way to bound drift.

We reparameterize each frame pose as a rotation about the measured vertical axis, composed with a gravity-determined tilt and a fixed but unknown sensor-to-image mounting rotation $R_m$, plus a translation. Pose estimation then optimizes only the per-frame yaw, the per-frame translation, and the single shared $R_m$. The new element is not "add an IMU (inertial measurement unit)" as an extra input. It is treating gravity as a hard constraint that shrinks the space of possible poses, inside a joint optimization of pose and volume. Under that constraint, we predict translation drift falls substantially, not just tilt error.

Research Program

The method builds on two existing systems. ImplicitVol represents the volume as a neural field: a small network mapping a 3D coordinate to image intensity. It refines all six pose parameters per frame jointly with this field, using an image-similarity loss. It starts from a learned pose predictor (PlaneInVol) and needs repeated network reinitialization to escape bad starting poses. DCL-Net instead regresses frame-to-frame motion with 3D convolutions over short clips. It adds attention on speckle-rich regions and a correlation loss that discourages over-smoothed trajectories.

Neither system uses any physical reference frame. Both motivate this proposal, because their residual errors are the gap. Their architectures also supply our machinery. We keep DCL-Net's clip encoder to propose initial yaw and translation, then run ImplicitVol-style joint refinement over the reduced parameters.

The mounting rotation $R_m$ creates a chicken-and-egg problem. The gravity constraint needs $R_m$, but $R_m$ is best estimated from a good reconstruction. This mirrors the gait-matching problem that Thanh et al. solved by alternating rotation estimation with sequence alignment. We borrow that alternation. We fix $R_m$, refine poses and field, then re-solve $R_m$ by aligning predicted and measured gravity directions over the whole sweep, a small rotation-fitting problem.

We do not reduce the IMU signal to its magnitude. As Thanh et al. note, the norm discards exactly the direction we need.

The weak point of gravity separation is fast motion. During fast motion, the probe's own acceleration corrupts the gravity estimate. We therefore weight the gravity constraint per frame by how far the measured acceleration magnitude departs from 1g. When the probe is moving abruptly, the constraint softens and image evidence takes over.

Data is the practical obstacle. None of the datasets in our file pairs ultrasound with a probe-mounted IMU. We use three sources, in increasing realism:

  1. NIH transrectal ultrasound (TRUS) dataset. Used by DCL-Net, it has 640 prostate sweeps with electromagnetic-tracker ground-truth poses. We synthesize accelerometer and gyroscope streams from the tracked orientations, adding noise, bias, and a random mounting rotation. Gravity is placed at a fixed but arbitrary direction in the tracker's frame. Because the tracker's transmitter is stationary, this is physically consistent with some table orientation.
  2. Fetal brain volumes. These are the atlas-aligned volumes of ImplicitVol's Datasets A and B. We sample synthetic freehand sweeps with known poses and simulated IMU streams.
  3. A new acquisition. We will record tissue-mimicking phantom sweeps with a clip-on consumer IMU and an optical tracker. This is the only test of real sensor errors and real mounting behaviour, and it will be released.

Baselines

The naive reference is DCL-Net run unchanged on images alone. On the NIH TRUS test split it reports a median distance error of 9.15 mm and a median final drift of 17.40 mm (DCL-Net). We retrain it on our splits to confirm we reproduce these numbers before comparing.

For the fetal sweeps, the strongest image-only competitor is ImplicitVol. It reports relative gains over PlaneInVol-based baselines: more than 50% in SSIM (a structural image-similarity score) and more than 40% in VIF (visual information fidelity) for reconstruction, and more than 35% and 40% in angular and translational error. No reported result found on Lacuna for ImplicitVol's absolute pose errors. We will therefore run it ourselves with its published settings.

The strongest physics-aware competitor is one we construct, since none exists in the file. It is DCL-Net with the IMU's orientation estimate concatenated as an extra input and trained end to end. All methods share the same training cases, the same encoder backbone, the same epoch budget, and the same hyperparameter search size.

Two ablations isolate the key ingredient:

  • Constraint versus input. The gravity constraint is replaced by the same gravity information fed only as an input feature. This tests whether the hard constraint matters or merely the information.
  • Mounting estimation. $R_m$ is fixed at its true value (an oracle) and at identity (naive). This measures how much self-calibration matters.

Beating the field means two things on NIH TRUS. First, final drift must fall below the concatenation baseline with non-overlapping bootstrap confidence intervals. Second, out-of-plane translation error must also improve, not only tilt error, since the core claim is that removing tilt reduces translation errors.

Why not just…?

Why not just concatenate IMU readings to an image network? This is the real threat. With abundant in-distribution training sweeps, a network may learn to use gravity nearly as well as a hard constraint. Our bet is that the constraint generalizes better under two conditions: unseen sweep patterns, and a mounting rotation not seen in training, where a learned fusion must extrapolate. The constraint-versus-input ablation settles this. We train on near-identity mounting angles and test on uniformly random ones. If concatenation matches the constraint even there, we will report that gravity information, not its formulation, carries the gain, and narrow the claim to the self-calibration component.

Why not just take all three rotations from the IMU, with the gyroscope supplying yaw, and use images only for translation? For short sweeps, gyroscope yaw drift may be small, so this simpler split could capture most of the benefit. We include it as a baseline, along with joint refinement applied to translation only. If it ties with full joint refinement, the lesson is still useful: physical sensors should own rotation, and images should own translation. The image-based yaw correction would then simply be unnecessary at these sweep lengths. We report results by sweep duration to locate where gyroscope drift starts to matter.

Why not just use an electromagnetic or optical tracker? Trackers solve the problem at clinical-hardware cost, need setup, and suffer from metal interference. A clip-on IMU costs a few dollars. Trackers serve here only as ground truth.

Why not just make inputs rotation-invariant by taking signal magnitudes? Magnitude erases the direction of gravity, which is the whole signal (Thanh et al.), so it cannot recover tilt.

Proposed Experiments

NIH TRUS. We report median distance error and final drift, matching DCL-Net's metrics. We add per-axis angular error and out-of-plane translation error, broken down by sweep length.

Mechanism test. This is the decisive check. We correlate tilt error with out-of-plane translation error across frames, separately for DCL-Net and for our method. If the claimed tilt–translation confusion is real, the image-only model should show a strong positive correlation. It should largely disappear under the gravity constraint, and the drift reduction should be concentrated in the out-of-plane translation axis.

Fetal brain sweeps. We measure reconstruction SSIM and VIF, plus angular and translational pose error against ImplicitVol. We also record how many network reinitializations each method needs. We expect the constrained search to need fewer, which would reduce ImplicitVol's high optimization time.

Stress tests. We vary three factors: IMU noise and bias levels, mounting angle, and sweep speed. Sweep speed probes the regime where probe acceleration corrupts gravity and checks that per-frame down-weighting degrades gracefully toward the image-only result rather than below it.

Phantom acquisition. The phantom data tests whether gains seen with simulated IMU noise survive real sensors. If gains vanish here but not in simulation, our noise model is wrong. We will report that outcome as such.