When a robot seats a part on a fixture, plugs a connector, or presses a pad against a surface, the moment that matters is hidden. Its own wrist, fingers, cables, or tool shaft sit between the workspace camera and the interface. We propose to make that interface report itself in color. A thermochromic coating, a paint whose pigment changes color when heated, turns warm contact into a visible footprint, and uncontacted gaps stay their original color. We then treat the camera's lens as a controllable sensor rather than a fixed pinhole.
The physical fact we exploit is simple. A lens with a wide aperture, focused on the far surface, gathers light from each surface point over a whole disk of ray directions. A thin foreground object therefore blocks only part of that disk. The surface stays partly visible behind it, mixed with a blurred image of the occluder. A geometric perception task such as edge detection or pose estimation is ruined by this blur. A color readout is not, because blur averages color without changing it. The robot also knows exactly where its own links are from its joint angles, so it can compute, pixel by pixel, how much of the aperture each surface point can still see. It can then subtract the occluder's contribution.
The opportunity is a controller that jointly chooses aperture, focus, zoom, and a small camera move to keep every point of the contact interface at least partly visible. It recovers the thermochromic contact map while the robot is still pressing, rather than after it lets go.
Core Claim
For chromatic contact signals and occluders of known geometry, the camera aperture acts as a single-shot synthetic aperture: one exposure that combines many viewing directions at once. The image formation can be written per pixel $x$ as
$$I(x) = \alpha(x)\,S(p_x) + \big(1-\alpha(x)\big)\,B(x),$$
where $I(x)$ is the observed color, $S(p_x)$ is the color of the surface point $p_x$ in focus at that pixel, $B(x)$ is the defocused color of the occluding robot link, and $\alpha(x)\in[0,1]$ is the fraction of the aperture disk through which $p_x$ is unobstructed. The robot computes $\alpha$ by ray-casting its own mesh through a thin-lens model. Unmixing then amplifies sensor noise by roughly $1/\alpha$, because the surface term is scaled by $\alpha$. The controller therefore solves
$$\max_{N,\,s,\,f,\,\Delta T}\ \min_{p \in \mathcal{C}} \alpha(p) \quad \text{s.t. the depth of field covers } \mathcal{C},$$
where $N$ is the f-number, $s$ the focus distance, $f$ the focal length, $\Delta T$ a small camera pose change, and $\mathcal{C}$ the set of interface points to verify.
A short geometric argument shows what this can and cannot do, and it shapes the whole design. An occluder of width $w$ at height $h$ above a surface point, with the camera at distance $Z$ and aperture diameter $D=f/N$, blocks that point completely only if $w > Dh/Z$. At fixed framing, $f/Z$ is fixed, so $D/Z$ depends only on $N$. Zooming in from farther away (a dolly-zoom) buys nothing. Only opening the aperture or moving closer does.
The method is therefore aimed at thin occluders hovering above the interface: tool shafts, cables, edge-on gripper fingers, tendons. A palm pressed flat over the contact stays a blind spot that only camera motion can address. The joint controller is supposed to know which situation it is in. The new element is the pairing of a defocus-tolerant signal with known-geometry partial occlusion, so that blur the controller induces on purpose becomes an asset rather than a failure mode.
Relation to Prior Work
Active control of lens parameters is established in cinematography. CineMPC optimizes drone pose together with zoom, focus, and aperture under a thin-lens model to satisfy framing and depth-of-field goals. It treats defocus and occlusion as tracking hazards bridged by a Kalman filter. CinemAirSim supplies a simulator with physically grounded defocus. Both supply our optical model. Neither uses aperture to see past an occluder, and both pursue an aesthetic objective rather than a measurement one. Computational stereo camera system with programmable control loop shows that motorized focus and zoom can run in a tight hardware feedback loop. This constrains our hardware choices toward something feasible.
Occlusion-breaking by moving cameras is the main competing idea. Greedy Perspectives plans multi-drone positions to maximize rendered visible pixels. Pose-Assisted Multi-Camera Collaboration for Active Object Tracking re-orients occluded cameras from peers' poses. Both control only extrinsics with fixed lenses. We borrow their rendered-visibility objective but replace "visible pixels" with the aperture fraction $\alpha$, which is continuous rather than binary.
DeOccNet removes foreground occluders using parallax across a light-field camera array. It motivates the physics: points hidden in one view are visible in others. It also states the hard limit we inherit, that points hidden from every view cannot be recovered. We differ in two ways. Our multiple viewing directions come from inside a single lens aperture, and our occluder geometry is known rather than inferred.
On the contact side, HumanFT puts thermochromic pigment inside a fingertip watched by an internal camera. The thermochromic bolus study uses color change to expose air gaps between a radiotherapy bolus and skin. The bolus study is the closest analogue to our "seating gap" check. Both read the pigment from an unobstructed or internal view.
The visuotactile policies DECO and GelFusion respond to occlusion by switching to touch. Their results motivate us: occlusion during contact really costs success. They do not compete on our question, because fingertip touch reports what the finger feels, not whether a part is flush against a fixture elsewhere.
Research Program
The physical setup is a tabletop arm with a heated gripper pad, or pre-warmed workpieces, at about 40°C. Fixtures are coated with reversible leuco-dye film that changes color around 31°C. Full seating produces a continuous color footprint, and gaps remain uncolored.
Ground truth comes from building the fixtures out of clear acrylic with a second camera underneath. That camera sees the footprint unoccluded. It is used only for scoring and never shown to any method.
The workspace camera carries a motorized zoom, focus, and aperture lens of at least 25 mm maximum physical aperture. It is mounted on a short linear-plus-pan stage, which provides limited pose authority as in a cramped cell.
The software has three parts. The first is a GPU ray-caster that renders $\alpha(x)$ from joint angles, robot mesh, and the current lens state. The second is an occluder appearance model for $B(x)$, rendered from the calibrated link texture and then refined by a least-squares correction fitted on pixels where $\alpha\approx 0$. The third is the controller, a sampled search over a few hundred lens and pose candidates per cycle, scored by the objective above. The search is deliberately cheap. CineMPC's full nonlinear planning cycle took about 631 ms on embedded hardware, which is too slow for contact events lasting a few seconds, and our objective decomposes into ray-casts that batch well.
We will also run a temporal fusion step. It weights each frame's unmixed estimate by $\alpha^2$, because the noise variance scales as $1/\alpha^2$. Thermochromic hysteresis keeps the footprint stable for seconds, which makes such fusion valid.
Simulation is in CinemAirSim's thin-lens renderer with an inserted arm model and thermochromic textures. It allows sweeping occluder width, height, f-number, and camera distance to check the $w > Dh/Z$ boundary before hardware experiments.
Baselines
The naive reference is a fixed camera with a fixed mid-range f-number and focus set on the fixture. It thresholds the raw color image, so occluded pixels are simply counted as unknown or misclassified. This anchors how much of the interface is lost to self-occlusion at all.
The second reference exploits thermochromic memory with no optics. The fixed camera keeps, for each pixel, the most recent unoccluded observation, and it also inspects the footprint immediately after release. This is cheap and plausibly strong, and it tests whether in-contact verification adds anything.
The strongest existing approach to the occlusion half is pose-only active view planning. It is a single-camera adaptation of the rendered-visibility planner in Greedy Perspectives, using the same stage, the same time budget, the same mesh renderer, and a binary-visibility objective with the lens fixed at a typical f/4. That paper reports 14% greater view reward than a formation planner in its Merge, Corridor, and Forest scenarios. No reported result was found on Lacuna for contact-map recovery under self-occlusion. All of our comparison numbers will therefore be generated here on identical trials.
A second strong competitor is a motion-built synthetic aperture. The stage sweeps the camera to capture a small light field, which is then refocused or run through DeOccNet retrained on our rendered data with the same training budget as our unmixing model. DeOccNet reports 21.27 dB PSNR on the CD scene against 16.75 dB for plain refocusing. Plain refocusing is the light-field analogue of opening the aperture without unmixing.
Ablations isolate the key ingredient. One opens the aperture wide but skips unmixing. One unmixes at a fixed lens state. One runs full lens control with pose frozen. A last one replaces kinematics-derived $\alpha$ with $\alpha$ estimated from the image alone, to test whether known geometry is what carries the method.
Beating the strongest baseline means two things on trials where the occluder is thin relative to the angular aperture. First, our method must reach higher footprint intersection-over-union against the under-fixture ground truth and higher gap-detection area under the ROC curve. Second, it must do so in less wall-clock time than pose-only planning or the motion-built light field, with a pre-registered margin of at least 0.05 IoU. On thick-occluder trials we predict no advantage, and seeing none would confirm the analysis rather than embarrass it.
Why not just…?
Why not just move the camera to a clear view? In an open workspace this probably wins, and we expect pose-only planning to match us there. The claim is limited to cells where the stage cannot travel far enough to clear the arm. The experiment varies stage travel, from a few centimeters to full freedom, and reports where the curves cross. If pose-only planning matches us across the realistic range, the idea is not worth pursuing and we will say so.
Why not just look after the robot lets go? Thermochromic hysteresis makes this tempting, and it is a baseline above. It fails exactly when seating relaxes or shifts on release. It also cannot support correction while force is still applied, which is the point of verifying during contact. We will include fixtures with a compliant seat that springs back, and measure how often post-release inspection reports a different outcome than the in-contact ground truth.
Why not just add a second fixed camera or use a thermal camera? A second camera is a real and cheap fix where mounting space exists. We report it as a reference with doubled sensor count, not as a fair competitor. A long-wave infrared camera sees heat directly but is blocked by the same arm. It also blurs the thermal footprint by lateral conduction, so it gains nothing on occlusion. It does serve as a useful check that our color footprint matches the true contact region.
Why not just inpaint the hidden region with a learned model? Inpainting hallucinates plausible color precisely where the gap would be, which defeats verification. The image-only-$\alpha$ ablation and the retrained DeOccNet baseline quantify how much known kinematics adds over learned inference.
Proposed Experiments
The first check is in simulation. It sweeps occluder width $w$, height $h$, distance $Z$, and f-number $N$, and tests whether recovery error follows the predicted $1/\alpha$ amplification and collapses at $w = Dh/Z$. It also tests whether, at fixed framing, changing focal length leaves recovery unchanged, as the $D/Z$ argument predicts.
The second check uses the DeOccNet synthetic masked light fields. Summing sub-aperture views over a disk produces a physical-aperture image with a known mask, so we can measure PSNR of known-mask unmixing against DeOccNet's reported 21.27 dB on CD. This comparison deliberately gives us extra information. It measures the value of known occluder geometry, not a fair contest.
The decisive hardware check runs seating trials on acrylic fixtures with injected gaps from 0.2 to 2 mm, crossed with three occluder classes: a cable, an edge-on finger, and a palm-over grasp. Every baseline runs on the same trials. Results are footprint IoU, gap-detection AUROC, time to verdict, and controller cycle time.
The idea is confirmed if the thin-occluder classes show the predicted advantage while the palm class shows none. It is refuted if pose-only planning with equal travel closes the gap, or if unmixing without kinematic $\alpha$ performs just as well.
References
- CineMPC: A Fully Autonomous Drone Cinematography System Incorporating Zoom, Focus, Pose, and Scene Composition
- CinemAirSim: A Camera-Realistic Robotics Simulator for Cinematographic Purposes
- Computational stereo camera system with programmable control loop
- Greedy Perspectives: Multi-Drone View Planning for Collaborative Perception in Cluttered Environments
- Pose-Assisted Multi-Camera Collaboration for Active Object Tracking
- DeOccNet: Learning to See Through Foreground Occlusions in Light Fields
- HumanFT: A Human-Like Fingertip Multimodal Visuo-Tactile Sensor
- Innovative applications of visualized thermosensitive color-changing personalized boluses in post-mastectomy radiotherapy: a dosimetric analysis
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- GelFusion: Enhancing Robotic Manipulation under Visual Constraints via Visuotactile Fusion