Farms that grow crops underneath raised solar panels ("agrivoltaics") have to decide, hour by hour, how far to tilt the panels away from the sun so the plants below get enough light. Every degree of tilt given up for the crop is clean electricity that has to come from somewhere else, often fossil generation. Today that decision rests on a crude picture of the crop: a flat plane receiving a scalar fraction of sunlight. Drone-based 3D reconstruction could replace this picture with a measured canopy. The difficulty is that standard reconstruction methods cannot tell a leaf that is dark green from a leaf that sits in a panel's shadow. They "bake" the moving shadows into the plant's color and shape, which corrupts exactly the quantity the controller needs.
This proposal rests on one observation. Under a solar tracker, the shadow-casting object is not unknown. The panel's geometry, its tilt at every timestamp (from the tracker log), and the sun's direction are all known. The moving shadow is therefore a controlled, known change in lighting that sweeps across the canopy, much like the structured light in a 3D scanner. We propose to model it explicitly instead of treating it as noise. As the shadow edge passes over a point, the drop in brightness tells us how much direct sun that point was receiving. That reveals both its true color and how deep it sits inside the canopy. The output is a relightable 3D canopy whose leaf-level light distribution can be fed back into the panel-tilt optimizer. This lets us measure how much electricity the flat-ground crop model gives up.
Core Claim
Inverse rendering means recovering shape, surface color, and lighting from photographs instead of only fitting the photographs. It is badly underdetermined in general, because a dark pixel could be a dark surface or a dim light. Under an agrivoltaic tracker, part of the lighting is known in closed form: direct-sun visibility blocked by the panels can be computed from the logged panel pose. Multi-time drone captures can then be fit with an image model in which panel shadows are rendered, not learned. That constraint should separate leaf albedo (intrinsic surface color) from illumination well enough to recover per-leaf absorbed photosynthetically active radiation, PAR (the 400–700 nm light plants use). A generic inverse renderer cannot do this, and neither can a reconstruction that simply discards shadowed pixels. The resulting leaf-level PAR estimate should change the energy-versus-crop trade-off computed by current tilt optimizers. The size and direction of that change is the downstream question this project is built to answer.
Where This Sits
Three lines of prior work motivate the gap. UAV3DCrop benchmarks radiance-field and Gaussian-splatting reconstructions on multi-angle drone crop surveys. It finds that the methods with the best renderings have the worst depth, and vice versa. None of the methods separates lighting from color, and no panels are present. CropCraft fits a procedural soybean growth model to a NeRF reconstruction and produces simulation-ready meshes. Its authors assume uniform open-sky lighting, so it does not handle cast shadows. Optimizing Dual-Axis Solar Panel Operation in an Agrivoltaic System solves the tilt problem with a convex relaxation and a flat-ground linear shading factor. It shows the trade-off is steep: an 80% PAR requirement costs 0.7% of energy, while 88% costs 46.1%. That steepness is why the accuracy of the crop-light model matters. Our work supplies a measured 3D replacement for its flat-plane crop and leaves its optimizer unchanged.
Two papers supply the method. Extracting Triangular 3D Models, Materials, and Lighting From Images shows end-to-end differentiable recovery of geometry, physically based materials, and environment lighting for single objects. Differentiable Shadow Mapping makes cast shadows differentiable in a fast rasterizer by smoothing depth comparisons with variance shadow maps. It also documents "light bleeding" with multiple occluders and is restricted to point and directional lights. Our panel shadows fit inside that restriction because they come from a directional sun and a few large planar occluders. Leaf self-shadowing does not, and it is where light bleeding would hurt. We therefore compute self-shadowing by marching rays through the canopy's own density.
Two papers constrain the design. Daniel Cremers et al. and Rui Chen et al. emphasize that joint recovery of shape, light, and reflectance is ill-posed without priors. Our response is a known-occluder prior, not a learned one. Retrieval of Surface Solar Radiation through Implicit Albedo Recovery from Temporal Context is the closest conceptual relative. At 1.7 km satellite scale, it separates slowly changing ground albedo from fast-changing clouds by their different time scales. We exploit the same asymmetry at centimeter scale, with one advantage: our fast-changing term (panel shadow) is known, not inferred.
Research Program
The scene representation is a set of 3D Gaussians for the canopy and soil plus an explicit mesh for the panel array. A 3D Gaussian is a small soft ellipsoid; Gaussian splatting is a scene representation built from many of them and rendered quickly. Each Gaussian carries a diffuse albedo, an opacity, and a scalar leaf-transmission coefficient for light passing through thin leaves. The panel mesh is posed per frame from tracker logs, refined by a small learned per-row offset to absorb calibration error. For a canopy point $x$ seen in frame $t$, the predicted color is
$$ I_t(x) = a(x)\Big[E^{\mathrm{sun}}_t \max(0, n(x)\cdot s_t)\, V^{\mathrm{pan}}_t(x)\, V^{\mathrm{can}}_t(x) + E^{\mathrm{sky}}_t\, A(x)\Big] + \tau(x)\, T_t(x). $$
Here $a(x)$ is albedo and $n(x)$ the surface normal. $s_t$ is the sun direction from the timestamp and site location. $E^{\mathrm{sun}}_t$ and $E^{\mathrm{sky}}_t$ are direct and diffuse irradiance, initialized from on-site pyranometer or National Solar Radiation Database values and refined per frame. $V^{\mathrm{pan}}_t(x)\in[0,1]$ is panel visibility, computed geometrically with a soft penumbra set by the sun's angular size, with no learned parameters. $V^{\mathrm{can}}_t(x)$ is canopy self-visibility obtained by marching a ray toward the sun through the Gaussian densities. $A(x)$ is a sky-visibility term computed the same way over a few directions. $\tau(x)T_t(x)$ is light transmitted through overlying leaves. Camera exposure is handled by a per-frame scalar gain.
The key ingredient is that $V^{\mathrm{pan}}$ is computed rather than fit. When a pixel darkens exactly where and when the panel shadow arrives, the model cannot attribute this to albedo. The size of the darkening, compared across frames, constrains $V^{\mathrm{can}}$ and therefore how buried that point is. Training uses captures at several tracker angles within a short window, short enough that the plants are static up to wind. A wind-robustness term down-weights pixels with large residual inconsistency across adjacent frames.
The downstream product is a leaf-level PAR map. We integrate rendered direct and diffuse PAR absorbed by every leaf Gaussian, weighted by $1-a$ in the PAR band, over a day of sun positions and any candidate tilt schedule. That map replaces the flat-plane shading factor in the dual-axis optimizer of the agrivoltaic tilt study, whose solver we keep as is. A secondary product, not a central claim, is canopy albedo under the panels, which is the input bifacial modules would need for rear-side yield.
Baselines
The naive reference is that paper's flat-ground shading factor. It predicts crop PAR from panel geometry and sun angle alone, with a reported shading approximation error averaging 0.49% against its own geometric model. That figure is internal consistency, not agreement with measured leaf light. We run it with the same panel layout and irradiance inputs as our method. Beating it means lower error against under-canopy PAR sensors and against path-traced ground truth in synthetic scenes.
The strongest appearance-only reconstruction baselines come from UAV3DCrop: Scaffold-GS for geometry (best depth error, 0.722 m RMSE; canopy-height MAE about 0.09 m, tied with Splatfacto) and Splatfacto-big for appearance (best PSNR, 19.40 dB). We train both with their benchmark settings on identical image sets and the same GPU-hour budget. Each gets per-image appearance embeddings so it is not handicapped by lighting changes it was never designed for. Their PAR estimate comes from treating rendered brightness as a light proxy. Beating them means better geometry under panels, and albedo that stays the same color regardless of shadow state.
The strongest disentangling baseline is a generic inverse renderer in the style of Munkberg et al., augmented with differentiable shadow mapping. It learns lighting and occluders without the tracker log. It gets the same representation, budget, and hyperparameter search as our method. No reported result on vegetation or agrivoltaic scenes was found on Lacuna for either method.
Three ablations isolate our ingredient. The first replaces computed $V^{\mathrm{pan}}$ with a learned per-frame shadow field. The second keeps computed shadows but trains on a single tracker angle, removing the sweep. The third masks predicted-shadow pixels instead of modeling them. If the first ablation matches the full model, the known-occluder claim fails.
Why not just…?
Why not just mask out the shadowed pixels, since the panel pose is known anyway? This is the most serious alternative. It may well match our model on geometry and on albedo of sunlit leaves. What masking cannot do is recover light inside the canopy. It discards precisely the frames in which a point's brightness changes because of a known light change, and those frames carry the self-visibility signal. The masking ablation settles this directly. If masking matches us on leaf-level PAR error against sensors, the contribution shrinks to a convenience, and we would report that.
Why not just put quantum PAR sensors under the panels? Sensors are ground truth at a point and are what we validate against. A few sensors cannot resolve the spatial pattern across rows and canopy depth, and they read light at sensor height, not at leaves. We test this by holding out sensors. We compare our map at held-out locations against spatial interpolation from the remaining sensors. If interpolation from a handful of sensors is as accurate, the vision pipeline is not worth its cost for control.
Why not just run CropCraft and simulate light in Helios? CropCraft reports a soybean leaf area index error of 0.69, against 1.11 for Poisson reconstruction. Its meshes can be relit in a simulator. It assumes uniform open-sky lighting and fits morphology to NeRF depth and normals. Under panels, those depth and normal histograms come from a reconstruction that has baked shadows into geometry. We run CropCraft on panel scenes as an additional baseline and, separately, initialized from our recovered geometry. If our geometry improves CropCraft's leaf area estimate under panels, the two are complementary rather than competing.
Proposed Experiments
No public image benchmark of crops under trackers appears in the file, so evaluation has three parts. The first is synthetic. We build agrivoltaic scenes from procedural soybean canopies at several growth stages, place dual-axis panel rows modeled on the Campus Farm layout, and path-trace drone views under July sun positions, using Mitsuba 3, the path tracer the shadow-mapping paper benchmarks against. Ground-truth albedo, depth, and per-leaf absorbed PAR are then exact. Metrics are albedo error split by shadow state, depth RMSE, and per-leaf PAR error. The decisive check is the gap in albedo between shadowed and sunlit portions of the same leaves. Baked-in lighting shows up as a large gap. Our model should shrink it, and the learned-shadow ablation should not.
The second is a regression check on UAV3DCrop, which has no panels. Our model reduces to a sun-and-sky renderer there and should not fall below Scaffold-GS's 0.722 m depth RMSE or the ~0.09 m canopy-height MAE by more than a small margin. This shows the added physics does not cost accuracy in ordinary fields.
The third is a small real capture at an operating dual-axis agrivoltaic plot. It includes tracker logs, a pyranometer, and a grid of PAR sensors at canopy height, with drone flights at four or five tracker angles within an hour around noon. Metrics are held-out sensor PAR error and albedo consistency across flights.
The downstream test feeds each method's PAR map into the unchanged tilt optimizer, then evaluates the resulting schedule against sensor-measured (real) or path-traced (synthetic) leaf PAR. For each crop threshold, we record two quantities: the energy produced, starting from the paper's 1137.3 kWh tracking baseline (1129.6 kWh at 80%, 613.1 kWh at 88%), and how often the crop constraint is actually violated. The project succeeds if the leaf-level model either recovers energy at high thresholds without more constraint violations, or reveals violations the flat model hides. Either outcome changes how much fossil backup such a farm needs. A null result, where flat and 3D models produce the same schedules, would also be informative: it would say that panel control does not need canopy vision.
References
- UAV3DCrop: Benchmarking 3D Reconstruction in Repeated Multi-Angle UAV Crop Surveys
- CropCraft: Complete Structural Characterization of Crop Plants from Images
- Optimizing Dual-Axis Solar Panel Operation in an Agrivoltaic System and Implications for Power Systems
- Extracting Triangular 3D Models, Materials, and Lighting From Images
- Differentiable Shadow Mapping for Efficient Inverse Graphics
- On Neural BRDFs: A Thorough Comparison of State-of-the-Art Approaches
- Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation
- Retrieval of Surface Solar Radiation through Implicit Albedo Recovery from Temporal Context