Most systems that combine a 3D sensor with a camera begin by flattening one of them. Aerial LiDAR (a laser scanner that returns a cloud of 3D points) is usually collapsed into a canopy height model, which keeps only the height of the top surface at each ground pixel, and then stacked with the photograph. Plant point clouds are rendered into top-down or side views. Each of these flattenings is a small, hand-written program, and each one discards information in a way nobody tuned. Height models hide the understory beneath dominant crowns (Xiang et al.; Wielgosz et al.). Top-down views undercount overlapping leaves (DIY rosette phenotyping). Flat spectral images cannot tell a shadowed leaf from a diseased one without orientation information (Silwal et al.). The flattening step is where the damage happens, and it is also the one step that neural network training never touches.
We propose to search over that step directly. A language model writes and repairs candidate "projection programs": code that turns a raw point cloud into a stack of image-aligned channels. Example channels include height-slice densities, return-order ratios, local surface normals, and vertical-profile statistics. A genetic algorithm (a population-based search that keeps the best candidates and recombines them) selects among these programs, scoring each by how well a fixed, cheap image model classifies the fused LiDAR-plus-photo stack. The output is a readable program that states which 3D structure matters for the task. The hypothesis is that this program lets an ordinary 2D network match or beat heavier point-cloud fusion models at a fraction of their cost.
Core Claim
On PureForest, a large paired aerial LiDAR and orthophoto dataset for tree species classification, the unimodal baselines already show the cost of losing 3D information. RandLA-Net on points reaches 81.0% weighted F1, while ResNet18 on imagery reaches 74.6% (PureForest). The authors leave fusion open, and we found no reported fusion result on Lacuna for PureForest.
Our claim is specific. An evolved projection program, fused early with the orthophoto and read by a small 2D network, will beat both the naive height-model stacking baseline and a parameter-tuned family of hand-designed projections. It will also come within noise of, or exceed, a learned two-backbone fusion model, while using markedly less training compute per run. The new element is not "LLMs evolve code," and not "evolution tunes fusion." The new element is placing the search at the geometric interface where 3D becomes 2D, and treating that interface as open-ended code rather than a fixed menu of operators.
Relation to Prior Work
Two papers supply the method. A2DEPT evolves whole solver programs with an LLM, using a tree of program variants, simulated-annealing-style selection, and a self-healing loop that fills in missing functions. It reports a 9.8% reduction in normalized optimality gap over prior LLM heuristic-design methods on combinatorial benchmarks. Its evaluation is confined to routing and scheduling, and it documents a "partial migration" failure, where the LLM changes one component and breaks the functions that depend on it. We reuse its search loop. We sidestep partial migration by fixing a narrow interface contract: the program takes a points array and a ground grid and returns an H×W×C raster. Unit tests check shape, alignment, and determinism before any fitness evaluation.
Accelerated GPHH via Agentic AI shows that profiling-guided LLM refactoring cut a simulator's runtime by 85% with checksum-verified correctness, but only with a human navigator. That result constrains us. Autonomous code changes need automated invariants, which is why our contract tests and a numerical check against a reference implementation act as hard gates.
Two papers mark the nearest neighbours we must differ from. EvoFMVC evolves fusion topologies, but over a fixed set of operators acting on abstract probability vectors, never on raw geometry. ES-RBMO tomato segmentation uses a metaheuristic to tune scalar hyperparameters of a fixed 3D network. Neither can invent a new geometric feature.
The motivating evidence that geometry-aware fusion pays comes from the wheat nitrogen study. That study injected depth into an intensity-hue-saturation color transform and reached R² = 0.74 for leaf-layer nitrogen, but it admits its hand-built features miss leaf inclination. Its argument that pixel-level fusion beats feature-level fusion is exactly what our design tests: our programs produce early, pixel-aligned fusion, and we compare them against a learned intermediate-fusion model.
Research Program
The search space is Python/NumPy programs that operate on a PureForest patch's point cloud and emit channels on the orthophoto grid. The seed population contains the canopy height model, a height-sliced density stack, and a few intensity and return-number summaries.
Each generation, the LLM proposes three kinds of change:
- small edits to constants and bin choices;
- structural mutations, such as new per-column statistics, neighbourhood normals, or occlusion-aware "visible from above" masks;
- semantic crossover between two parent programs.
The mutation prompt includes the fitness breakdown, meaning the per-class F1 scores and the most-confused species pairs. This lets the model aim at specific failures, for example conifer pairs that look identical from above.
Fitness is the validation weighted F1 of a fixed ResNet18 trained for a short, fixed schedule on a training subset. The input is the orthophoto concatenated with the program's channels. We add a small penalty for channel count and runtime. Keeping the downstream learner cheap and fixed is what makes the search affordable; it answers the evaluation-cost bottleneck that both A2DEPT and the GPHH work flag. Programs that fail the contract tests go through a bounded repair loop driven by the traceback, then are discarded.
After the search, the top five programs are retrained with the full schedule and evaluated once on held-out test sites.
A second, smaller study checks that the idea is not specific to forests. Using Pheno4D, we evolve programs that render plant point clouds into multi-view 2D channels for leaf-length estimation. This asks whether an evolved projection lets a 2D model approach native 3D processing on dense plant structure, where self-occlusion is severe.
Baselines
Naive reference: orthophoto alone. ResNet18 on the orthophoto, which PureForest reports at 74.6% weighted F1. We rerun it on our splits.
Naive fusion: canopy height model stacked as a fourth channel. This is the common practice and the most direct embodiment of the 2D projection limitation. Beating it shows that the choice of projection matters.
Strongest unimodal method: RandLA-Net on points. PureForest reports 81.0% weighted F1 and 80.3% overall accuracy.
Strongest fusion competitor: a learned two-backbone model. We will build a ResNet18 image branch and a RandLA-Net point branch fused by concatenation or cross-attention before the classifier. We found no reported number on Lacuna for this on PureForest, so we implement it ourselves. It receives the same total GPU-hour budget that the evolutionary search consumes, spent on hyperparameter tuning.
Parameterized search baseline: Bayesian optimization. This baseline searches a hand-designed projection family: slice count, slice bounds, grid resolution, and which of a dozen standard statistics to include. It gets the same number of fitness evaluations as our search.
Ablations. These isolate the key ingredients:
- an LLM refinement loop with a single lineage and no population;
- evolution with the confusion-pair feedback removed from prompts;
- one-shot LLM generation of 50 programs with the best one selected.
What counts as winning. Beating the canopy-height-model and Bayesian-optimization baselines by at least two F1 points on test sites, averaged over three seeds, supports the central claim. Matching the learned two-backbone model within its seed variance, at lower per-run training cost, supports the efficiency claim. Exceeding it would be the strong outcome.
Why not just…?
Why not just stack the canopy height model with the image? It is the default and costs nothing. But it is precisely the 2.5D surface that Xiang et al. and Wielgosz et al. show hides intermediate and understory layers. The PureForest gap between points and imagery suggests vertical structure carries species signal. It is a direct baseline here. If it closes most of the gap to RandLA-Net, the search is unnecessary, and we will report that.
Why not just train an end-to-end LiDAR-camera fusion network? This is the real threat. With enough tuning it may simply win. We give it an equal compute budget and report both accuracy and cost. The proposal survives a tie only on efficiency and inspectability grounds, and we say so. It fails if the learned model wins clearly at similar cost.
Why not just run AutoML over a menu of projection features? If the useful features are already in a menu a human would write, Bayesian optimization should find them more cheaply than an LLM. A2DEPT argues that template-bound search cannot make structural changes, but that evidence comes from combinatorial solvers, not geometry. The matched-evaluation Bayesian-optimization baseline decides this directly. We will also inspect whether the winning programs contain operations outside the menu.
Why not just prompt an LLM once for good features? The one-shot best-of-50 ablation measures this. The evidence that iteration matters (A2DEPT's maintenance loop, and the checksum loop in the GPHH work) concerns executability, not feature quality. So this ablation may turn out closer than we expect.
Proposed Experiments and Decisive Checks
All PureForest experiments use site-level splits, so that no forest contributes to both search and test data. This guards against the search overfitting to local texture. Metrics are weighted F1, overall accuracy, per-class F1 for the most-confused species pairs, and training GPU-hours per run.
The decisive result is a comparison of the evolved program against the Bayesian-optimization baseline and the learned fusion model on the held-out sites.
Robustness to misalignment. PureForest's LiDAR and imagery can be up to three years apart. We report accuracy separately for patches with large and small acquisition gaps, to test whether early, pixel-aligned fusion is fragile under misalignment, as the feature-fusion literature predicts.
Interpretability. We will read the winning programs and test whether their added channels specifically fix the species pairs that the image-only baseline confuses.
Generality on plants. On Pheno4D, the reference point is LatticeNet's reported 8.9 mm mean leaf-length error on tomato. An evolved 2D projection that approaches this would show the approach transfers to close-range plant geometry. A large remaining gap would mark where flattening cannot be rescued by a better program.
References
- Automated forest inventory: Analysis of high-density airborne LiDAR point clouds with 3D deep learning
- SegmentAnyTree: A sensor and platform agnostic deep learning model for tree segmentation using laser scanning data
- A “Do-It-Yourself” phenotyping system: measuring growth and morphology throughout the diel cycle in rosette shaped plants
- Towards an AI-Driven Cyber-Physical System for Closed-Loop Control of Plant Diseases
- PureForest: A Large-Scale Aerial Lidar and Aerial Imagery Dataset for Tree Species Classification in Monospecific Forests
- $A_2$DEPT: Large Language Model–Driven Automated Algorithm Design via Evolutionary Program Trees
- Accelerated Genetic Programming Hyper-Heuristics for Simulation-Based Scheduling via Agentic AI
- EvoFMVC: Trusted Federated Multi-View Clustering with Evolutionary Fusion
- Tomato Stem and Leaf Segmentation and Phenotype Parameter Extraction Based on Improved Red Billed Blue Magpie Optimization Algorithm
- Spectrum- and RGB-D-Based Image Fusion for the Prediction of Nitrogen Accumulation in Wheat
- Pheno4D: A spatio-temporal dataset of maize and tomato plant point clouds for phenotyping and advanced plant analysis