A targeted sprayer, a machine that opens a nozzle only over weeds instead of coating the whole field, needs two things from its camera. It needs a pixel-accurate map of weed foliage, so it can decide how much herbicide to release. It also needs the location of each plant's center or stem, so it can time when the valve opens. Today both come from segmentation networks trained on hand-traced pixel masks, and tracing them is slow. Full polygon tracing takes up to 79 seconds per object, while clicking a single point costs roughly the same as an image-level tag (What's the Point). We propose to train a compact U-Net from one click per plant, placed on its stem, together with a vegetation-versus-soil mask that is nearly free to compute. A U-Net is an encoder-decoder network whose "skip connections" copy high-resolution early features straight into the decoder. We then judge the result by how much herbicide a simulated sprayer would put on weeds versus on crop and soil, not only by mask overlap.
The idea rests on one observation about farm images. Weak-supervision methods built for natural photos spend most of their effort on finding an object's full extent, because classifier heatmaps light up only the most distinctive part of an object. In crop fields, extent is the easy part. Separating vegetation from soil is close to solved: a fully supervised vegetation-versus-soil model reaches 87.7% mIoU on CropAndWeed (CropAndWeed), and a color index or the near-infrared band gives a usable vegetation mask with no labels at all. The hard part is deciding which plant each green pixel belongs to, because crop and weed leaves touch and share the same color. A stem click answers exactly that question for one pixel per plant, and it is also the coordinate the sprayer needs for timing.
Core Claim
Weakly supervised crop-weed segmentation is best treated as a vegetation-partitioning problem, not an extent-recovery problem. Under that framing, a U-Net trained from stem clicks plus a free vegetation mask should approach a fully supervised segmenter on spray-relevant outcomes at a small fraction of the labeling time, and it outputs stem locations as a by-product.
The mechanism is that the network assigns each unlabeled vegetation pixel to a clicked stem along the cheapest path through vegetation, where path cost is measured in the U-Net's own high-resolution decoder features. The network is then retrained on that assignment, and the two steps alternate. Each round sharpens the features, and sharper features give a better assignment. The claim is falsifiable in a specific way. If assigning pixels to the nearest click in plain image distance, with no learning, does as well as the learned assignment even where canopies overlap, the learned component is unnecessary.
Research Program
The model is a lightweight U-Net with a MobileNet-class encoder, following the backbone swaps in Motta et al., and two output heads. One head predicts per-pixel class (soil, crop, or weed species). The other predicts a stem heatmap whose peaks give actuation coordinates.
Supervision comes from three sources. The first is a vegetation mask: excess green (a color index, $2G-R-B$) thresholded on RGB data, or a near-infrared vegetation index on multispectral data. Pixels it marks as soil are treated as reliable soil labels, and pixels it marks as vegetation are left unlabeled. The second is a stem click per plant, carrying a class label. A small disk around each click receives a cross-entropy loss on the class head and a Gaussian target on the stem head. The third is a pseudo-label for every unlabeled vegetation pixel $p$. That label is the class of the stem
$$s^*(p)=\arg\min_{s}\ \min_{\pi:\,s\to p,\ \pi\subset V}\ \sum_{(q,q')\in\pi}\big(\lVert f(q)-f(q')\rVert+\lambda\big),$$
where $V$ is the vegetation mask, $\pi$ ranges over pixel paths that stay inside vegetation, $f$ is the U-Net's highest-resolution decoder feature map (the level fed by the first skip connection), and $\lambda$ is a small per-step cost that breaks ties toward nearby stems. The assignment is computed with a multi-source shortest-path pass on a downsampled grid. Pseudo-labels are refreshed every few epochs and weighted by the margin between the best and second-best stem distance, so ambiguous pixels at leaf contacts count less.
This differs from the closest prior work in specific ways. SL-Net trains DeepLab-V3+ on maize and weeds from freehand scrawls. It builds pseudo-masks once, with fixed heuristics: excess green, Otsu thresholding, GrabCut, and a conditional random field (a classical smoothing model). We use single clicks rather than strokes, and our assignment is learned and refined during training rather than frozen. PCAMs feeds points into classifier heatmaps and expands them with a separate affinity network, IRNet. The authors report choppy edges and missed small objects. We drop the heatmap step entirely, because the vegetation mask already supplies extent, and we get affinities from the segmenter's own skip features instead of a second network. What's the Point uses a generic objectness prior to stop points from spreading into background. That prior would mark crop and weed alike as foreground, and our vegetation mask with stem-based partition takes its place. The tea-garden U-Net (Improved U-Net with skeleton refinement) extracts weed centers by thinning fully supervised masks, using 5,366 dense labels. The laser-weeding work (WSD) regresses stems directly but outputs no masks, so it cannot meter dose. Our stem head takes both outputs from one cheap label.
Evaluation adds a spray simulation. Predicted masks are rasterized onto a boom of virtual nozzle cells, and the cell size is swept to cover plausible nozzle footprints. A cell fires if its predicted weed fraction exceeds a threshold. Against ground-truth masks we report weed coverage (the fraction of true weed pixels sprayed), off-target fraction (sprayed area on crop or soil), and crop-hit rate. Mask mIoU is reported too, but the spray metrics decide our conclusions, because mIoU can reward accurate leaf edges that a nozzle cannot resolve.
Baselines
The do-nothing reference is broadcast spraying, which gives 100% weed coverage with off-target fraction equal to the non-weed area. A slightly smarter reference sprays wherever the vegetation mask says green, which reaches weed coverage near 100% but sprays every crop plant. Both are computed exactly from the ground truth and anchor the spray metrics.
The strongest label-free competitor is nearest-click partitioning. We take the same vegetation mask, assign each green pixel the class of its nearest stem click in straight-line image distance, and use the result either directly or as fixed pseudo-labels to train the identical U-Net. It is the strongest competitor because at early growth stages, when plants do not touch, it may be nearly perfect. It receives the same clicks and vegetation mask and the same U-Net, with the same training schedule. Beating it means a significant gain in off-target fraction at matched weed coverage on the overlapping-canopy strata. We do not expect to beat it on sparse seedlings.
The strongest existing weak-supervision method is a PCAMs-style pipeline (PCAMs): point-supervised class heatmaps, IRNet random-walk refinement, then a segmenter trained on the result. We run it with the same clicks, with both its original DeepLabv3+ and our U-Net as the final segmenter, and give it the same hyperparameter search budget. The only reported number is on natural images, 70.5% mIoU on PASCAL VOC 2012, recovering 89% of full supervision. We found no reported result on Lacuna for any point-supervised method on CropAndWeed or WeedsGalore. We also run an SL-Net-style heuristic pseudo-label pipeline (SL-Net), seeded with our clicks in place of scrawls. That paper reports 87.30% MIoU on its own maize dataset, which is not comparable to ours.
The fully supervised upper references are the same U-Net trained on all dense masks, and a fully supervised U-Net trained on a dense-mask subset whose labeling time equals that of the click annotations. Published fully supervised numbers anchor these. CropAndWeed reports 87.7% mIoU on Coarse1, 76.5% maize IoU on Fine24, and stem detection of 73.1% with 29.1 pixels mean error (CropAndWeed). WeedsGalore reports DeepLabv3+ on multispectral input at 82.90% mIoU for 3 classes and 55.52% for 6 classes (WeedsGalore).
Ablations isolate the key ingredient by swapping the feature space used in path costs. We compare raw RGB, encoder bottleneck features, and high-resolution skip features. We also test fixed versus iteratively refreshed assignments, removal of the stem head, and color-index versus near-infrared vegetation masks.
Why not just…?
Why not just assign green pixels to the nearest click and skip learning? This is the real threat, and it may win on young seedlings. The material contains no evidence on how it fares when canopies interlock. We settle the question by stratifying test images by canopy contact, computed from ground-truth instance masks, and by using WeedsGalore's multiple growth stages as a natural overlap axis. If nearest-click partitioning matches our method in the high-contact strata, we drop the learned assignment. The contribution then shrinks to the click-plus-vegetation labeling protocol and the spray evaluation, and we report it that way.
Why not just prompt a promptable foundation segmenter with the stem clicks and train on its masks? Promptable segmenters have been used to produce masks from cheap prompts in other domains (prompt-optimized SAM). This is a credible competitor for pseudo-label generation. A stem click on touching plants is ambiguous, though: it could mean one leaf, one plant, or the whole green clump, and nothing in the file shows such models resolve crop-weed contact. We include click-prompted foundation-model pseudo-masks as an additional baseline feeding the same U-Net. If they win, our assignment step can start from them instead, which is a useful outcome either way.
Why not just draw boxes and run a fast detector? Boxes on irregular weeds include soil and crop leaves, so a box-driven nozzle sprays non-targets. Box centers miss stems by 4.33 pixels against 2.48 for a regressed stem in the laser-weeding study (WSD). We nonetheless train a detector baseline in the spirit of YOLO comparisons and spray each predicted box. The spray simulation then measures directly how much extra off-target dose boxes cost.
Why not just hand-trace a few dozen images and augment? On ACRE, 50 target-season images lifted a detector only from 0.34 to 0.42 mAP (ACRE). That suggests small dense sets cover too little variety, but it is a detection result, not a segmentation one. Our matched-time dense baseline tests this directly.
Proposed Experiments
The main benchmark is CropAndWeed. It ships stem positions and dense masks, so stem clicks come straight from its annotations and evaluation uses the dense masks. We run its Crop/weed and Fine24 granularities. The second benchmark is WeedsGalore, where we simulate clicks at an interior point of each instance near its base and take the vegetation mask from near-infrared. That lets us test whether spectral bands make the free mask, and hence the partition, more reliable. Its out-of-distribution Maize2024 split tests cross-season transfer. The WSD dataset (WSD) evaluates stem localization error for our stem head against the reported 2.15-pixel figure, recognizing that our head is trained without boxes.
Annotation cost is modeled with the ratios in What's the Point and checked by timing annotators on a 100-image subset. Every method is plotted as spray metrics against annotation hours.
The decisive checks are three. First, on high-contact strata, our method must beat nearest-click partitioning and click-prompted foundation-model masks on off-target fraction at matched weed coverage. Second, the skip-feature path cost must beat both raw-RGB and bottleneck-feature costs; otherwise the U-Net's skip connections are not doing the work we credit them with. Third, at equal annotation time, our method must match or exceed the matched-time dense U-Net and approach the all-dense U-Net on spray metrics. Our target is recovering most of its weed coverage at about one tenth of the labeling time.
We also report frames per second and memory on an embedded GPU. The published 21.2 frames/s for SL-Net was measured on other hardware and is cited only as context.
References
- What's the Point: Semantic Segmentation with Point Supervision
- The CropAndWeed Dataset: A Multi-Modal Learning Approach for Efficient Crop and Weed Manipulation
- Automatic COVID-19 and Common-Acquired Pneumonia Diagnosis Using Chest CT Scans
- A Weakly Supervised Semantic Segmentation Model of Maize Seedlings and Weed Images Based on Scrawl Labels
- PCAMs: Weakly Supervised Semantic Segmentation Using Point Supervision
- Research on Precise Segmentation and Center Localization of Weeds in Tea Gardens Based on an Improved U-Net Model and Skeleton Refinement Algorithm
- Towards Efficient and Intelligent Laser Weeding: Method and Dataset for Weed Stem Detection
- WeedsGalore: A Multispectral and Multitemporal UAV-Based Dataset for Crop and Weed Segmentation in Agricultural Maize Fields
- Object Detection and Segmentation of Power Equipment in Infrared Images via Improved YOLOv8 and Prompt-Optimized SAM
- Comparative Analysis of YOLOv9, YOLOv10 and RT-DETR for Real-Time Weed Detection
- The ACRE Crop-Weed Dataset for Benchmarking Weed Detection Models on Maize and Beans Fields