Image segmentation means labeling every pixel of an image ("crack" or "not crack"). It is the only way to measure how long and how wide a crack is. Its weakest point is the thin, faint early-stage crack, which is also the one an inspector most wants to catch. Hand-drawn pixel masks for such cracks are expensive and unreliable. Cheaper labels taken from image-level classifiers produce noisy masks that miss exactly these fine structures. Video of a loaded structure offers a supervision signal that nobody uses. Physical damage does not reverse. A pixel that is cracked at time $t$ is still cracked at every later time, and later frames usually show the same crack wider and with more contrast. The late, easy frames can therefore tell us, after the fact, where the early, hard crack was.
We propose to turn this irreversibility into training labels. On aligned image sequences of a growing crack, we run a per-frame segmenter. We then project each pixel's predicted probability onto the nearest non-decreasing sequence over time, threshold the result into pseudo-labels (machine-generated stand-ins for human masks), and retrain. The final model segments single images, so it needs no video at deployment. The claim is specific. Monotone-in-time pseudo-labels should recover thin, low-contrast cracks better than three alternatives: label refinement from static images, temporal smoothing that treats past and future symmetrically, and post-hoc temporal filtering at test time. The same sequences also give crack growth rates, which current pixel-level crack pipelines cannot measure.
Core Claim and How It Differs From Prior Work
The obvious idea at this intersection is to bolt a temporal module onto a segmentation network, and that has been done. STFCN places an LSTM (a recurrent unit that carries memory from frame to frame) on downsampled FCN features and gains under a point of mIoU on CamVid street scenes. Spatio-Temporal Attention for Consistent Video Semantic Segmentation in Automated Driving lets the current frame attend to two past frames with a fixed decay. That method improves temporal consistency on BDD100k by 9.20 points, but it degrades when the window grows beyond three frames because objects move. Both methods target rigid, moving objects in driving scenes. Both treat time as a source of smoothing, and both use only the past. Our setting is the reverse in every respect. The scene is static or can be registered, the target only grows, and the most informative evidence lies in the future. We use time as a direction-aware constraint that produces labels, rather than as an extra input feature. A static-scene, monotone-growth constraint avoids the long-window failure the driving work reports. That failure was caused by motion misalignment, not by the length of history itself.
Two other lines constrain and motivate the design. Crack-Net marches a U-Net forward through 150 steps of simulated phase-field crack growth. This shows that crack evolution is learnable, but it works only on clean synthetic fields and forecasts rather than segments camera images. We reuse its simulation data as a testbed with dense ground truth, and we do not attempt forecasting. Binocular Video-Based Automatic Pixel-Level Crack Detection and Quantification measures crack length, width (0.6-pixel width error with a median plumb-line method) and angle from stereo video. It processes each frame independently, however, and cannot report growth rates. We adopt its measurement heads and add the temporal dimension it lacks.
On the label-noise side, CAC refines pseudo-labels from patch-level classifiers by splitting them into confident and unconfident sets for co-training. Its authors acknowledge that it misses very thin cracks and breaks crack connectivity. Mind marginal non-crack regions (CIRL) identifies low-contrast boundary zones as the main source of error and handles them with clustering losses on single images. Both papers provide motivating evidence: the failure is concentrated in thin, ambiguous pixels. Both work from one image at a time. Our proposal supplies evidence that a single image cannot contain.
Mechanism
We register the frames of a sequence, either with a fixed lab camera or with a planar homography fitted to the concrete surface. A per-frame segmenter then gives probabilities $p_t(i)$ for pixel $i$ at frames $t=1,\dots,T$. For each pixel we compute the monotone projection
$$ q(i) = \arg\min_{q_1 \le q_2 \le \dots \le q_T} \sum_{t=1}^{T} \big(q_t - p_t(i)\big)^2, $$
which is isotonic regression. It is solvable exactly in linear time by the pool-adjacent-violators algorithm. When an early frame is confidently uncracked but a later frame is confidently cracked, the projection places the onset somewhere in between. When a faint early response is followed by strong later responses at the same location, the projection raises the early value. Pseudo-labels are $q_t(i) > \tau$, where $\tau$ is a confidence threshold. We exclude pixels whose $q$ lies within a margin of $\tau$, borrowing CAC's confident/unconfident split.
Registration is never perfect, and opening cracks shift their edges slightly. We therefore also train with a neighborhood-relaxed hinge penalty
$$ \mathcal{L}_{\text{mono}} = \sum_{t,i} \max\big(0,\; p_t(i) - \max_{j \in N_r(i)} p_{t+1}(j)\big), $$
where $N_r(i)$ is the window of radius $r$ pixels around $i$. This penalty flags a pixel only if crack probability disappears from its entire neighborhood at the next frame. The student network is a lightweight single-frame segmenter. We use EfficientCrackNet's architecture so that deployment cost stays fixed. It is trained on the pseudo-labels plus $\mathcal{L}_{\text{mono}}$ for two or three rounds of self-training.
Research Program
The program has three parts. The first is a controlled testbed. We render the 554 phase-field sequences from Crack-Net as camera-like images, texturing them with concrete patches taken from Crack500 and CrackSeg5k backgrounds and adding blur, illumination drift and sensor noise. This gives exact pixel ground truth at every step and exact growth rates, and it lets us sweep crack contrast, registration error and frame spacing. The second part is real data, which no file resource provides and which we must collect. These are fixed-camera, and where possible stereo, recordings of concrete beams and slabs under stepped loading. Dense masks will be drawn on a sparse subset of frames for evaluation only, and crack gauges will give width ground truth. The third part is a transfer test. Student models trained on sequence-derived pseudo-labels are evaluated on the standard static benchmarks Crack500, DeepCrack, CFD and CrackTree200. This asks whether temporal supervision yields a better single-image segmenter, not just a better video tool.
The analysis focuses on where the gain comes from. We report recall stratified by crack width (one-pixel, two- to three-pixel, wider) and by time-to-visibility, meaning how many frames before a crack is confidently detected per frame the method first labels it. We also report the rate of monotonicity violations, and the error in growth rates ($\Delta$length$/\Delta t$ and $\Delta$width$/\Delta t$) computed by applying the binocular paper's skeleton and plumb-line measurements to each frame's mask.
Baselines
The naive reference point is a per-frame segmenter trained only on whatever sparse human masks or classifier-derived labels are available, applied independently to every frame. We use EfficientCrackNet EfficientCrackNet, a 0.26M-parameter model combining edge filters, light attention and MobileViT blocks. It reports 81.33% mIoU on Crack500 and 87.10% on DeepCrack with full supervision. Those figures are the fully supervised ceiling we hope to approach without dense labels. To match the student, it uses the same architecture, input resolution and training schedule.
The strongest existing weak-supervision method is CAC. It reports an ODS (best F-measure at a single dataset-wide threshold) of 71.01% on DeepCrack and 25.31% on CFD when initialized from GPLL pseudo-labels, with gains of up to 15.39% over GPLL on Crack500. We will run CAC from the same initial pseudo-labels, on the same frames, with the same backbone and an equal hyperparameter search budget. The only difference is that CAC sees frames as independent images. Beating it means higher ODS and higher one-pixel-width recall on the static benchmarks, plus higher per-frame F1 on our sequences. We also run CIRL's boundary-clustering loss on the same student as a second single-image competitor. It reports F1 gains of 1.86 points on Crack500 and 2.34 on CrackTree200 over its JTFN base.
The strongest temporal competitor is the spatio-temporal attention module of Spatio-Temporal Attention for Consistent Video Semantic Segmentation, inserted into a SegFormer backbone with $T=3$. On BDD100k it reports +9.20 points of temporal consistency and up to +1.76 mIoU over single-frame models. It has no reported result on crack data. We train it on the same pseudo-labels the per-frame baseline uses and give it the same registered frames.
Two ablations isolate the key ingredient. The first is a symmetric temporal smoother: the per-pixel temporal median or mean of $p_t$ replaces the isotonic projection, so it uses the same frames but ignores the direction of time. The second is a time-reversed control, in which the projection enforces non-increasing probability. If neither ablation loses to the full method, the claimed mechanism is wrong.
Why not just…?
Why not just run a per-frame segmenter and apply the monotone projection at test time? This is the most serious threat. On registered sequences, post-hoc isotonic filtering uses exactly the same physical prior, and it may match our method on video. We include it as a baseline and expect it to be competitive there. The proposal's claim differs. Training on projected labels should improve the single-frame model itself, which matters for one-off inspection photos where no sequence exists. The transfer test on Crack500, DeepCrack and CFD settles this. If the student is no better than the baseline on static images, the training contribution reduces to a post-processing trick, and we would report it as such.
Why not just add temporal attention, as in the driving work? That module was designed to stop flicker on moving objects. It uses only past frames with fixed decay and degrades beyond three frames (Spatio-Temporal Attention). For a crack that only becomes visible later, past frames carry no extra evidence. The module also requires sequences at inference and costs speed: its larger variant runs at 6–7 FPS. The width-stratified recall comparison shows directly whether looking backward captures any of the early thin-crack gain.
Why not just spend the budget on more dense annotation, or a better static loss such as CIRL's? Annotators face the same problem the model does. A one-pixel crack in an early frame is often invisible to a person until they see the later frame, so dense labels for early damage are themselves noisy. CIRL and CAC both report that their residual errors sit in thin, low-contrast regions. The comparison against CIRL at matched labeling cost, together with the synthetic testbed where true onset times are known, will show whether temporal evidence recovers pixels that static methods and humans miss.
Decisive Checks
On rendered phase-field sequences, the method must detect cracks earlier and with higher one-pixel recall than both CAC and the symmetric smoother. It must also lower growth-rate error relative to per-frame measurement with the binocular pipeline's quantification heads. On static benchmarks, the student trained on sequence pseudo-labels must exceed CAC's reported 71.01% ODS on DeepCrack under matched initialization. Through all registration-error levels in the sweep, the time-reversed control must perform worse than the forward constraint. If the gain survives only on video and vanishes on static images, we will conclude that the useful object is the post-hoc filter, not the training scheme. That would still be a useful but smaller result.
References
- STFCN: Spatio-Temporal FCN for Semantic Video Segmentation
- Spatio-Temporal Attention for Consistent Video Semantic Segmentation in Automated Driving
- Crack-Net: Prediction of Crack Propagation in Composites
- Binocular Video-Based Automatic Pixel-Level Crack Detection and Quantification Using Deep Convolutional Neural Networks for Concrete Structures
- CAC: Confidence-Aware Co-Training for Weakly Supervised Crack Segmentation
- Mind marginal non-crack regions: Clustering-inspired representation learning for crack segmentation
- EfficientCrackNet: A Lightweight Model for Crack Segmentation