Continuous sign language has no clean pauses between words. The hands flow from one sign into the next, and the end of one sign overlaps the start of the following one, a blending that linguists call co-articulation. Current segmenters split the stream in one of two ways. Some label every frame as beginning, inside, or outside a sign using a model that sees the whole video. Others slide a fixed window and accept any class whose softmax probability exceeds 0.51. Neither approach can say how long it waits before committing to a boundary, or how often it invents one. Both fail in two recurring ways. They insert spurious boundaries inside long signs that contain a brief pause, and they miss boundaries between short or similar-looking signs.
Quickest change detection is the statistics of declaring, as early as possible and with a controlled false-alarm rate, that a data stream has switched regimes. We propose to treat each sign boundary as one such change, with one specific twist. When the hands depart from the current sign, the detector does not immediately declare a boundary. It first waits out a transient of unknown length, then tests whether the stream has settled into a new regime or resumed the old one. The resume branch is aimed at the over-segmentation failure. Accumulating evidence over time is aimed at the under-segmentation failure. The threshold becomes an explicit dial between how late a boundary is declared and how many false boundaries appear per minute of signing.
Core Claim
A causal detector built on three pieces should beat a causal frame tagger on natural co-articulated corpora when both are held to the same false-boundary budget, and the gains should concentrate on long signs with internal pauses and on pairs of kinematically similar signs. The first piece is a learned embedding of 3D hand and pose features. The second is a running reference for the current sign that is reset at each declared boundary. The third is a "transient-then-compare" likelihood-ratio statistic. Beyond the accuracy claim, the method yields something current segmenters do not produce at all: a measured curve of boundary delay against false-boundary rate, and a threshold set by calibration rather than by a 0.51 heuristic.
The closest existing bridges each leave a specific gap:
- Minimum Delay Moving Object Detection runs a sequential likelihood-ratio test on composed optical flow. It handles only a single stationary-to-moving onset.
- Unsupervised Change Detection using DRE-CUSUM applies cumulative-sum statistics to neural density ratios on activity and video data. Its authors leave multi-change guarantees and online operation open.
- A Contrastive Approach to Online Change Point Detection learns a discriminator between past and current data. It costs $O(t^2)$ per step, needs ad-hoc restarts, and assumes independent observations.
None of the three models an intermediate blended regime, and none distinguishes "changed and came back" from "changed to something new". Those two elements are what this proposal adds.
Research Program
Evidence features. We reuse the compact geometric features of Hands-On: Segmenting Individual Signs from Continuous Sequences: 3D hand meshes from HaMeR plus skeleton joint angles. That work shows these features outperform heavier I3D video features, so we take them as the evidence stream. A small causal temporal encoder maps each frame to an embedding $e_t$. It is trained contrastively from gloss labels so that frames of the same sign token cluster together and frames of adjacent tokens separate. Transition frames are excluded from the positive pairs.
Reference and per-frame evidence. The detector keeps one running reference for the current sign, a prototype $\mu$ averaged over frames since the last declared boundary. This reference is what removes the quadratic history cost of the contrastive detector: memory is constant within a segment and resets at each boundary. A lightweight discriminator, trained jointly with the encoder, turns the pair $(e_t, \mu)$ into a per-frame log-ratio $\ell_t$. This quantity measures how much more likely frame $t$ is under "a different sign" than under "the current sign".
The detection statistic. Let $j$ be a candidate frame where the hands begin to leave the current sign, and let $k \ge j$ be a candidate frame where they settle. The transient-then-compare statistic at time $t$ is
$$ S_t = \max_{t-W \le j \le k \le t} \Big[ \sum_{i=k}^{t} \ell_i \;-\; \lambda\,(k-j) \Big], $$
with the following terms:
- $W$ is a maximum look-back window.
- The frames between $j$ and $k$ are treated as an uninformative transition and contribute nothing to the evidence.
- $\lambda \ge 0$ is a small per-frame penalty on transition length, which stops the maximizer from explaining away genuine within-sign motion as a long transition.
A boundary is declared at the first $t$ with $S_t > h$. The retrospective boundary estimate is the maximizing $j$. This separates when we commit (the detection delay) from where we place the boundary (the localization quality), which lets us report both.
Why the resume test matters. Suppose a long sign contains an internal pause. The pause frames carry some departure evidence, but after the pause the frames return to the old prototype. The $\ell_i$ after $k$ are then negative and $S_t$ falls back. A plain cumulative sum on a boundary score would already have fired during the pause. Conversely, when two consecutive signs differ only slightly, small positive $\ell_i$ keep accumulating after $k$ until the threshold is crossed. A single-frame threshold would never fire in that case.
Setting the threshold. Human motion is strongly autocorrelated, so the independence-based guarantees of the contrastive detector do not apply. The martingale-based guarantees of Reducing sequential change detection to sequential estimation would require a null model of within-sign motion that we cannot justify. We therefore do not claim a theoretical false-alarm bound. Instead, we calibrate $h$ empirically. We run the detector over held-out, annotated within-sign intervals from signers not seen in training, and choose $h$ so that spurious boundaries per minute of within-sign signing stay at a target rate. We then check whether that rate transfers to new signers and new languages. This is a weaker claim than a guarantee, and we state it as such. It is still more than the 0.51 rule offers, since that rule controls nothing.
Baselines
Naive reference points. The first is the sliding-window softmax rule of A Transformer Model for Boundary Detection in Continuous Sign Language: a 50-frame window, stride 1, a 0.51 threshold, and blanking of repeated detections. We retrain it on our features and data. On synthetic concatenations of RKS-PERSIANSIGN it reports 9 false recognitions over 10,000 signs, and 8 over 700 signs on ASLLVD. Its authors state that it was never tested on natural co-articulation. The second is a velocity-drop detector: threshold the smoothed hand speed, using the prosody cue from Linguistically Motivated Sign Language Segmentation. These two mark the floor.
Strongest existing method. This is Hands-On. On the Public DGS Corpus it reports a sign-level F1 of 0.86 and an IoU of 0.76, against a previous 0.63. It sees the whole clip, so it is an upper reference rather than a direct competitor. We run it in two forms under identical features, splits, and hyperparameter search budget:
- Offline: as published.
- Causal: attention masked to past frames plus a fixed lookahead $L$, with its frame outputs decoded by a threshold with hysteresis. Both the threshold and the hysteresis are tuned on the validation set.
Criterion for beating it. We sweep lookahead and threshold to trace the causal tagger's full curve of delay against false-boundary rate. Our method beats it if it achieves higher sign-level F1 and lower mean delay at the same calibrated false-boundary rate, across most of that curve. We will also report how much of the gap to the offline 0.86 our method closes at a delay budget of a fraction of a second. On BSLCorpus the offline reference point is the 68.68 mean boundary F1 (mF1B) reported by Sign Language Segmentation with Temporal Convolutional Networks, and 58.48 mF1B on BSL-1K.
Off-the-shelf change detectors. We run the contrastive detector (with periodic restarts, as its authors do) and DRE-CUSUM on the same embeddings $e_t$. This tests whether any sequential statistic helps, or whether our specific one does. As a sanity check that our implementation of the contrastive detector is competent, we reproduce it on WISDM, where it reports an average detection delay of 16.4 steps.
Ablations isolating the key ingredient. We run three variants:
- No resume test: set $k=j$, which gives a plain cumulative sum on $\ell_t$. This is essentially a leaky temporal smoother.
- Fixed-reference: replace the running prototype with a global "inside a sign" class score.
- Uncalibrated: a fixed threshold instead of the calibrated one.
Why not just…?
Why not just threshold a causal frame tagger with smoothing and hysteresis? This is the real threat. A cumulative sum with no resume test is close to exponential smoothing followed by a threshold, and a well-tuned causal tagger may capture most of the benefit. We do not argue this away. The causal-tagger baseline and the no-resume ablation exist precisely to test it. The proposal's specific bet is narrower. On signs with internal pauses, a smoother cannot distinguish a pause from a boundary until the sign resumes. By then it has already fired, unless its lag is so long that it misses short signs. If matched-rate curves show no gain on the pause and similar-sign subsets, the contribution reduces to calibration alone, and we will report it that way.
Why not just calibrate the 0.51 softmax threshold on held-out data? Calibration fixes the false-alarm dial but not the evidence. The published results already show softmax confidence collapsing on harder data: average confidence falls to 0.59 on ASLLVD in Word separation in continuous sign language using isolated signs and post-processing, and to 0.6820 in A non-anatomical graph structure for boundary detection in continuous sign language. This happens even without real co-articulation. A per-window decision with a calibrated threshold still cannot accumulate weak evidence or recognize a resumed sign. We include the calibrated sliding window as an extra baseline to measure exactly this.
Why not just run the offline model with a short lookahead buffer? For applications that tolerate latency, the offline model is the right choice, and we treat it as the ceiling. Two considerations limit it. First, the multi-stage temporal network of Sign Language Segmentation with Temporal Convolutional Networks refines predictions bidirectionally over the whole sequence. Second, its authors report over-segmentation of fingerspelling and short signs, which the extra context does not fix. The lookahead sweep of the causal Hands-On variant is exactly this alternative, and it shows the latency each model needs to reach a given F1.
Why not just apply an existing change detector directly to pose features? Existing detectors assume an abrupt switch between two regimes. The contrastive detector also assumes independent frames and pays a quadratic cost. Running them on the same embeddings shows whether the transition-aware statistic, rather than the embedding, is responsible for any gain.
Proposed Experiments
Datasets and metrics. The main evaluation uses the Public DGS Corpus with signer-disjoint splits. We report sign-level F1 and IoU as in Hands-On, and add three online metrics: mean detection delay, spurious boundaries per minute inside annotated signs, and the full delay-versus-false-boundary curve. Secondary evaluations use BSLCorpus and BSL-1K, measured by mF1B.
Calibration transfer. We fix the threshold on DGS and measure the realized false-boundary rate zero-shot on two corpora:
- PHOENIX14, where the temporal network drops to 46.75 mF1B cross-lingually versus 71.50 when trained in-domain.
- MEDIAPI-SKEL, where the linguistically motivated segmenter reports a ROC-AUC of 0.76.
Failure-mode subsets. Two subsets are constructed from the annotations. The pause subset contains long signs whose hand speed drops below the within-sign median for several frames. The similar-pair subset contains consecutive gloss pairs whose mean pose features lie close together. These subsets test directly the two failure modes documented by Hands-On.
Synthetic-versus-natural contrast. We also run all methods on concatenated RKS-PERSIANSIGN streams. We expect every method to score near ceiling there, with the resume test's advantage appearing only on natural corpora. That pattern would show that co-articulation, not boundary detection in general, is what the method addresses.
Decisive checks. There are three:
- At a matched calibrated false-boundary rate on DGS, the full detector must produce fewer spurious boundaries on the pause subset and fewer misses on the similar-pair subset than both the causal tagger and the no-resume ablation.
- The calibrated rate must hold within a modest factor on held-out signers and on PHOENIX14.
- The full method must reach a better point on the delay-versus-F1 curve than the lookahead-swept causal tagger.
If the first check fails, the resume idea is wrong. If only the second holds, the paper becomes a calibration study of online sign segmentation, which is still a result the field currently lacks.
References
- Minimum Delay Moving Object Detection
- Unsupervised Change Detection using DRE-CUSUM
- A Contrastive Approach to Online Change Point Detection
- Hands-On: Segmenting Individual Signs from Continuous Sequences
- Reducing sequential change detection to sequential estimation
- A transformer model for boundary detection in continuous sign language
- Linguistically Motivated Sign Language Segmentation
- Sign Language Segmentation with Temporal Convolutional Networks
- Word separation in continuous sign language using isolated signs and post-processing
- A non-anatomical graph structure for boundary detection in continuous sign language