Stories, clinical notes and news reports are not told in chronological order. A patient summary moves back and forth between today's admission, last year's surgery and a childhood illness. To reconstruct what happened when, a system has to do two things. It must cut the text into stretches that each describe one period of time, and it must place those stretches on a timeline so that the ordering never contradicts itself (A before B, B before C, but C before A). Current systems do these steps separately. First they draw hard, certain cuts, then they solve an ordering puzzle over the pieces. As a result, a wrong cut can never be repaired by what the ordering step learns. Separately, the video community has shown that boundaries are better modeled as probability distributions than as exact points, but those models make no attempt at a globally consistent order.
We propose one model that does both jobs at once. Every candidate stretch of text is assigned an interval on a single learned timeline. A boundary is then no longer a separate binary decision. It is the probability that two adjacent stretches occupy different timeline intervals, averaged over all possible ways of cutting the document. Because every segment lives on one number line, the resulting order cannot contain a cycle. Because the model sums over all segmentations with an exact dynamic program, boundary uncertainty is represented explicitly and passes gradient into the ordering objective. The concrete prediction is that sharing a timeline interval across a whole segment, rather than giving each clause its own coordinate, is what makes end-to-end segmentation and ordering beat the best pipeline built from existing parts.
What exists and where the gap is
The closest existing system is Inducing Temporal Graphs. It splits clinical summaries into "temporal segments," contiguous text with one temporal focus, using a binary boundary classifier (83% F-measure). It then orders the segments with integer linear programming, a solver that picks the highest-scoring set of pairwise "before" decisions subject to hard transitivity constraints. This raises relation accuracy from 81.6% to 84.3% on manual segments. Its boundaries are deterministic, and the ordering stage cannot revise them. Its own annotators agreed on segments at only kappa 0.71, compared with 0.98 on the ordering graph. This is direct evidence that the boundaries, not the order, are the uncertain part. The paper motivates our proposal and supplies our main benchmark.
A second line of work guarantees consistency by construction instead of by solving constraints. Temporal Information Extraction by Predicting Relative Time-lines predicts a start and a duration for each event mention on one axis, trained with a hinge loss that pushes $e_j$ to start after $e_i$ ends whenever the gold relation says $e_i$ comes before $e_j$. Loops become impossible. We reuse this mechanism. That work, however, assumes the event mentions are already given, predicts single points rather than distributions, and gives each mention its own coordinate.
Three further papers take the constraint-solving route, and all of them assume pre-identified event spans:
- Jointly Identifying Temporal Relations with Markov Logic uses weighted logic formulas.
- Clinical Temporal Relation Extraction with Probabilistic Soft Logic Regularization and Global Inference, called CTRL-PG, turns transitivity into a differentiable penalty and adds greedy consistency checking at test time.
- Beyond Pairwise: Global Zero-shot Temporal Graph Generation has a large language model write the whole graph, then merges several runs with integer linear programming.
CTRL-PG also reports that its formulation breaks on recurring events, such as a headache on several different days. A segment-level timeline handles this naturally, because each recurrence is a separate segment with its own interval.
On the boundary side, TriDet and BLP predict distributions over boundary positions within local windows. Learning Temporal Video Procedure Segmentation from an Automatically Collected Large Dataset softens labels with Gaussian heatmaps to absorb misalignment between speech and action. When the Ground Truth is not True shows that annotators systematically round boundaries. These papers supply our reasons for representing boundaries probabilistically. They also constrain the design, since none of them couples one detected boundary to another. DisTime brings distributional timestamps into video language models, but it still generates them one at a time with no global check.
Two results from event segmentation shape the method:
- Computational Complexity of Segmentation proves that optimal one-dimensional segmentation is polynomial-time. This licenses exact summation over segmentations rather than approximate search.
- The latent-cause account in Statistical Computations Underlying the Dynamics of Memory Updating says a new event begins when the current state can no longer explain the input. This is exactly our boundary definition: a new segment begins when the text stops fitting the current timeline interval.
Core Claim
Treat the document as a sequence of $N$ clauses. For every candidate span of clauses $a$ through $b$, with length at most $L$, an encoder pools the span and produces:
- a coherence score $\phi(a,b)$;
- a timeline interval, with start $s_{ab}$ and duration $d_{ab}$.
A segmentation $z$ is a set of non-overlapping spans covering the document. Its probability is
$$ p(z) \propto \exp \sum_{(a,b) \in z} \phi(a,b). $$
This is a semi-Markov model: a sequence model whose states are whole spans rather than single clauses. Forward-backward summation over it costs $O(NL)$ and yields the marginal probability $\mu(a,b)$ that span $(a,b)$ appears as a segment.
Each clause $i$ receives an expected start and end on the timeline. These are the $\mu$-weighted averages of the intervals of the spans that cover $i$. We apply the hinge ordering loss of the relative-timeline model to these expected coordinates, for every pair of clauses whose gold segments are ordered. We add a boundary loss against soft (Gaussian-smoothed) gold boundaries, where the probability of a boundary between clauses $i$ and $i+1$ is the total marginal mass of spans ending at $i$.
Because everything is differentiable through the forward-backward pass, errors in ordering move the boundaries, and errors in boundaries move the ordering. At test time, exact best-path decoding picks the most probable segmentation, and ordering segments by their intervals gives a cycle-free graph at linear cost with no solver.
The claim to test has two parts. First, this joint model should beat an equally strong pipeline on end-to-end ordering accuracy computed on automatically found segments. Second, its boundary probabilities should be calibrated against real annotator disagreement.
Research Program
We will first build the model on the NEJM clinical corpus from Inducing Temporal Graphs, with a pretrained text encoder shared by all systems. This corpus is almost entirely past tense, so tense cues are weak and ordering must come from content and discourse. That is where coupling segmentation to ordering should matter most.
We will then extend the model with a single "off-timeline" interval for hypothetical or planned content. The motivating examples are future-oriented verbs of the kind catalogued in Distinguishing Past, On-going, and Future Events, and the planned or modal events that the relative-timeline model is reported to handle poorly. We will test whether this extension preserves consistency while absorbing that failure mode.
Finally, we will run a transfer test on procedural video, YouCook2. Here the "clauses" become clips, and speech-action lag makes boundaries uncertain. The question is whether the same coupling helps when ordering is mostly, but not strictly, monotone.
The main analysis compares documents where the pipeline's cuts were wrong, and asks whether the joint model repaired them and whether the ordering signal was the cause. We will measure the cause by switching off the ordering gradient into $\phi$.
Baselines
The naive reference point is natural reading order: assume the text is chronological. Inducing Temporal Graphs reports about 73% accuracy for its chronological baseline on NEJM. Any method must clear this comfortably, since clinical summaries are largely forward-moving.
The historical baseline is the original pipeline, which reports 85.0% relation accuracy with best-first ordering on automatic segments and 84.3% with integer linear programming on manual segments. We will re-run it with our encoder replacing its hand-built features. This separates gains from the architecture from gains that come merely from modern representations.
The strongest competitor is a modern pipeline assembled from the best existing parts, built on the same encoder:
- a boundary detector trained on Gaussian-smoothed targets, in the style of the video boundary literature;
- hard cuts at its peaks;
- the CTRL-PG recipe of soft-logic transitivity regularization plus greedy consistency-checked insertion over the resulting segments. CTRL-PG reports 80.20 F1 on I2B2-2012 and 65.2 F1 on TimeBank-Dense, but only with gold spans.
It is the strongest competitor because it already has soft boundaries and global consistency; it only lacks the coupling between them. The joint model and every baseline get the same splits, the same encoder, and the same number of hyperparameter trials, chosen on a shared development fold. With only 60 documents, we will use repeated cross-validation and report paired differences with confidence intervals. Beating this pipeline means a statistically reliable gain in end-to-end relation accuracy on automatic segments, without losing boundary F1.
Two ablations isolate the key ingredient:
- Clause-level timeline. The relative-timeline model applied to every clause, with boundaries read off wherever adjacent predicted intervals separate. This removes span-level sharing entirely.
- Frozen segmentation. The joint architecture with the gradient from the ordering loss into the segmentation scores cut, which is equivalent to a pipeline with the same parameterization.
For video, the reference is DisTime, which reports 26.4 F1 and 31.0 CIDEr on YouCook2. We do not expect to beat a large video language model on captioning. Instead, we will test whether a consistency layer placed on its localized steps reduces ordering violations at equal localization quality.
Why not just…?
Why not just fix the boundary detector and keep the pipeline? This is the most serious alternative. If modern encoders make boundaries nearly perfect, coupling buys nothing. The file offers indirect evidence against it: annotators themselves agree on boundaries only at kappa 0.71, and best-first ordering on automatic segments in the original work did not fall below ordering on manual ones. Both suggest that the "right" cut is partly defined by what orders well. The frozen-segmentation ablation settles the question directly. If it matches the joint model, we will report that coupling is unnecessary, and the contribution reduces to calibrated boundaries.
Why not just predict a timeline position for every clause, skipping segmentation? This is the relative-timeline model extended to clauses. It already guarantees consistency, and boundaries fall out of where positions jump. Its weakness is that each clause must estimate its time alone, from sparse cues, in past-tense clinical text. Recurring or simultaneous events are also reported to confuse it. Our span model pools evidence across a segment before committing to an interval. The clause-level ablation is designed to show whether that pooling matters. If it does not, the simpler model wins, and we will say so.
Why not just ask a large language model to segment and order the document? Beyond Pairwise shows that whole-graph generation with ensemble aggregation reaches near-zero inconsistency. It also reports that accuracy degrades as the number of events grows, and it gives no calibrated boundary probabilities. We will run it with a segment-then-order prompt on the same folds as a zero-shot comparison. It is a real threat on short documents, and the relevant test is how its accuracy scales with document length relative to ours.
Proposed Experiments
The decisive experiment is end-to-end NEJM relation accuracy on automatically found segments, comparing the joint model against the modern pipeline and the two ablations. It is reported by document length and by the number of non-chronological jumps each document contains. We expect the joint model's advantage to grow with the number of jumps, since those are the documents where a single miscut misplaces a whole flashback.
The second check is boundary calibration. It needs the double-annotated NEJM documents, if the corpus includes them. If they are available, we will test whether predicted boundary probabilities match the rate at which annotators disagree, using expected calibration error, compared against the Gaussian-smoothed detector.
The third check applies the model to I2B2-2012 without gold spans, using clause-level segments. We will report relation F1 alongside CTRL-PG's gold-span 80.20 as an upper reference, not as a target, and count the recurring-event pairs each method orders correctly.
The fourth check is the YouCook2 transfer: whether adding the shared-timeline layer reduces ordering violations among localized steps at unchanged boundary F1.
We will count the proposal as confirmed only if two conditions hold together: the joint model beats the frozen-segmentation ablation on end-to-end accuracy, and it beats the clause-level timeline on documents with flashbacks.
References
- Inducing Temporal Graphs
- Temporal Information Extraction by Predicting Relative Time-lines
- Jointly Identifying Temporal Relations with Markov Logic
- Clinical Temporal Relation Extraction with Probabilistic Soft Logic Regularization and Global Inference
- Beyond Pairwise: Global Zero-shot Temporal Graph Generation
- TriDet: Temporal Action Detection with Relative Boundary Modeling
- BLP - Boundary Likelihood Pinpointing Networks for Accurate Temporal Action Localization
- Learning Temporal Video Procedure Segmentation from an Automatically Collected Large Dataset
- When the Ground Truth is not True: Modelling Human Biases in Temporal Annotations
- DisTime: Distribution-based Time Representation for Video Large Language Models
- Computational Complexity of Segmentation
- Statistical Computations Underlying the Dynamics of Memory Updating
- Distinguishing Past, On-going, and Future Events: The EventStatus Corpus