When a small model learns to reason by copying a large model's step-by-step explanations, it also copies the large model's mistakes. For questions that chain several facts over time, such as "who led the country right before the person who signed treaty X?", most of those mistakes take a specific, checkable form: an intermediate step names the wrong date, or puts two events in the wrong order. We propose to treat each step's timestamp as something that can be verified and corrected before the small model ever sees it. Steps that contradict a temporal knowledge base, or break the before/after constraint stated in the question, get repaired. The student then reads the repaired chain with a positional encoding indexed by each fact's actual date rather than its token position, so that "earlier than" becomes a geometric relation inside attention instead of a string comparison between year tokens.
The claim to test is that this pairing produces a student that is right on multi-hop temporal questions its teacher gets wrong. It should also shrink the compositionality gap, the share of multi-hop questions a model fails even though it answers each single-hop sub-question correctly. Temporal question answering suits this test because its intermediate steps carry ground-truthable coordinates (entities and timestamps). That makes step-level repair possible here in a way it is not for free-form commonsense rationales.
Core Claim
Three findings motivate the proposal. Press et al. (Measuring and Narrowing the Compositionality Gap in Language Models) show that the gap stays near 40% as GPT-3 grows from 0.35B to 175B parameters, and that explicit decomposition (Self-Ask, where the model poses and answers its own follow-up sub-questions) narrows it. Chen et al. (Post-Semantic-Thinking) show the other side: a student distilled on teacher rationales is highly sensitive to small early errors in those rationales. Their remedy predicts the answer before the rationale, which sidesteps poisoned steps but also gives up the decomposition that Press et al. found necessary. Prog-TQA (Self-Improvement Programming for Temporal Knowledge Graph Question Answering) reports that training on teacher programs filtered only by final-answer correctness admits "spurious programs", meaning wrong reasoning that happens to land on the right answer.
These results leave a tension. Decomposition is needed, decomposed teacher chains are poisoned, and filtering by the final answer does not detect the poison. Our claim is that in temporal reasoning the poison is mostly located in the timestamps of intermediate steps, and that this localization makes the problem solvable. Verifying and repairing step timestamps keeps the decomposition and removes the poison. Repaired chains also exist for questions the teacher answered wrongly, which gives the student supervision the teacher could not have produced. A time-indexed positional encoding then gives the student an inductive bias toward the order structure those chains rely on. The weak-to-strong analysis of Moniri et al. (On the Mechanisms of Weak-to-Strong Generalization) supplies a theoretical reason to expect this bias to matter: in their linear and multi-index models, a student with a prior aligned to the true structure ignores teacher noise along the wrong directions. Their analysis covers only static feature composition in simple models, so here it serves as motivation, not a guarantee.
Research Program
The setting is temporal knowledge graph question answering. A temporal knowledge graph stores facts as (subject, relation, object, time) quadruples. Following Local and Global (LGQA), which fuses a 3-hop local subgraph with a text encoder, each question comes with a retrieved few-hop subgraph serialized as text. The teacher is a large instruction-tuned language model prompted with Self-Ask over that context. It produces chains of steps, and each step asserts one fact together with its time.
The first component is step repair. Each teacher step is parsed into a quadruple and checked in two ways. The first check asks whether the fact exists in the subgraph with the stated time. The second asks whether the step's time satisfies the temporal operator in the question; if the question says "before X," the step's interval must precede X's interval. When a step fails, we substitute the subgraph fact that matches the step's entity and relation and satisfies the constraint, then ask the teacher to regenerate the remaining steps conditioned on the corrected prefix. Only chains whose every step verifies are kept.
This procedure differs from Prog-TQA in two ways. It verifies steps, not just the final answer. It also repairs failed chains rather than discarding them, so questions the teacher got wrong become training data instead of disappearing. It differs from Selective Reflection-Tuning (Selective Reflection-Tuning), which lets the student veto teacher data using its own perplexity scores. Those scores measure how compatible the data is with the student, not whether it is logically valid, so they cannot detect a fluent chain with a wrong date.
The second component is a time-indexed positional encoding in the student. We use rotary encoding, which rotates query and key vectors by an angle tied to position so that attention scores depend on relative offsets. Every token belonging to a fact or step receives an extra set of rotary channels whose angle is driven by that fact's timestamp $\tau$ rather than its token index. For a query from a fact at time $\tau_i$ and a key from a fact at time $\tau_j$, the time channels contribute
$$ s_{ij} = \sum_{m} \mathrm{Re}\left[\, q_{i,m}\, \overline{k_{j,m}}\; e^{\,\mathrm{i}\,\omega_m (\tau_i - \tau_j)} \right], $$
where $q_{i,m}$ and $k_{j,m}$ are the complex-valued query and key coordinates in frequency band $m$, and $\omega_m$ is that band's frequency.
Three properties follow from this form. The score depends only on the elapsed interval $\tau_i - \tau_j$, which matches the relative periodic encoding of time gaps that TiPNN (Temporal Inductive Path Neural Network) used to generalize to unseen entities and eras. The sine component changes sign with the interval, so "before" and "after" stay distinguishable without the auxiliary time-order loss TSQA (Improving Time Sensitivity) needed. Geometric spacing of $\omega_m$ from days to decades targets the granularity mismatch that the literature lists as an open problem. Ordinary token-index rotary channels remain in place for word order. Prior temporal encodings were attached to external timestamps in graph neural networks or trajectory models. We instead place time inside an autoregressive student's attention over its own reasoning steps.
The student is a small language model, GPT-2 scale following ATM's setup plus one roughly 1B-parameter model. It is trained to produce the repaired chain and then the answer.
Baselines
The naive reference point is the same student fine-tuned on question–answer pairs with no rationale, which measures what direct answering achieves without decomposition. The second reference is the teacher itself, evaluated with Self-Ask on the same contexts. Because the core claim is about piercing the teacher's ceiling, every student result is reported against teacher accuracy both overall and on the subset of questions the teacher gets wrong.
The strongest distillation competitor is vanilla chain-of-thought distillation with answer-correctness filtering. This is the rejection-sampling recipe that Prog-TQA's self-improvement loop resembles, and it is cheap and widely used. We run it with the same teacher samples, the same number of kept training chains, the same student, and the same tuning budget. That means an equal number of hyperparameter trials, chosen on the same development split. Next to it we run ATM from Post-Semantic-Thinking, the existing method aimed directly at rationale poisoning; it reports 65.06% on StrategyQA with GPT-2 against 63.61% for pre-thinking. We also run Prog-TQA itself, a fine-tuned model writing programs for an external symbolic executor. It is the strongest method that uses the same weak supervision, though on MultiTQ the file gives only relative gains (+60.9% on multi-constraint and +47.0% on single-constraint Hits@1), so no reported absolute result was found on Lacuna for Prog-TQA on MultiTQ.
For absolute context we cite task-specific supervised systems without claiming to match their architectures. LGQA reports Hits@1 of 0.969 on CRONQUESTIONS, against 0.831 for TSQA. JMFRN (Joint Multi-Facts Reasoning Network) reports Hits@1 of 0.628 on TimeQuestions and 0.553 on its ordinal questions. CRONQUESTIONS is close to saturated, so we treat it mainly as a source of compositionality-gap measurements and center the headline comparisons on MultiTQ and TimeQuestions.
Two ablations isolate the key ingredients. Crossing repair against filter-only and against raw teacher chains isolates the data intervention. Crossing time-indexed rotary channels against dates written as plain text isolates the encoding, with the extra channels kept but driven by token index so that parameter count stays equal.
Beating the strongest distillation baseline means three things, each holding with confidence intervals over three seeds: higher Hits@1 on multi-constraint MultiTQ and on ordinal TimeQuestions, a smaller compositionality gap, and nonzero accuracy above the teacher on the teacher-wrong subset.
Why not just…?
Why not just filter teacher chains by whether the final answer is right? This is the main threat. If spurious chains are rare, repair adds nothing. Prog-TQA's own account of spurious programs suggests they are not rare, but the file gives no rate. We therefore measure it directly. On a sample of answer-correct teacher chains, we report the fraction containing at least one step that fails verification. We then compare a filter-only student against a repaired-chain student trained on the same number of chains, so the comparison is not confounded by data volume. If that fraction is small and the two students tie, we drop the repair claim and report the encoding result alone.
Why not just hand the temporal algebra to a symbolic executor, as Prog-TQA does? On benchmarks with a clean knowledge graph, an executor may well win, and we report it head to head. Our target is a different and weaker claim: a model that reasons about time end to end can inherit less of its teacher's error. Executors fail when parsing fails or when questions refer to time implicitly ("during the tenure of"), a failure mode the literature flags. We therefore break results down by TimeQuestions' implicit category, where we expect the end-to-end student to be most competitive.
Why not just use a bigger teacher, or a bigger student? Press et al. show that scale leaves the compositionality gap near 40%, so scale alone is unlikely to remove temporal step errors. We nevertheless include a 3× larger student trained on raw chains, so that any gain from the encoding cannot be explained by capacity.
Why not just write dates as text and let the model read them? TSQA reports that without explicit order supervision, systems treat temporal words as a bag of words and confuse "before" with "after"; adding order supervision gave a 32% absolute error reduction on complex queries. Our text-date ablation tests whether a pretrained language model that reads chains as text escapes this problem. We also add a held-out split of later time periods. If text dates match the rotary encoding on both the in-distribution and the later-period splits, the encoding is unnecessary, and we will say so.
Proposed Experiments
The data are CRONQUESTIONS, MultiTQ, and TimeQuestions with their standard splits. We add a temporal-extrapolation split, built by holding out all questions whose answers fall in the latest years, to test whether time-indexed positions extend to unseen dates.
To measure the compositionality gap, we take each complex question in CRONQUESTIONS and MultiTQ, which are template-generated, and derive its single-hop sub-questions from the template and the underlying quadruples. The gap is the fraction of composite questions answered wrongly among those whose sub-questions are all answered correctly, following Press et al. The metrics are Hits@1, the compositionality gap, the step-validity rate of student chains under the same verifier, and accuracy on the teacher-wrong subset.
The first decisive check is the ceiling test. A student trained on repaired chains must exceed chance-corrected zero on questions the teacher gets wrong, while the filter-only student should stay near zero there. The second decisive check is the encoding ablation. The time-indexed student should show a smaller compositionality gap than the text-date student, with the largest difference on before/after and first/last questions and on the extrapolation split.
A diagnostic separates where errors come from. Applying the step verifier to student outputs shows whether remaining errors are date errors, which repair should reduce, or entity errors, which it should not. A result where repair improves accuracy but leaves the gap unchanged would show that the gain comes from cleaner labels rather than better composition. That outcome is still informative, and we would report it as such.
References
- Measuring and Narrowing the Compositionality Gap in Language Models
- Distilling Reasoning Ability From Large Language Models With Adaptive Thinking
- Self-Improvement Programming for Temporal Knowledge Graph Question Answering
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
- Local and Global: Temporal Question Answering via Information Fusion
- Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
- Temporal inductive path neural network for temporal knowledge graph reasoning
- Improving Time Sensitivity for Question Answering over Temporal Knowledge Graphs
- Joint Multi-Facts Reasoning Network for Complex Temporal Question Answering Over Knowledge Graph