Borrowed Pasts: Forecasting New Entities from the Age-Aligned Histories of Look-Alikes

Forecasting systems over evolving graphs fail most visibly on entities they have never seen before. A product listed this morning, a politician who just entered office, or a person who first appears in a user's photo album has no history to learn from. Models that predict future links from past links can only guess, typically by averaging over structurally similar entities or by falling back on popularity. Yet such newcomers usually arrive with a picture, a description, or a recording. More importantly, the system's memory already holds many earlier newcomers that looked similar and whose early lives it watched unfold. We propose to forecast a new entity's next interactions by retrieving perceptually similar entities from a multimodal memory and reading off what happened to them at the same age, measured as time elapsed since each entity first appeared rather than calendar time.

The core claim is that perceptual similarity alone gives only a static prior ("this looks like a running shoe"). Pairing it with a memory whose clock starts at each entity's birth gives a dynamic prior ("running shoes in their second week tend to get reviewed by these kinds of users"). This prior should beat both structural averaging and simply feeding image features into an existing forecaster, and it should do so specifically in the first few interactions of an entity's life. If image features injected into a strong forecaster close the gap, the idea is wrong, and the experiments are designed so we would find that out.

Background and the gap

A temporal knowledge graph stores facts as timestamped quadruples (subject, relation, object, time), and extrapolation means predicting facts at future times. Its defining difficulty is that future edges do not exist yet. DiMNet addresses this by splitting entity features into fast-changing and stable parts and synthesizing virtual subgraphs. TimeTraveler trains a reinforcement-learning agent to walk backwards through history, with time encoded as the offset between query and historical event. For unseen entities it uses an "Inductive Mean": it averages embeddings of known entities that co-occur with the same relations. Lifelong temporal KG reasoning extends this to streams with growing entity sets, initializing newcomers by edge-aware message passing and parameterizing actions by relative displacement. It reports that removing the displacement costs 18%. All three solve cold start from graph structure, and structure is exactly what a newcomer lacks. These papers motivate the proposal and supply its strongest competitors.

Relative temporal encoding means representing time as elapsed intervals instead of absolute dates. It is what lets learned patterns transfer to unseen periods: TILP reports out-of-distribution "time shifting" MRR of 0.2657 against 0.0976 for the best embedding baseline. Existing work, including Noether Embedding and the interval-aware recommender of Not Just What, But When, always measures the interval from the query back to an event, or between consecutive events. None measures it from the entity's own first appearance. That choice of origin is our one technical departure, and it matters because it makes a donor entity's early life comparable to a newcomer's early life regardless of when each was born.

Multimodal memory systems organize heterogeneous experience into graphs for retrieval. Examples are AssoMem, with relevance, PageRank importance, and timestamp-match signals; MAGMA, with separate temporal, causal, semantic, and entity graphs; LifeGraph, which links personal events to raw photos; and VAT-KG. They answer questions about the past using absolute timestamps or chronological order, and never forecast. Separately, MM-GRAPH shows that visual node features lift static link prediction on Amazon-Sports from about 11.1 to about 30.45 MRR. It also shows that a plain GraphSAGE on pre-aligned CLIP or ImageBind features beats specialized multimodal graph networks. That second finding constrains our design: the method should consume frozen aligned embeddings and avoid elaborate fusion.

The closest existing idea is "add multimodal features to a temporal KG model." It is under-explored in the file but obvious, so we treat it as the main baseline rather than the contribution.

Core Claim

Let a newcomer $e$ be born at time $b_e$, with a frozen perceptual embedding $x_e$ from an aligned image–text encoder. Memory stores, for every past entity $u$, its events re-indexed by age $a = t - b_u$. For a query $(e, r, ?, t_q)$ at age $a_q = t_q - b_e$, the borrowed-history score of a candidate object $o$ is

$$ s(o) = \sum_{u \in D_k(e)} w(x_e, x_u) \sum_{(u, r, o', a) \in M} \kappa(a - a_q)\, \phi(o', o) $$

The terms are defined as follows. $D_k(e)$ is the set of $k$ donors retrieved by perceptual similarity. $w$ is a learned similarity weight. $\kappa$ is a learned kernel over age differences, a small mixture of Gaussians in log-age. $\phi(o', o)$ is 1 when the donor's partner is the candidate and an embedding similarity otherwise, so that partners can transfer by type when they do not recur exactly.

This score is combined with a strong base forecaster through a gate that depends only on how many events $e$ has accumulated. With zero history the borrowed score dominains; as history grows, the base model takes over. The claim is falsifiable in a precise way. Gains should concentrate in the low-history strata, and age alignment should beat query-offset alignment and no-time alignment on the same donors. If perceptual features concatenated into the base model match the full method, the dynamic-prior hypothesis fails.

Research Program

We will build the age-indexed memory as a lightweight store keyed by perceptual embedding. Each entry holds its age-stamped event list, so retrieval costs one nearest-neighbour search plus a windowed scan. This deliberately avoids the slow LLM consolidation passes that MAGMA requires. The kernel, the donor weights, and the gate are trained on entities born in the training period by simulating cold start: we truncate their histories and predict their next events. This also tests whether the learned kernel is interpretable, for instance whether product lifecycles show the early burst one would expect.

The main testbed is a cold-start temporal split built from Amazon product data. The temporal graph benchmark's tgbl-review (TGB) supplies timestamped user–product reviews, and MM-GRAPH's Amazon datasets supply product images. Matching the two on product identifiers is an engineering risk. If coverage is poor, we fall back to adding timestamps to MM-GRAPH's Amazon-Sports and Amazon-Cloth graphs from the original review records.

The second testbed is the lifelong ICEWS benchmarks used by lifelong TKG reasoning. There the "perceptual" key is a text embedding of the entity's name and type. This tests whether the mechanism helps when the side information is weak and the base model is strongest.

The third, exploratory testbed is Life-Bench, which has 2,887 images and simulated timelines. From it we derive a forecasting task: given a person's first photo appearances, predict which concepts they co-occur with next. This is the setting that ties the method to personal multimodal memory.

All results will be stratified by the newcomer's number of prior events: 0, 1–3, 4–10, and more than 10. We report filtered MRR and Hits@1/10 in the time-aware filtered setting, plus results on the "time shifting" style split where test entities are born after every training entity.

Baselines

The naive reference points are popularity and recency. Popularity predicts the globally most frequent objects for relation $r$ in the recent window. Recency/frequency is the newcomer's own history once it has one. Do TKG Embedding Models Learn or Memorize Shortcuts? shows that such shortcuts explain much of what TKG models achieve. TGB reports a moving average reaching NDCG@10 of 0.559 against 0.315 for TGN on tgbn-reddit. Any claim of gain must clear these first, on identical splits and with zero tuning advantage.

The strongest existing method is the lifelong relative-displacement RL forecaster of Temporal Extrapolation and Knowledge Transfer for Lifelong TKG Reasoning. It reports MRR 0.385 on ICEWS18-lifelong against 0.293 for the best prior baseline, and its edge-aware initialization is the most direct structural answer to cold start. We will rerun it with released or reimplemented code, give it the same hyperparameter search budget as our method, and use it as the base forecaster inside our gate. Beating it then means the gated combination exceeds it, not merely that a separate model does. On ICEWS we count a win as an improvement in the 0 and 1–3 strata that exceeds seed variance across five seeds, with no loss above 0.005 MRR in the >10 stratum. For tgbl-review and our Amazon cold-start split, no reported result was found on Lacuna for cold-start-stratified forecasting, so all numbers there will be produced by us under one protocol. We also run TimeTraveler with its Inductive Mean, and DiMNet, as additional structural baselines.

The key-ingredient baseline is "static perceptual injection." The same frozen CLIP embeddings initialize the entity embeddings of the strongest base forecaster through a linear map trained end to end. This follows MM-GRAPH's finding that plain models on aligned features are hard to beat. It has exactly the information our method has, minus the age-indexed donor histories. Three ablations isolate the mechanism on fixed donors: age-aligned kernel, query-offset kernel (TimeTraveler-style $t_q - t_h$ applied to donor events), and no time weighting at all. Two further ablations vary the donors themselves: perceptual donors versus structurally chosen donors versus random donors.

Why not just…?

Why not just concatenate image embeddings into DiMNet or the lifelong forecaster? This is the real threat, and the proposal is built around testing it. Injected features tell the model what a newcomer is, which may suffice whenever "what it is" determines its partners regardless of age. Our hypothesis predicts a specific divergence. Static injection should help uniformly across ages, whereas borrowed histories should help most when partner sets shift with age, as with early reviewers versus late ones. If static injection matches the full method in the 0–3 strata on Amazon, we will report that the dynamic prior adds nothing and recommend the simpler method.

Why not just caption new entities with a vision-language model and prompt an LLM forecaster? In-context TKG forecasting and GenTKG show that LLMs can forecast from serialized history. But a newcomer has no history to serialize, so the prompt reduces to a caption plus whatever the LLM knows about popular objects. That knowledge is precisely the shortcut that frequency baselines capture. We will run this pipeline with the same candidate set and with retrieved donor histories serialized into the prompt. If the LLM with serialized donor histories matches our scorer, the contribution is the age-indexed retrieval, not the scorer, and we will say so.

Why not just use the Inductive Mean or edge-aware initialization, which already handle new entities? Both need at least one observed edge. At zero history they degenerate to relation-level averages. The stratified evaluation shows directly whether they recover by 1–3 events, in which case our window of advantage is narrow, and we will report its width.

Why not just feed the whole multimodal memory to a long-context model? LongMemEval reports 30–60% declines for long-context models relative to oracle evidence, and LifeGraph finds that larger retrieved context lowers accuracy. Our retrieval budget of $k$ donors is the oracle-like filter these results argue for. We test this only on the Life-Bench task, where the context fits.

Decisive checks

Three results settle the proposal. First, age alignment must beat query-offset alignment on identical donors in the 0–3 strata on Amazon. If it does not, the relative-encoding choice is irrelevant. Second, the full method must beat static perceptual injection into the strongest base forecaster in those strata. If it does not, the memory is unnecessary. Third, the learned gate must hand control to the base model as history grows without hurting mature-entity accuracy. On ICEWS, where the side information is only names, we expect small gains at most. A null result there, alongside a positive result on Amazon, would itself confirm that the mechanism depends on informative perceptual keys rather than on extra parameters.