Sky surveys now return far more candidate transients (new or changing points of light) than people can inspect. Automated classifiers split into two camps. Image models look at a few small pixel crops, called cutouts, around the source. Time-series models look at the light curve, the source's brightness over time. The best image model on real survey data, the TAO-Net of Deep-TAO, reads exactly three cutouts: before, at, and after peak brightness. It stacks them as fixed channels, so it has no notion of clock time. Everything else in each sequence is thrown away, even though that dataset holds about 1.25 million images across 16,307 sequences, roughly 75 epochs per object.
Generative audio ran into the same failure and fixed it. Models that condition on a single global prompt blur when events happen. Models that feed a time-aligned control signal into every layer, such as the envelope-driven feature modulation in T-FOLEY, place events where they belong. We propose to carry that fix over to classification. A light-curve encoder produces a continuous, time-varying control signal. That signal is sampled at each cutout's true observation time and modulates how the image encoder reads that cutout. The result is one forward pass over all available epochs, with no need to first find the peak and then fetch three frames. We pair it with explicit geometric input channels and a controlled "offset-swap" probe that tests whether the model uses where the transient sits relative to its host galaxy.
Core Claim
Classification of real survey transients improves when each image is interpreted in light of the source's brightness state at that image's exact observation time, rather than when image and light-curve features are merged after the fact. The prediction is specific. Timeline conditioning should beat late fusion, meaning two encoders whose summary vectors are concatenated before a classifier. It should beat it most on classes separated by how brightness and morphology change together, and it should hold up when epochs are randomly dropped. If well-tuned late fusion matches it, the claim is false and the idea reduces to "use more data."
Each input direction plays a different role. Fine-grained temporal control supplies the mechanism: FiLM (feature-wise linear modulation, a per-channel scale and shift computed from a conditioning signal), used as in T-FOLEY and, in ControlNet form, FoleyCrafter. The manual classification bottleneck supplies the application and benchmark. Spatial reasoning deficiency supplies a design constraint and an evaluation lens. Iterative refinement latency supplies a hard budget: a single pass, milliseconds per alert.
Relation to Prior Work
Several papers motivate the gap. Deep-TAO showed that raw cutouts beat one-dimensional light-curve features, with 5-class F1 of 52.37% for the CNN against 45.49% for a random forest. It uses a rigid three-frame schedule, and we replace that schedule with continuous-time conditioning over every epoch. Learned Bayes Factors shows that a transformer can ingest irregularly sampled light curves without interpolation and give amortized decisions in about 10 ms. It discards pixels entirely, while we use its style of irregular-time encoder as the source of our control signal. The PLAsTiCC results show that winning systems relied on Gaussian-process interpolation and hand-built features. That makes a gradient-boosted tabular model a baseline we must beat, not a straw man.
Other work supplies the method. The audio literature separates "what" from "when." Examples are FiLM on envelopes in T-FOLEY, injection of time, pitch, and energy curves in Audio Generation with Multiple Conditional Diffusion Model, and time-aligned symbolic tracks in JASCO. All of these generate audio. None classify, and none condition one modality on another modality's timeline at irregular sample times, which is what we do. AudioTime documents what global conditioning costs: the best model has an ordering error rate of 0.758 against 0.220 for ground truth. We read this as evidence that pooled summaries lose timing, and our late-fusion baseline is exactly such a pooled summary.
Other work constrains the design. What's "up" with vision-language models? shows that vision-language models (VLMs) fall back on priors instead of geometry. ViewSpatial-Bench reports GPT-4o at 34.98% on allocentric localization, which means judging position from a reference frame other than the viewer's. Both argue against routing spatial judgments through a general VLM. We instead give the network explicit geometry: difference images plus coordinate channels encoding offset from the host centroid.
That same paper supplies our evaluation idea. It isolates spatial relations by holding objects fixed and varying only their arrangement, and our offset-swap probe applies that logic to transients. At First Sight shows that zero-shot GPT-4o reaches about 83% on low-surface-brightness galaxies, against about 92% for a supervised CNN and about 58% for LLaVA-NeXT. It works on static single-epoch images, so it is a baseline and not a competitor on time.
Iterative refinement bounds the latency budget. Volcano improves visual grounding with critique and revision but raises latency from 2.7 s to 5.8 s. GW-Eyes chains multi-turn calls to frontier LLMs to associate alerts. Meta-Reflection folds reflection into one pass, at 246 ms against 951 ms, but only for text. Our framing is that the Deep-TAO pipeline is itself a two-pass refinement. It reads the light curve, locates the peak, then goes back for the relevant images. Timeline conditioning amortizes that "look again" step into the network.
Research Program
The model has three parts. The first is a light-curve encoder: a transformer over (time, magnitude, error) triples with continuous time embeddings, following the irregular-sampling design of Learned Bayes Factors. From its output we define a control function $c(t)$ that can be queried at any time $t$ by cross-attending from a time embedding of $t$ to the encoded observations.
The second is a per-epoch image encoder. It is a small CNN over each cutout, the matching difference image against a reference, and two coordinate channels giving pixel offset from the host centroid. For the cutout taken at time $t_k$, every block $\ell$ is modulated as
$$h_\ell^{(k)} \leftarrow \gamma_\ell\big(c(t_k)\big)\odot h_\ell^{(k)} + \beta_\ell\big(c(t_k)\big),$$
where $h_\ell^{(k)}$ is that block's feature map for epoch $k$, and $\gamma_\ell$ and $\beta_\ell$ are learned linear maps. Brightness state therefore changes what the image encoder attends to. For example, the encoder can suppress host-dominated features when the source is faint and weight the point-source residual near peak.
The third part is an aggregator: attention over the modulated epoch features, with time embeddings of $t_k$, followed by a classification head. We train the whole model end to end on Deep-TAO. Training includes random epoch dropout, keeping between 3 and all epochs, so the model learns to cope with irregular cadence.
Two analyses accompany the model. The first is the offset-swap probe. Starting from Deep-TAO non-transient cutouts, we inject a point source that follows a fixed light curve at controlled offsets from the host: nuclear, near the disk, or well outside it. We then check whether predicted class probabilities shift the way astrophysics says they should, since nuclear variability points to active nuclei and off-nuclear variability points to supernovae. This asks whether the model reasons about geometry or leans on class frequencies. The second analysis measures per-alert latency on one GPU, in both batched and single-alert modes.
Baselines
The naive reference is the random forest on one-dimensional light-curve features reported by Deep-TAO, at 5-class F1 of 45.49%. Next is TAO-Net, the published three-epoch CNN, at 5-class F1 of 52.37%, binary F1 of 84.76%, and non-transient filtering F1 of 95.29%, all as reported by Deep-TAO. We retrain it on our splits so the numbers are comparable, and we also report its published figures.
The strongest competitor on our question is late fusion. This pairs the same light-curve transformer and the same per-epoch image CNN, with coordinate channels and all epochs, but joins them by concatenating pooled vectors or by cross-attention at the final layer only. No reported result was found on Lacuna for late fusion on Deep-TAO. It is the right competitor because it uses identical inputs and parameter count, so any gain can only come from where and how time conditions the image features. All three learned models share splits, input epochs, parameter count within 10%, and the same hyperparameter search budget (the same number of trials per model), and each is reported over five seeds.
We also run a tabular baseline in the PLAsTiCC-winner style, with no reported result found on Lacuna for Deep-TAO. It uses Gaussian-process light-curve features plus summary image statistics, fed to gradient-boosted trees. Finally we run a zero-shot GPT-4o baseline, following the rubric protocol of At First Sight, on a 500-object subsample. For it, cutouts and a plotted light curve go into the prompt, and no reported result was found on Lacuna for this setting.
Four ablations isolate the key ingredient. The first replaces continuous $c(t_k)$ with a learned slot index per epoch. The second replaces FiLM with plain concatenation of $c(t_k)$ onto the image features. The third removes the coordinate channels. The fourth restricts input to Deep-TAO's three epochs.
Beating the field means three things. The model must exceed late fusion in 5-class macro F1 by more than the seed-to-seed spread. The gain must hold or grow when epochs are dropped. And latency must stay near the roughly 10 ms scale that Learned Bayes Factors reports for amortized detection, orders of magnitude below the multi-second budgets of Volcano.
Why not just…?
Why not just feed all epochs to TAO-Net as extra channels? This fixes the data waste but not the timing. Channel stacking assigns meaning by slot position, so it breaks when cadence is irregular and epoch counts vary. Our slot-index and all-epoch ablations test this directly. If slot stacking over all epochs matches our model, then the gain comes from more data and not from timeline conditioning, and we would report that.
Why not just fuse a light-curve model and an image model at the end? This is the real threat, and it is why late fusion is our primary baseline and not an afterthought. The audio evidence (AudioTime) suggests pooled summaries lose timing, but that evidence comes from generation, not classification. The head-to-head test with matched parameters, together with per-class analysis of classes that show similar light curves but differ in image structure, settles the question. We commit to reporting a null result.
Why not just use Gaussian-process features and boosted trees, as PLAsTiCC winners did? Those pipelines are strong on light curves. However, Deep-TAO shows that one-dimensional features alone trail image models on real CRTS data, and PLAsTiCC reports that classical systems misfiled the unmodeled "Other" class. We run this baseline with image summary statistics added, so any loss it suffers is attributable to feature design, not missing inputs.
Why not just prompt a frontier multimodal model, or wrap it in a critique loop? ViewSpatial-Bench and What's "up" show weak geometric reasoning in such models, and Volcano shows that refinement roughly doubles already multi-second latency. Survey alert rates cannot absorb that. The GPT-4o subsample and the offset-swap probe measure how much accuracy that path gives up, if any.
Proposed Experiments and Decisive Checks
The main experiment trains all models on Deep-TAO's 5-class and binary tasks. We report macro F1, expected calibration error (the average gap between predicted confidence and observed accuracy), and milliseconds per alert. We then run the epoch-dropout sweep, evaluating each model with 3, 10, 25, and all epochs. This is where continuous-time conditioning should separate from slot-based and late-fusion designs.
Next comes the offset-swap probe. Its decisive pattern is that predicted class probabilities move with injected offset while the light curve stays fixed. A model that uses geometry should show a strong monotone dependence, and one that ignores it should show a flat response. As a secondary check of the light-curve branch alone, we train the encoder on PLAsTiCC and compare it with the reported winning log-loss of 0.685. This confirms the control signal is competitive before we attribute any fused gains to it.
The whole proposal hinges on two outcomes: timeline conditioning must beat matched late fusion, and it must hold that lead when epochs are dropped. If either fails, the audio-derived mechanism does not transfer, and that negative result is worth publishing.
References
- Deep-TAO: The Deep Learning Transient Astronomical Object data set for Astronomical Transient Event Classification
- T-Foley: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis
- FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds
- Microlensing Detection and Inference via Learned Bayes Factors
- Results of the Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC)
- Audio Generation with Multiple Conditional Diffusion Model
- JOINT AUDIO AND SYMBOLIC CONDITIONING FOR TEMPORALLY CONTROLLED TEXT-TO-MUSIC GENERATION
- AudioTime: A Temporally-aligned Audio-text Benchmark Dataset
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models
- Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision
- An agentic framework for gravitational-wave counterpart association in the multi-messenger era
- Meta-Reflection: A Feedback-Free Reflection Learning Framework