Canonical correlation analysis (CCA) finds a function of one data stream and a function of another that agree as closely as possible. Methods that find non-linear versions of these functions routinely throw away one solution: the constant function, which agrees with itself perfectly and so is treated as trivial. This note argues that, when the two streams are the same physical system seen at two different times, the solutions that come right after that constant are the most physically meaningful output the method can give. A function that takes the same value now and a moment later, for every trajectory, is by definition a conserved quantity, such as energy or angular momentum. We propose to find conserved quantities directly from raw, multi-sensor observations. Each sensor gets one non-linear function that is applied at both times (the same function, not two). That function must agree with itself across random time lags and must also agree with a matching function of a second sensor. We find it by eigendecomposition of a nonparametric operator, not by training a classifier. Every conserved quantity the system has should then appear together as a near-unit eigenspace, rather than one at a time.
This matters because the two literatures each solve half the problem. Interpretable invariant discovery works well when it is fed clean physical coordinates. Non-linear multi-view methods recover shared latent structure from pixels and sensor arrays, but they produce coordinates nobody can read. Tying the function across time is what makes the bridge possible, and the next section explains why the obvious untied version fails.
Core Claim
Let $a_t$ and $b_t$ be the observations of two sensors watching one system at time $t$, and let $\tau$ be a time lag drawn from a broad distribution $\rho$. We seek functions $f$ of sensor A and $g$ of sensor B that maximize
$$ J(f,g) \;=\; \mathbb{E}_{\tau\sim\rho}\,\mathrm{corr}\big(f(a_t), f(a_{t+\tau})\big) \;+\; \mathbb{E}_{\tau\sim\rho}\,\mathrm{corr}\big(g(b_t), g(b_{t+\tau})\big) \;+\; 2\,\mathrm{corr}\big(f(a_t), g(b_t)\big). $$
The same $f$ is applied at both times, and $J$ reaches its maximum of 4 exactly when $f$ and $g$ are conserved and shared between the sensors.
Each design choice blocks a specific failure.
Tying the function across time. For deterministic dynamics, $a_{t+\tau}$ determines $a_t$. Untied lagged CCA, which uses separate functions at the two times, therefore finds a perfect correlation for every function of the state. Its spectrum is all ones and tells us nothing.
Averaging over random lags. Tying alone is not enough for oscillatory systems. In the symmetrized operator, a phase-like function rotating at frequency $\omega$ has eigenvalue $\cos(\omega\tau)$. For a harmonic oscillator this stays close to 1 at a single lag. Averaged over a broad $\rho$, it shrinks toward zero unless $\omega = 0$.
The cross-sensor term. This term removes quantities that one sensor conserves but the physical system does not, for example a fixed calibration offset or a rotating distractor icon seen by only one camera.
Counting invariants. The near-unit eigenspace contains all functions of the invariants, such as $E$, $E^2$, and $EL$, not just $E$ and $L$. We therefore read off the number of independent invariants as the intrinsic dimension of the embedding that this eigenspace defines, not as its multiplicity.
Relation to Prior Work
Nonparametric Canonical Correlation Analysis supplies the estimator. It finds non-linear CCA functions by a singular value decomposition of a joint-density operator estimated with sparse k-nearest-neighbour kernels, and it discards the leading constant solution. We change the operator in three ways: we symmetrize it under tying, average it over lags, and couple it across sensors. We also retain, rather than discard, the solutions near that constant.
Local Canonical Correlation Analysis for Nonlinear Common Variables Discovery motivates the multi-sensor setting. It recovers shared latent variables, such as pendulum frequencies, from two non-linearly observed sensors, but it outputs diffusion coordinates with no conservation semantics. We use its coupled-pendulum and rotating-icon setups as testbeds.
The invariant-discovery line both motivates and constrains the design:
- Discovering Symmetry Invariants and Conserved Quantities by Interpreting Siamese Neural Networks trains twin networks with a one-dimensional bottleneck to tell whether two states share a trajectory. It recovers one invariant at a time, and it needed hand-curated data to expose energy once angular momentum dominated. That "hierarchical discovery" behaviour is exactly the failure our joint eigenspace is meant to avoid.
- Noether's Razor: Learning Conserved Quantities finds the full symmetry group of an $n$-harmonic oscillator, $n^2$ conserved quantities. It does so only for quadratic invariants and only from phase-space coordinates.
- Noether Networks: meta-learning useful conserved quantities learns a conservation embedding to steer a single-stream predictor at test time.
Our method differs from all three on one point: none of them takes raw multi-sensor views as input, and we do.
PySINDy supplies the symbolic readout. It fits sparse combinations of candidate terms, such as polynomials and sines of the state, to data. We use it to find a sparse library combination lying in the discovered eigenspace, which turns each recovered invariant into a formula wherever interpretable coordinates are available.
Research Program
Estimator. For each sensor, we reduce the raw observations with a diffusion-map embedding, which keeps the density estimate low-dimensional without linear PCA flattening the geometry. We then build the k-nearest-neighbour joint-density operator over pairs $(a_t, a_{t+\tau})$ pooled across random lags, symmetrize it, and take its leading eigenfunctions after the constant. The cross-sensor term is applied in a second stage: finite-dimensional linear CCA between the two sensors' near-unit eigenspaces, keeping the directions whose correlation is near 1.
Neural variant. A parameterized version trains two encoders on $J$ with a whitening constraint. It targets the case where nearest-neighbour search does not scale, and it lets us test whether the nonparametric estimate is the source of any advantage.
Model selection. The lag distribution $\rho$ is chosen by the spectral gap below the near-unit cluster, not by ground-truth invariants. The number of invariants is the estimated intrinsic dimension of the near-unit embedding.
Readout. Wherever a state library can be built, we run sparse regression to find, for each independent direction, the sparsest library combination whose span lies in the recovered subspace. In real raw-data settings that library may not exist. There the claim is restricted to recovering the invariant functions, not formulas.
Baselines
Naive reference. Linear tied lagged CCA. This is the linear-function case of our objective and is essentially slow feature analysis across sensors. It is cheap and closed-form, and it should fail on curved invariants such as Kepler energy observed through a camera. Any gain we report must exceed it on the same diffusion-map inputs.
Strongest existing invariant-discovery method. The Siamese interpretation-bottleneck network of Wetzel et al., run in two forms: as published with a one-dimensional bottleneck, and widened to a $k$-dimensional bottleneck with distance $\lVert f(x)-f(x')\rVert^2$. Both are trained on the same raw sensor pairs labelled same-trajectory or different-trajectory, with an equal hyperparameter search budget and matched compute. The paper reports similarity classification accuracy of 92% to 99% across its systems, from coordinate inputs. It reports no invariant-recovery score, so no reported result was found on Lacuna for the metric that matters here. Beating it means a higher $R^2$ for every true invariant regressed on the learned features, and recovering all invariants jointly where the bottleneck model does not, without curating the data.
Strongest non-linear multi-view method. Local CCA with diffusion maps (Yair and Talmon), using its published neighbourhood construction. For the scaling comparison, we also run untied NCCA and Deep CCA on lagged pairs. On Noisy MNIST clustering, NCCA's paper reports 99.2% for NCCA, 99.1% for Deep CCA, and 95.6% to 96.7% for kernel CCA. We cite these only to show the estimators are competitive at extracting shared structure. Beating them here means their shared coordinates explain the invariants worse than ours do, measured by $R^2$ or mutual information.
Ablations isolating each ingredient. Four variants, all on identical data:
- untied lagged CCA, which is predicted to give a degenerate spectrum near 1;
- a single fixed lag instead of averaging, which is predicted to keep oscillator phases;
- a single sensor without the cross-view term, which is predicted to keep sensor-specific invariants;
- a $\tau = 0$ cross-view-only version, which is predicted to return all shared state, not only conserved state.
Upper reference. Null-space sparse regression on true coordinates, searching for $\xi$ with $\Theta(x_t)\xi \approx \Theta(x_{t+\tau})\xi$.
Why not just…?
Why not just widen the Siamese bottleneck to several dimensions? This is the most direct competitor and we run it. Nothing in its contrastive loss stops several bottleneck units from encoding redundant functions of the dominant invariant. The published hierarchical-discovery behaviour suggests this is what happens. Our eigenspace is orthogonal by construction. Whether that difference matters in practice is settled empirically by the per-invariant $R^2$ comparison on Kepler orbits with energy and angular momentum both varying freely.
Why not just learn coordinates with an autoencoder and then run SINDy or Noether's Razor on them? Autoencoder latents can be warped arbitrarily, the non-identifiability problem flagged in the multi-view literature, so the downstream invariant search is no longer restricted to a polynomial in the true state. This path is genuinely viable when the latent happens to be near-linear in phase space. We include an autoencoder-then-null-space-SINDy pipeline as a baseline. If it matches our method on the pendulum and oscillator tasks, we will report the contribution as limited to the setting with nuisance distractors.
Why not just use kernel slow feature analysis or kernel tied CCA? That is the same objective with a different estimator, and it is a real threat on small data. The evidence against it is only about scale and robustness. Kernel methods need $O(N^3)$ dense inversion, and on Local CCA's coupled pendulum, kernel CCA failed under structured sensor noise. We run a kernel version of our objective at subsampled sizes. If it ties ours, the contribution is the objective, not the nearest-neighbour estimator, and we will say so.
Why not just run null-space SINDy on the state? On coordinate inputs this probably works as well as our method or better. That is why it serves as the upper reference rather than a competitor. It cannot be applied when the only inputs are pixels or raw sensor arrays, which is the regime this proposal targets.
Proposed Experiments
Kepler orbits. We render Kepler orbits, the central-potential system of Wetzel et al., through two cameras at different poses. Each camera view includes its own rotating distractor icon, following Yair and Talmon. The decisive check has three parts:
- the estimated intrinsic dimension of the near-unit eigenspace is 2;
- energy and angular momentum are each recovered with high $R^2$, together and from uncurated data;
- the distractor rotation, which is invariant within one sensor only, is absent.
$n$-harmonic oscillators. We render $n$-harmonic oscillators for $n$ up to 4 through two sensors and test whether the count matches the $n^2$ invariants that Noether's Razor reports. This is the case where single-lag tying should fail because of phase leakage, so it directly tests the lag-averaging claim.
Coupled pendulums. On the coupled pendulums of Yair and Talmon, the check is whether the two mode energies emerge as separate invariants. Their diffusion coordinates recover frequencies but not conserved quantities.
Shallow water (stretch). We use the PDEBench shallow-water dataset (PDEBench) with two disjoint partial sensor arrays as the views. The check is whether the leading invariant correlates with total mass, as scored against PDEBench's conservation RMSE, while neither array observes the whole field.
Metrics. Throughout, we report the number of invariants recovered against ground truth, per-invariant $R^2$, the spectral gap, the symbolic-readout match to the true formula wherever a library exists, and wall-clock time against Deep CCA and the Siamese network.
Falsification. The method fails if lag-averaged tying does not separate invariants from phases on the oscillator, or if the widened Siamese network recovers every invariant equally well on Kepler orbits.
References
- Nonparametric Canonical Correlation Analysis
- Local Canonical Correlation Analysis for Nonlinear Common Variables Discovery
- Discovering symmetry invariants and conserved quantities by interpreting siamese neural networks
- Noether's Razor: Learning Conserved Quantities
- Noether Networks: meta-learning useful conserved quantities
- PySINDy: A Python package for the sparse identification of nonlinear dynamical systems from data
- PDEBench: An Extensive Benchmark for Scientific Machine Learning