Generators that turn noise into images in a single network call are attractive because they are fast, but they are hard to train. A recent result makes the difficulty concrete. When a model learns the average velocity that carries a noise sample all the way to an image (the MeanFlow approach), the curved paths of the underlying generative process make the training loss spiky and unstable. The fix in Overcoming the Curvature Bottleneck in MeanFlow is to borrow straightened noise–image pairings from an already-trained teacher model and to discard the 10% of pairs that are farthest apart. This proposal asks whether the same instability can be handled from the optimizer side instead, using the training data the model already has.
The idea fits in two sentences. A small subset of training pairs, the long, high-curvature ones, sends the weights into steep, narrow regions of the loss. We train on those pairs anyway, but we make the weights robust to the worst-case perturbation computed from that subset alone. This is a targeted version of sharpness-aware minimization: it keeps all the data, needs no teacher, and its extra cost is a fraction of standard SAM's doubled cost. The first deliverable is a measurement. We will establish whether "rugged landscape" in these models really means high curvature in weight space, or whether it means noisy gradients. Those two diagnoses call for different remedies, and the answer decides whether any SAM variant belongs here.
Background and the gap
Flow matching (Flow Matching for Generative Modeling) trains a network to predict the velocity along straight lines from noise $x_0$ to data $x_1$. When noise and data are paired at random, these lines cross. The network must then average conflicting velocities, so the learned field curves and few-step sampling breaks. Three lines of work straighten the paths by changing what the network is trained on, not how it is optimized:
- Reflow retrains on the model's own noise–image pairs. Simple ReFlow refines this with loss normalization and reduced dropout. Analyzing and Mitigating Model Collapse in Reflow Methods adds real-data anchoring.
- Learning Straight Flows by Learning Curved Interpolants learns curved per-pair paths so that the averaged field becomes straight.
- Geometry-Aware Image Flow Matching moves the paths onto a hypersphere.
None of these papers measures weight-space geometry. Re-MeanFlow is the one paper that ties path curvature to loss roughness, but it infers roughness from loss instability. It never computes Hessian eigenvalues.
Sharpness-aware minimization (Sharpness-aware Minimization for Efficiently Improving Generalization) takes a gradient-ascent step of radius $\rho$ to find the locally worst nearby weights. It then descends using the gradient at that point. In the supervised setting it is notably robust to corrupted training targets, reaching 79.9% accuracy on CIFAR-10 with 80% label noise. High-curvature pairs in flow training behave somewhat like noisy targets, because the regression target $x_1 - x_0$ for a long pair disagrees sharply with the average field near it. This analogy motivates the proposal, but it is only an analogy, and the measurement phase tests it directly.
One paper constrains the design. Simple ReFlow reports that straight velocity fields require a larger Lipschitz constant, meaning the field must change faster with its input. It also reports that regularizers like heavy dropout hurt. If SAM's weight-space flatness also smooths the function in input space, it could push the model toward exactly the curved, low-Lipschitz fields we want to avoid. We treat this as a primary risk and measure it, rather than assuming that flatness helps.
Core Claim
In one-step average-velocity models trained without a teacher, we expect loss spikes to coincide with growth in the largest Hessian eigenvalue $\lambda_{\max}$. We expect that growth to be driven disproportionately by the longest noise–data pairs in each batch.
If that holds, a SAM ascent step computed only from those pairs should suppress the spikes while still learning from them. It should match or beat the truncation heuristic and close part of the gap to teacher-based Re-MeanFlow.
Concretely, at each step we rank the batch pairs by distance $\|x_1 - x_0\|$, the same curvature proxy Re-MeanFlow uses for pruning. Let $S$ be the top fraction $q$ of that ranking. We compute the perturbation
$$\hat\epsilon = \rho\,\frac{\nabla_w L_S(w)}{\|\nabla_w L_S(w)\|_2},$$
where $L_S$ is the MeanFlow loss on the subset $S$ and $w$ are the weights. The update uses the full-batch gradient $\nabla_w L(w+\hat\epsilon)$.
The ascent pass touches only a fraction $q$ of the batch, so overhead is roughly $1+q$ times a standard step rather than 2 times. The construction follows SAM's own observation that perturbations computed on sub-batches (its "$m$-sharpness") track generalization better than full-batch ones. Here the sub-batch is chosen by trajectory curvature rather than at random.
Research Program
Phase one: diagnosis. We train vanilla MeanFlow from scratch on CIFAR-10 and log three quantities through training:
- $\lambda_{\max}$ of the training loss, estimated by power iteration on Hessian–vector products;
- the gradient noise scale, which is the ratio of gradient variance to squared mean gradient;
- both quantities decomposed by pair-distance quantile and by timestep bin.
The decision rule is fixed in advance. If loss spikes are preceded by $\lambda_{\max}$ growth concentrated in the far-pair quantile, the sharpness hypothesis stands. If spikes instead track gradient variance with flat $\lambda_{\max}$, the problem is noise. In that case the right tools are larger batches, clipping, or variance reduction, and the proposal pivots to reporting that negative result.
Following Beyond Spectra: Eigenvector Overlaps in Loss Geometry, which argues that train-loss spectra alone are incomplete, we also compute the overlap between the top Hessian directions of the far-pair loss and the near-pair loss. Low overlap would mean the far pairs pull the weights along their own directions, which is what targeted ascent exploits.
Phase two: method. We implement targeted SAM and tune $\rho$ and $q$. SAM is known to be sensitive to $\rho$ (Adaptive Sharpness-Aware Minimization with a Polyak-type Step size), so every SAM variant gets the same tuning budget. A secondary variant scales $\rho$ by timestep bin, but only if phase one shows sharpness concentrated near particular timesteps. The Lipschitz-boundary story in Simple ReFlow suggests this might happen near $t\to 0$ and $t\to 1$.
Phase three: mechanism checks. For every trained model we measure three things:
- the input-Jacobian norm of the learned field, as a proxy for its Lipschitz constant;
- trajectory straightness, measured as the deviation of many-step ODE paths from the chord between their endpoints;
- the gap between 1-step and many-step samples.
These checks tell us whether flatness bought stability at the cost of straightness, which is the Simple ReFlow risk.
Baselines
The naive reference is vanilla MeanFlow trained from scratch with AdamW and an exponential moving average of the weights, all at the standard settings. The file reports no CIFAR-10 1-step MeanFlow FID; no reported result found on Lacuna for that configuration. We therefore produce our own number under our compute budget.
The baseline closest to the proposal's spirit is MeanFlow with Re-MeanFlow's distance truncation but without the teacher, so that the top 10% longest pairs are dropped. This isolates whether keeping and regularizing the hard pairs beats discarding them.
The strongest existing method is full Re-MeanFlow with teacher couplings. On ImageNet-512 it reports 1-step FID 8.6, against 30.9 for baseline MeanFlow (Overcoming the Curvature Bottleneck in MeanFlow). We do not expect a teacher-free method to beat this. The bar for success is recovering a substantial fraction of the 30.9-to-8.6 gap without a teacher. A further question is whether targeted SAM on top of teacher couplings improves on 8.6.
For few-step context on CIFAR-10, the file reports the following FIDs:
| Method | Steps (NFE) | FID | Source |
|---|---|---|---|
| 1-RF | 1 | 15.27 | Analyzing and Mitigating Model Collapse in Reflow Methods |
| 1-RF-RA | 1 | 12.27 | Analyzing and Mitigating Model Collapse in Reflow Methods |
| Learned curved interpolants | 2 | 4.61 | Learning Straight Flows by Learning Curved Interpolants |
These two-stage or specially parameterized methods are reference points, not like-for-like comparisons.
Fairness is defined by wall-clock compute. Every baseline gets extra optimizer steps equal to the proposal's overhead, so that roughly 20% more steps are matched against $q=0.2$. Global SAM is compared at its full doubled cost, again against step-matched baselines. All methods share the data, architecture, batch size and hyperparameter-search budget.
The key ablation replaces the distance-ranked subset $S$ with a random subset of the same size. This keeps cost and perturbation radius identical. If curvature-selected ascent does not beat random-subset ascent, the trajectory-curvature link is not doing the work, and the method reduces to cheap SAM.
Why not just…?
Why not just use mini-batch optimal-transport pairing? Optimal-transport pairing reduces path crossing in the training targets. However, the curved-interpolants paper argues that the averaged field stays curved, because mini-batch transport only approximates the global one. FM-OT in Flow Matching for Generative Modeling still relies on multi-step sampling, reaching ImageNet-64 FID 14.45. Pairing and optimizer choice are not exclusive, so we include an OT-paired MeanFlow arm, with and without targeted SAM. If OT pairing alone eliminates the far-pair $\lambda_{\max}$ spikes in phase one, that is a genuine threat. We would report it as such.
Why not just add gradient clipping, a bigger batch, or more weight averaging? This is the most serious alternative, because these tools fix gradient noise directly. Phase one exists to separate the two explanations. If spikes are a variance phenomenon, a clipped, large-batch, compute-matched run should equal targeted SAM, and we would conclude that SAM is unnecessary. Stochastic weight averaging (Averaging Weights Leads to Wider Optima and Better Generalization) finds wide basins after the fact. It cannot stop a mid-training spike from corrupting the trajectory, but we run it anyway as a zero-cost flatness baseline.
Why not just use standard global SAM? Global SAM spreads its perturbation budget over pairs that are already easy. It also doubles cost, which is a significant burden for large generative models. The comparison against global SAM at doubled compute shows whether targeting matters beyond cost savings. We would also report global SAM winning outright if that is what happens.
Why not just drop the long pairs, as Re-MeanFlow does? Truncation is cheap and evidently works with a teacher. But discarding the farthest pairs may cost coverage of unusual images. We therefore report precision and recall alongside FID. Targeted SAM must beat truncation on recall at equal FID to justify itself.
Proposed Experiments
The main grid runs on CIFAR-10 at 1 and 2 network evaluations and on ImageNet-64. The method arms are vanilla MeanFlow, truncation, OT-paired, clipped large-batch, global SAM, random-subset SAM and targeted SAM. Each reports FID, precision/recall, the loss-spike count, peak $\lambda_{\max}$, the input-Jacobian norm and trajectory straightness. A stretch run applies targeted SAM to teacher-coupled Re-MeanFlow on ImageNet-512, against the reported 8.6.
There are three decisive checks:
- whether far-pair sharpness precedes spikes, testing the diagnosis;
- whether targeted SAM beats both random-subset SAM and truncation at matched compute, testing the method;
- whether it does so without lowering trajectory straightness, testing the Lipschitz concern from Simple ReFlow.
Failing the first check redirects the project to a clean negative result on SAM for generative flows. That result is still worth publishing, because the file shows no prior evaluation of it.
References
- Overcoming the Curvature Bottleneck in MeanFlow
- Flow Matching for Generative Modeling
- Simple ReFlow: Improved Techniques for Fast Flow Models
- Analyzing and Mitigating Model Collapse in Reflow Methods
- LEARNING STRAIGHT FLOWS BY LEARNING CURVED INTERPOLANTS
- Geometry-Aware Image Flow Matching
- Sharpness-aware Minimization for Efficiently Improving Generalization
- Beyond Spectra: Eigenvector Overlaps in Loss Geometry
- Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler
- Averaging Weights Leads to Wider Optima and Better Generalization