Does Recurrence Make Point Cloud Models Need a Curriculum?

State space models (SSMs) are sequence models that carry a running memory and update it one token at a time. Because their cost grows linearly with sequence length, they have become a popular cheap alternative to attention for 3D point clouds. To use them, the unordered set of 3D points has to be flattened into a line, usually by tracing a space-filling curve through space. Real scans are messy: dense in some places, sparse in others, full of background clutter. When such a scan is flattened, consecutive tokens are sometimes neighbours in space and sometimes far apart, because the curve jumps across a boundary or crosses an empty region. A recurrent memory sees each such jump as an abrupt cut in its input. We think this makes SSMs especially fragile on irregular data early in training. If so, an easy-to-hard training schedule should help them more than it helps attention models, which have no running memory to break.

This note proposes to test that claim directly. The test pairs a curriculum over geometric irregularity (training first on clean, isolated, evenly sampled objects, then gradually adding clutter and density variation) with a simple architectural fix: the SSM is told how far apart in 3D two consecutive tokens are. Each ingredient gets measured on its own and both together. If the curriculum's benefit is specific to recurrence, we can say why curricula help on point clouds, rather than just adding one more training trick.

Core Claim

The claim has three parts, each of which could turn out false. First, on real-world scans, a curriculum ordered by geometric irregularity improves a pure bidirectional SSM more than it improves an attention model trained the same way. Second, much of that extra benefit comes from the recurrence being disrupted at serialization jumps. If the SSM is made aware of inter-token distances, the curriculum's benefit should shrink. Third, the benefit appears mainly in the regimes where When Do Curricula Work? found curricula help at all, namely high noise and tight training budgets. It should vanish on clean synthetic data with a generous budget.

The closest existing work combines only two of these pieces. The point cloud SSM line serializes points with Hilbert curves and dual scans in PointMamba, with six axis permutations and order prompts in Point Cloud Mamba, and with zigzag paths in ZigzagPointMamba. It adds coupled forward-backward SSMs behind an attention front-end in PointABM. All of these train with uniform sampling throughout. In this proposal they supply the method, and we reuse their serialization. On the curriculum side, MambaNUT is the only SSM-plus-curriculum work in the file. It moves a 2D Mamba tracker from daytime to nighttime frames and reports up to 3.7% precision gain. It works on image grids, so serialization jumps and 3D density never arise, and it does not ask whether the gain depends on the recurrent architecture. Our contribution is the interaction test: curriculum × architecture × geometry-awareness. A new tracker or classifier is not the goal.

Research Program

The testbed is a pure, attention-free bidirectional SSM for point patches. It keeps PointMamba's patch tokenizer and Hilbert/Trans-Hilbert serialization. It replaces each unidirectional Mamba block with a coupled forward and backward scan over the same sequence. Two variants of that block are compared. The first shares projection weights between directions, as in the BiMamba block of Caduceus. The second uses direction-specific projections, which Mamba in Speech found better for learning direction-dependent dynamics. We do not claim this backbone as new. It is the cleanest way to isolate recurrence while staying linear-time, unlike PointABM's quadratic attention front-end.

The geometry-aware variant changes one thing. In a selective SSM, each token chooses a discretization step $\Delta_t$, which controls how much the memory decays before the token is absorbed. We add a term driven by the physical gap between neighbouring tokens:

$$\Delta_t^{\rightarrow} = \mathrm{softplus}\big(w^\top x_t + \beta \log(\epsilon + \lVert p_t - p_{t-1} \rVert)\big), \qquad \Delta_t^{\leftarrow} = \mathrm{softplus}\big(w^\top x_t + \beta \log(\epsilon + \lVert p_t - p_{t+1} \rVert)\big).$$

Here $x_t$ is the token feature, $p_t$ is its patch centre in 3D, $w$ and $\beta$ are learned, and $\epsilon$ is a small constant. With a large $\Delta_t$ the memory largely forgets its past before absorbing the new token. A curve jump can therefore act as a soft state reset, in the forward direction using the previous token and in the backward direction using the next one. This takes a few lines of code and leaves the hardware-friendly parallel scan untouched.

The curriculum orders samples by geometric difficulty, using signals that need no extra labels. ScanObjectNN already provides stages of increasing irregularity for the same physical objects. OBJ_ONLY is the isolated object, OBJ_BG adds background points, and PB_T50_RS adds clutter, translation, rotation and scaling, as described in Revisiting Point Cloud Classification. Within a stage, samples are further ranked by two quantities: a density-irregularity score (the coefficient of variation of nearest-neighbour distances) and the number of large serialization jumps after Hilbert ordering.

The pacing schedule moves the sampling mixture from easy to hard, much like MambaNUT's sampling scheduler. We test it in two forms. The first is a fixed linear schedule. The second is a self-paced version in the spirit of Self-Paced Learning for Latent Variable Models and Adaptively Point-weighting Curriculum Learning, which advances when training loss on the current stage plateaus. The design respects the "feasible initialization" condition from A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula: isolated objects are a base stage the model can actually solve. Stages that differ only moderately in difficulty match that paper's Goldilocks zone. We cite it as motivation for the stage design, not as a guarantee, since its analysis concerns autoregressive self-improvement.

Fairness is built in. Every non-curriculum run draws from the identical pool of training objects and variants, sees the same total number of samples, and receives the same augmentation. The only difference is the order and mixture over time.

Baselines

The naive reference point is the same bidirectional SSM trained on PB_T50_RS training data with standard augmentation and uniform sampling. It is the "do nothing" condition every effect is measured against. We also train a stronger uniform version that samples from the full multi-variant pool in fixed proportions. This rules out a gain that comes simply from seeing the easy variants at all.

The strongest published SSM competitor is PointMamba, which reports 89.31% overall accuracy on PB_T50_RS (PointMamba). That paper also reports Point-MAE at 89.04% and PointGPT at 89.17% on the same split. We rerun PointMamba from its released configuration under our seeds and epoch budget, with and without the curriculum. Beating it means exceeding our reproduced number by more than the seed standard deviation, over at least five seeds. The published figure is a sanity check, not the target, because pre-training protocols differ.

PointABM claims margins of +3.33% over PointMamba on PB_T50_RS, yet reports 88.29% for its pre-trained model on that split (PointABM). That is below PointMamba's own 89.31%. The two papers evidently used different protocols, so we do not lean on either cross-paper number. We include PointABM rerun under our protocol as the strongest existing bidirectional SSM. Point Cloud Mamba (88.1% overall accuracy reported on ScanObjectNN, Point Cloud Mamba) serves as the multi-permutation, unidirectional reference.

The attention control is Point-MAE trained with exactly our curriculum and budget. This is the comparison the core claim depends on.

Two ablations isolate the key ingredient. In the pacing-only control, the training subset grows on the same schedule, but samples are added in random order. This addresses the pacing artifact identified in When Do Curricula Work?. In the anti-curriculum control, hard samples come first.

Why not just…?

Why not just use heavier static augmentation (random point dropping, jitter, clutter mixing) throughout training? This is the most likely way the proposal fails, so it is a full arm of the experiment rather than a strawman. The matched-exposure uniform baseline sees every corruption level that the curriculum does, in the same total amount, from the first epoch. If it matches the curriculum, our ordering claim is wrong and we will say so. There is modest prior evidence that static mixing falls short under noise. Adaptively Point-weighting Curriculum Learning found plain Mixup inferior to weight-scheduled Mixup on noisy WebVision. Revisiting Point Cloud Classification shows that standard augmentation does not close the synthetic-to-real gap. Neither result concerns point cloud SSMs.

Why not just fix the architecture and drop the curriculum? If distance-aware steps let the memory reset at curve jumps, the fragility we blame might disappear, and the curriculum would be unnecessary. We take this seriously, and the factorial design is built to settle it. If the geometry-aware model gains nothing from the curriculum while the plain bidirectional SSM does, that supports our causal story, and the deliverable becomes the cheap architectural fix. If both ingredients add up, both are worth using. If neither helps, recurrence fragility is not the bottleneck.

Why not just train longer, or use a curriculum-free better serialization such as zigzag or six permutations? When Do Curricula Work? found that with ample budget on clean data, curricula add nothing. We therefore expect, and will report, a shrinking gap as the budget grows, and the budget sweep shows where it closes. Better curves like those in ZigzagPointMamba reduce jumps but cannot remove them. We include zigzag serialization as an extra backbone variant to check whether the curriculum effect depends on the curve chosen.

Proposed Experiments

The main experiment is a factorial study on ScanObjectNN PB_T50_RS. It crosses backbone (bidirectional SSM vs. Point-MAE), geometry-aware steps (on vs. off, SSM only) and schedule (uniform, matched-exposure uniform, pacing-only random, anti-curriculum, fixed curriculum, self-paced curriculum). Every cell is trained at three budgets, from 25% of the standard epoch count to 100%. Each cell is run with five seeds, and we report overall accuracy with standard deviations. We also report OBJ_BG and OBJ_ONLY accuracy, to check that the curriculum does not trade easy-split performance for hard-split performance.

The first decisive check is the interaction. The core claim holds only if the curriculum-minus-uniform gain for the SSM exceeds the same gain for Point-MAE by more than seed noise, and if that gap is clearest at reduced budgets.

The second check is mechanistic. We bin test samples by number of serialization jumps and by density-irregularity score, then plot accuracy against bin. If our story is right, the curriculum's gain should be concentrated in high-jump samples for the plain SSM, and that concentration should flatten once geometry-aware steps are on. We also measure how much each token's hidden state changes across large jumps, early and late in training, with and without the curriculum.

The third check is transfer under stress. We train on ModelNet40 and test on ScanObjectNN, where Revisiting Point Cloud Classification reports severe degradation. In this setting the curriculum stages are made from synthetic clutter and density corruption, since ModelNet40 has no natural variants. We further evaluate under held-out point dropout rates, to see whether the curriculum gives robustness beyond the levels seen in training.

If compute allows, a final probe applies the density-based schedule to S3DIS segmentation. Density there varies within a single scene, so the ranking has to be done per region rather than per sample. Point Cloud Mamba reports 74.1 mIoU there (Point Cloud Mamba). That reference fixes the backbone scale, and this experiment is not needed for the core claim.