Geometry Matters in Packed 3D Attention

Amil Khan, Anantajit Subrahmanya, B. S. ManjunatharXiv 2026 · arXiv

TL;DR

PACK3D is a novel attention operator designed for high-token 3D volumetric data that decouples physical geometry from memory layout. By indexing attention using true 3D grid coordinates rather than sequence positions, it prevents models from erroneously learning the "storage order" of data, enabling efficient, spatially-aware long-range modeling that is invariant to how the data is packed in memory.

What Problem Does This Paper Address?

When we process 3D volumes (like medical scans or microscopy) with Transformers, we first "patchify" the volume and flatten it into a 1D sequence of tokens. This process, called linearization, creates a major headache: the order of tokens in computer memory (the layout) no longer matches their physical 3D relationship.

Standard positional encoding methods—like learned embeddings or Rotary Positional Embeddings (RoPE)—usually index tokens by their position in that 1D sequence. The authors identify a "packed-geometric attention problem": if the model learns to rely on the memory layout to understand shape, it fails the moment you change the packing strategy (e.g., switching from a simple raster scan to a more cache-friendly Morton/Z-order curve). Essentially, the model learns the "map" of the memory instead of the "territory" of the 3D object.

Methods

The authors introduce PACK3D, a local–global attention operator that enforces a strict separation between systems-level packing and modeling-level geometry.

  1. Local-Global Factorization: To handle hundreds of thousands of tokens, PACK3D uses exact attention for nearby neighbors (preserving high-frequency details) and compresses distant context into 3D block summaries.
  2. Axial Grid-Coordinate RoPE: Instead of using sequence indices, PACK3D applies RoPE based on the actual $(z, y, x)$ coordinates of the tokens in the 3D grid.
  3. Coordinate-Aware Compression: Crucially, the authors prove that applying these rotations before pooling tokens into summaries (Theorem 7) ensures that the resulting attention logits remain spatially interpretable. The query doesn't just attend to a "blob" of data; it attends to the content through the lens of its relative 3D displacement.
  4. Hardware-Aligned Implementation: The operator is implemented using fused Triton kernels. This avoids materializing massive intermediate tensors, allowing the model to match the speed of simple windowed attention while providing the long-range context that windowing lacks.

Datasets

  • Allen WTC–11 hiPSC Dataset: A massive 3D live-cell microscopy dataset containing over 200,000 cells. The task involves predicting 15 different cellular structure masks from fluorescence, brightfield, and DNA channels.
  • Synthetic Repacking Benchmark: A controlled environment where the physical labeling rule is fixed, but the memory order of tokens is scrambled (Raster, Morton, Block-contiguous, and Random) to test if the model truly understands geometry.
  • Medical Segmentation Decathlon (MSD): Used for external transfer testing on Heart and Spleen 3D volumes.

Key Insights and Ideas

  • Packing is for Systems, Coordinates are for Modeling: This is the paper's central design principle. A model's understanding of space should not break just because an engineer optimized the data loader.
  • The Failure of Sequence-Index RoPE: In the synthetic task, models using standard sequence-based positional encodings solved the training layout perfectly but collapsed to near-random performance when the tokens were reordered. In contrast, PACK3D remained exact across all layouts.
  • Rotation Before Pooling: The mathematical "aha" moment is Proposition 2. By rotating keys before they are averaged into a block summary, the attention mechanism preserves the relative 3D geometry of the tokens that formed that summary. This prevents the global context from collapsing into "anonymous pooled content."

Key Results

PACK3D demonstrates that geometry-aware attention is both more accurate and more robust than standard approaches:

  • Competitive with U-Nets: On the Allen-15 joint task, PACK3D (30.1M parameters) outperformed a parameter-matched 3D U-Net (31.9M parameters), achieving a best macro Dice of 0.8228 vs. 0.8134. Table 1: Allen-15 joint task results
  • Superior to Learned Positions: At a scale of 262k tokens, PACK3D improved over a matched learned-position baseline by a massive +0.0978 best macro Dice, showing that learned positions struggle to capture geometry at high resolutions.
  • Efficiency at Scale: The Triton implementation is 6.49x faster than dense attention at 65k tokens and can scale to handle over 8 million tokens on an H200 GPU. Table 2: Positional mechanism comparison
Table 1: Allen-15 joint task: PACK3D is competitive with a parameter-matched 3D U-Net, using a coordinate-indexed attention operator in place of convolution. Best and final macro Dice, mean ± std over three seeds. The +0.0094/+0.0135 gap is consistent in sign but modest against seed variation; see text for the significance annotation.
Table 2: At matched parameter count, physical-coordinate RoPE beats sequence-index RoPE and compact learned alternatives. Short Allen gates; best/final macro Dice. The matched-capacity comparisons are the 3.06M rows — in particular packed-sequence vs. axial RoPE at identical 3.06M — which isolate the coordinate system. The depth-8 pair differs in capacity (158.5M vs. 24.3M) and is shown only to confirm the effect persists at depth.

Limitations and Future Work

The authors are careful to bound their claims. They do not claim PACK3D is a "U-Net killer" or the new SOTA for all segmentation. The findings are primarily architectural: for packed 3D fields, physical coordinates are the only reliable way to index attention.

Future work includes exploring more sophisticated frequency allocations for the axial RoPE (currently a balanced default) and testing the operator's performance across more diverse imaging modalities, as initial transfer results to medical CT/MRI data were mixed. Additionally, the current model uses a single seed for some transfer tasks, which the authors note is insufficient for establishing definitive generalization.

Summary updated

Abstract

Packing a 3D volume into a token sequence is not merely long-sequence modeling. Once the volume is packed, sequence order no longer encodes the geometry of the sample---so a positional mechanism indexed by sequence position can learn the memory layout rather than the object. This is a general failure mode for attention over packed geometric data. We introduce PACK3D, a local-global attention operator that separates memory layout from attention geometry. PACK3D keeps local neighborhood attention exact, compresses only distant context into contiguous 3D block summaries, and indexes every query-key interaction with axial RoPE on true token-grid coordinates rather than packed positions. Because the rotation is applied before a block is summarized, the query-to-summary logit remains a weighted sum of query-relative 3D displacements; compressed global context therefore stays spatially interpretable after packing, instead of collapsing to anonymous pooled content. We measure the failure mode directly. On a controlled synthetic task---train on one packing, change only the memory order of the tokens, hold the physical labeling rule fixed---learned positional embeddings and packed-sequence RoPE solve the training layout perfectly yet collapse under Morton, block-contiguous, and random repacking, while axial grid-coordinate RoPE stays exact throughout. The same coordinate system carries to real data: on Allen WTC-11 hiPSC microscopy, PACK3D improves over a parameter-matched 3D U-Net on the 15-structure joint task and over a matched learned-position local-global baseline, with controls isolating the coordinate system, not encoder capacity, as the source of the gain. A fused Triton implementation matches windowed-attention latency at 65k tokens while supplying the long-range context windowing cannot---so the operator is not only correct but trainable at scale.
BibTeX
@misc{khan2026geometry,
  author = {Amil Khan and Anantajit Subrahmanya and B. S. Manjunath},
  title = {{Geometry Matters in Packed 3D Attention}},
  year = {2026},
  eprint = {2610.38694},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38694},
}
Figure