TL;DR
ST-JEPA is the first foundation model for spatial transcriptomics to use a Joint-Embedding Predictive Architecture (JEPA), allowing it to learn the relationship between gene expression and spatial organization across multiple biological scales. By predicting masked information in latent space rather than reconstructing raw data, it achieves state-of-the-art performance in identifying tissue "niches" and integrating data across different imaging technologies without needing explicit batch correction.
What Problem Does This Paper Address?
Spatial transcriptomics (ST) is a powerful technology that measures gene activity while preserving the spatial coordinates of cells within a tissue. To truly understand biology, we need to model how molecular programs inside a cell relate to its "neighborhood" and how those neighborhoods form functional tissue structures (niches).
Existing computational methods usually fall into two traps:
- Single-scale focus: They are often tailored for one specific task, like clustering neighborhoods, and fail to provide a general-purpose representation of the data.
- Spatial neglect: Some recent foundation models either ignore spatial coordinates entirely during training or lack a hierarchical structure that connects genes to cells to neighborhoods.
Furthermore, integrating data from different platforms (e.g., MERFISH vs. STARmap) is notoriously difficult because different technologies often measure different sets of genes, creating "batch effects" that obscure the underlying biology.
Methods
The authors introduce ST-JEPA, which stands out through three main innovations:
- Joint-Embedding Predictive Architecture (JEPA): Unlike traditional models that try to reconstruct "pixels" or raw gene counts (which are often noisy), ST-JEPA predicts the latent representation of masked parts of a neighborhood. This forces the model to learn high-level biological structures rather than getting bogged down in low-level noise.
- Multi-scale Graph Tokenization: The model organizes cells into a $k$-nearest neighbor graph. It converts this graph into a sequence of tokens representing genes, cells, and neighborhoods. This allows a single forward pass to generate embeddings at three different biological resolutions.
- Technology Metatokens: To handle data from different platforms, the authors append a "metatoken" to the sequence. This acts as a signal to the model about the data's origin, allowing it to learn batch-invariant features implicitly.
Datasets
The model was trained and evaluated on mouse brain spatial transcriptomics data from two major technologies:
- MERFISH: 1,085 genes.
- STARmap: 1,015 genes.
- The Challenge: These two panels only have 431 genes in common. ST-JEPA was trained on the full panels of both, demonstrating its ability to generalize even when gene sets don't fully overlap.
Key Insights and Ideas
- Rank Trumps Coordinates: One of the most surprising findings from their ablation studies was that rank-based spatial encoding (e.g., "this is the 1st closest neighbor") dramatically outperformed absolute coordinate-based encoding ($x, y$ positions). Absolute coordinates vary too much between samples, while relative neighborhood structure is a more robust biological signal.
- Latent Prediction as a Filter: By predicting in latent space, the model naturally ignores batch-specific artifacts. If a piece of information (like a technical glitch) isn't useful for predicting the representation of a neighboring cell, the model learns to ignore it.
- Raw Counts are Sufficient: In a departure from standard single-cell workflows that require complex normalization (like Pearson residuals), ST-JEPA performed best using raw gene counts. The rank-based tokenization provides enough implicit normalization on its own.
Key Results
- Niche Identification: ST-JEPA achieved a weighted Normalized Mutual Information (NMI) of 0.67, outperforming specialized state-of-the-art methods like CellCharter (0.65) and Banksy (0.63).
- Zero-Shot Integration: Despite no explicit objective to "align" datasets, ST-JEPA achieved superior batch integration (measured by iLISI) compared to other high-performing niche identification methods.
- Stability: The model was found to be the most stable across different clustering resolutions, meaning its biological "maps" are more reliable regardless of how finely you tune the clustering parameters.
Limitations and Future Work
- Tissue Diversity: The current study focuses primarily on the mouse brain. While the brain is highly structured and a great benchmark, future work is needed to see how ST-JEPA handles more "disordered" tissues, such as tumors.
- Scaling: With 23.7 million parameters, ST-JEPA is a "small" foundation model by modern standards. The authors suggest that scaling the architecture and training it on even larger, more diverse datasets (like the Human Cell Atlas) could unlock even deeper biological insights.
- Cross-Panel Prediction: While the model handles non-overlapping genes well, the authors haven't yet fully explored using it to "impute" or predict the expression of genes that were not measured in a specific technology.
Summary updated