BigBang-Proton Technical Report: Next-Word-Prediction is Scientific Multitask Learner

Hengkui Wu, Liujiang Liu, Jihua He, Qihao Wang, Keke Zhao, Shuyang Hu, Renle Fu, Dahao Liang, Lingyu Zeng, Bruce Liu, Yuan Liu, Jin Zhan, Jiaqiang Niu, Xinglong Jia, Yaqin Hu, Wenjun Ji, Panpan Chi, Ken Chen, Hengyuan Wu, Yingsi Xin, Yongfeng Zhu, Yuexin Wang, Manqi Ruan, Ningtao Bian, Xiaohua Wu, Weipeng XuarXiv 2025 · DOI · arXiv

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing, yet they frequently struggle with the precision required for scientific and mathematical reasoning. This technical report introduces BigBang-Proton, a 1.5-billion-parameter multi-task model designed to bridge this gap. By conceptualizing scientific data—ranging from subatomic particle trajectories to genomic sequences—as a structured language, BigBang-Proton achieves state-of-the-art performance across diverse scientific domains.

The model addresses two primary limitations of current architectures: the fragmentation of numerical data caused by standard Byte-Pair Encoding (BPE) and the quadratic scaling constraints of traditional Transformers. Through novel architectural shifts, BigBang-Proton functions as a unified foundational model capable of cross-disciplinary knowledge transfer.

Architectural Innovations

BigBang-Proton introduces three fundamental shifts to the standard Transformer architecture:

  1. Binary Patch Encoding: To avoid the mathematical errors inherent in subword tokenization, the model operates directly on raw UTF-8 byte sequences. This preserves numerical integrity and allows the model to emulate the logic of an Arithmetic Logic Unit (ALU).
  2. Monte Carlo Attention: This sparse attention mechanism utilizes "Inter-Patch Delegation," where specific bytes represent entire data patches. This reduces computational complexity to a linear scale, theoretically allowing for context lengths capable of modeling massive systems.
  3. Theory-Experiment Learning: The model aligns experimental numerical data with theoretical natural language descriptions. This multimodal approach allows the model to pair raw data, such as particle momentum, with conceptual labels like "charged pion."
Figure 2: The Binary Patch Encoding process converting jet information into byte patches.

Training Methodology and Datasets

The model underwent pretraining on a heterogeneous corpus designed to foster "Structure Learning." Key datasets included:

  • Arithmetic: 300 million synthetic examples of operations up to 50 digits.
  • Particle Physics: 110 million particle jets for Jet Origin Identification (JoI).
  • Genomics: 27.2 billion nucleotide tokens from the OpenGenome project.
  • Materials Science: 1.5 million crystalline structures from the MPtrj dataset.
  • General Text: High-quality corpora including Wikipedia, ArXiv, and GitHub (SlimPajama).

Key Results

BigBang-Proton demonstrates significant performance gains over both general-purpose LLMs and specialized scientific models.

Figure 1: Comparative performance of BigBang-Proton across scientific domains.
  • Mathematical Precision: The model achieved 100% accuracy on 50-digit addition tasks. In contrast, frontier models like DeepSeek R1 and ChatGPT o1 scored 13% and 5%, respectively. This suggests the model has mastered the underlying "carry principle" of arithmetic rather than relying on memorization.
  • Materials Science: In predicting crystal formation energy, the model reached a Mean Absolute Error (MAE) of 0.043 eV, significantly outperforming projected benchmarks for GPT-5 and Grok 4.
  • Particle Physics: The model attained 51.29% accuracy in 11-class jet tagging, nearly matching specialized models like ParticleNet (55.29%) and far exceeding the near-random performance of standard LLMs.
  • Genomics: BigBang-Proton outperformed the biology-specific model Evo, achieving lower perplexity (2.8 vs 3.1) despite using only one-third of the training data.

Discussion and Future Work

The results suggest that learning the structural patterns of one scientific domain (e.g., particle jets) enhances the model's ability to understand others (e.g., protein structures). While Monte Carlo Attention provides a path toward massive context windows, current implementations still face hardware-bound constraints.

Future iterations will focus on scaling to "universe-scale" simulations. The research team aims to apply BigBang-Proton to complex phenomena such as nuclear fusion and quantum materials, testing the hypothesis that a model trained on the totality of physical data will eventually converge toward the fundamental laws of the universe.

Summary updated

BibTeX
@misc{wu2025bigbang,
  author = {Hengkui Wu and Liujiang Liu and Jihua He and Qihao Wang and Keke Zhao and Shuyang Hu and Renle Fu and Dahao Liang and Lingyu Zeng and Bruce Liu and Yuan Liu and Jin Zhan and Jiaqiang Niu and Xinglong Jia and Yaqin Hu and Wenjun Ji and Panpan Chi and Ken Chen and Hengyuan Wu and Yingsi Xin and Yongfeng Zhu and Yuexin Wang and Manqi Ruan and Ningtao Bian and Xiaohua Wu and Weipeng Xu},
  title = {{BigBang-Proton Technical Report: Next-Word-Prediction is Scientific Multitask Learner}},
  year = {2025},
  eprint = {2510.00129},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.00129},
}
Figure