Introduction
JT-Safe is a 35.4B parameter Large Language Model (LLM) designed to address the fundamental limitations of current AI safety paradigms. While most safety efforts focus on post-training alignment, JT-Safe integrates safety and trustworthiness directly into the pre-training phase. By utilizing "Data with World Context" (DWC)—which includes metadata such as source, time, and safety ratings—the model is anchored in real-world knowledge, significantly reducing hallucinations and improving safety without compromising general performance.
The Limitations of Post-Training Safety
Current LLM safety strategies predominantly rely on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). These methods act as peripheral controls rather than addressing the root cause of safety issues: the pre-training data itself. Standard pre-training treats data as a flat sequence of tokens, lacking grounding in the real world. This leads to "cognitive uncertainty," where a model cannot distinguish between a reliable fact and a biased or incorrect statement seen in a low-quality context. Furthermore, aggressive post-training alignment can lead to "performance reverse scaling," where a model's general capabilities degrade as it becomes overly cautious.
Methodology: Data with World Context (DWC)
The core innovation of JT-Safe is the Data with World Context (DWC) framework. Instead of training on raw text, the model processes text paired with rich metadata, including:
- Basic Attributes: Time and location of the data.
- Categories: Domain-specific labels (e.g., Physics, Politics).
- Source Information: Distinguishing between authoritative journals and personal blogs.
- Safety-related Attributes: Explicit safety ratings and context.
This metadata acts as a semantic routing mechanism, allowing the model to activate specific parameter subspaces based on the context of the input.
Training Pipeline and Architecture
JT-Safe follows a rigorous three-stage pre-training pipeline totaling 6.2 trillion tokens:
- General Pre-training: 4.5 trillion tokens of high-quality general data.
- DWC Enhancement: 1.5 trillion tokens of knowledge-intensive and safety-enhanced corpora enriched with world context.
- Annealing: 200 billion tokens focused on long-context (up to 32k) and extremely high-quality data.
The architecture is a decoder-only Transformer utilizing Grouped Query Attention (GQA), SwiGLU activation, and RMSNorm. Notably, the researchers found that removing QK-Norm improved performance in their specific setup.
Following pre-training, the model undergoes alignment using a Hierarchical Principle System (HPS), a four-level SFT system covering Safety, Foundational Tasks, Advanced Tasks, and Persona. A key feature is Safe CoT, which trains the model to maintain safety throughout its internal "Chain-of-Thought" reasoning steps. Finally, Reinforcement Learning with Group Relative Policy Optimization (GRPO) is used to align the model with honesty and knowledge boundaries without the need for a separate critic model.
Results and Discussion
JT-Safe-35B demonstrates exceptional performance across safety and industrial benchmarks:
- Safety: It achieved a 97.32% score on safety benchmarks, rivaling the much larger Qwen3-235B (97.63%) and outperforming DeepSeek-V3.1 (93.98%).
- Industry Performance: On the CCR-Bench for complex industrial scenarios, JT-Safe-35B scored 51.95%, significantly leading competitors like Qwen3-30B (30.60%) and DeepSeek-V3.1 (33.50%).
- Efficiency: These results were achieved with 6.2 trillion tokens, a fraction of the data used by other state-of-the-art models.
Ablation studies confirmed that the inclusion of DWC data during the second stage of pre-training resulted in a lower training loss and better performance uplift during post-training compared to baseline models.
Conclusion
JT-Safe represents a significant step toward intrinsic AI safety. By anchoring the model in world context during pre-training, the researchers have created a model that is not only safer but also more reliable in specialized industrial domains. Future work will focus on further mitigating risks in long-form generation and expanding the model's reasoning capabilities to match specialized logic-focused models.
Summary updated