JT-Safe: Intrinsically Enhancing the Safety and Trustworthiness of LLMs

Junlan Feng, Fanyu Meng, Chong Long, Pengyu Cong, Duqing Wang, Yan Zheng, Yuyao Zhang, Xuanchang Gao, Ye Yuan, Yunfei Ma, Zhijie Ren, Fan Yang, Na Wu, Di Jin, Chao DengarXiv 2025 · DOI · OpenReview

Introduction

JT-Safe is a 35.4B parameter Large Language Model (LLM) designed to address the fundamental limitations of current AI safety paradigms. While most safety efforts focus on post-training alignment, JT-Safe integrates safety and trustworthiness directly into the pre-training phase. By utilizing "Data with World Context" (DWC)—which includes metadata such as source, time, and safety ratings—the model is anchored in real-world knowledge, significantly reducing hallucinations and improving safety without compromising general performance.

The Limitations of Post-Training Safety

Current LLM safety strategies predominantly rely on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). These methods act as peripheral controls rather than addressing the root cause of safety issues: the pre-training data itself. Standard pre-training treats data as a flat sequence of tokens, lacking grounding in the real world. This leads to "cognitive uncertainty," where a model cannot distinguish between a reliable fact and a biased or incorrect statement seen in a low-quality context. Furthermore, aggressive post-training alignment can lead to "performance reverse scaling," where a model's general capabilities degrade as it becomes overly cautious.

Methodology: Data with World Context (DWC)

The core innovation of JT-Safe is the Data with World Context (DWC) framework. Instead of training on raw text, the model processes text paired with rich metadata, including:

  • Basic Attributes: Time and location of the data.
  • Categories: Domain-specific labels (e.g., Physics, Politics).
  • Source Information: Distinguishing between authoritative journals and personal blogs.
  • Safety-related Attributes: Explicit safety ratings and context.

This metadata acts as a semantic routing mechanism, allowing the model to activate specific parameter subspaces based on the context of the input.

Overview of the JT-Safe framework, including DWC data samples and performance impact.

Training Pipeline and Architecture

JT-Safe follows a rigorous three-stage pre-training pipeline totaling 6.2 trillion tokens:

  1. General Pre-training: 4.5 trillion tokens of high-quality general data.
  2. DWC Enhancement: 1.5 trillion tokens of knowledge-intensive and safety-enhanced corpora enriched with world context.
  3. Annealing: 200 billion tokens focused on long-context (up to 32k) and extremely high-quality data.

The architecture is a decoder-only Transformer utilizing Grouped Query Attention (GQA), SwiGLU activation, and RMSNorm. Notably, the researchers found that removing QK-Norm improved performance in their specific setup.

Following pre-training, the model undergoes alignment using a Hierarchical Principle System (HPS), a four-level SFT system covering Safety, Foundational Tasks, Advanced Tasks, and Persona. A key feature is Safe CoT, which trains the model to maintain safety throughout its internal "Chain-of-Thought" reasoning steps. Finally, Reinforcement Learning with Group Relative Policy Optimization (GRPO) is used to align the model with honesty and knowledge boundaries without the need for a separate critic model.

Results and Discussion

JT-Safe-35B demonstrates exceptional performance across safety and industrial benchmarks:

  • Safety: It achieved a 97.32% score on safety benchmarks, rivaling the much larger Qwen3-235B (97.63%) and outperforming DeepSeek-V3.1 (93.98%).
  • Industry Performance: On the CCR-Bench for complex industrial scenarios, JT-Safe-35B scored 51.95%, significantly leading competitors like Qwen3-30B (30.60%) and DeepSeek-V3.1 (33.50%).
  • Efficiency: These results were achieved with 6.2 trillion tokens, a fraction of the data used by other state-of-the-art models.

Ablation studies confirmed that the inclusion of DWC data during the second stage of pre-training resulted in a lower training loss and better performance uplift during post-training compared to baseline models.

Conclusion

JT-Safe represents a significant step toward intrinsic AI safety. By anchoring the model in world context during pre-training, the researchers have created a model that is not only safer but also more reliable in specialized industrial domains. Future work will focus on further mitigating risks in long-form generation and expanding the model's reasoning capabilities to match specialized logic-focused models.

Summary updated

Abstract

The hallucination and credibility concerns of large language models (LLMs) are global challenges that the industry is collectively addressing. Recently, a significant amount of advances have been made on post-training and inference techniques to mitigate these challenges. However, it is widely agreed that unsafe and hallucinations of LLMs intrinsically originate from pre-training, involving pre-training data and the next-token prediction learning mechanism. In this paper, we focus on enhancing pre-training data to improve the trustworthiness and safety of LLMs. Since the data is vast, it's almost impossible to entirely purge the data of factual errors, logical inconsistencies, or distributional biases. Moreover, the pre-training data lack grounding in real-world knowledge. Each piece of data is treated as a sequence of tokens rather than as a representation of a part of the world. To overcome these issues, we propose approaches to enhancing our pre-training data with its context in the world and increasing a substantial amount of data reflecting industrial scenarios. We argue that most source data are created by the authors for specific purposes in a certain spatial-temporal context. They have played a role in the real world. By incorporating related world context information, we aim to better anchor pre-training data within real-world scenarios, thereby reducing uncertainty in model training and enhancing the model's safety and trustworthiness. We refer to our Data with World Context as DWC. We continue pre-training an earlier checkpoint of JT-35B-Base with 1.5 trillion of DWC tokens. We introduce our post-training procedures to activate the potentials of DWC. Compared with the Qwen model of a similar scale, JT-Safe-35B achieves an average performance improvement of 1.79% on the Safety and Trustworthy evaluation benchmarks, while being pretrained with only 6.2 trillion tokens.
BibTeX
@misc{feng2025jt,
  author = {Junlan Feng and Fanyu Meng and Chong Long and Pengyu Cong and Duqing Wang and Yan Zheng and Yuyao Zhang and Xuanchang Gao and Ye Yuan and Yunfei Ma and Zhijie Ren and Fan Yang and Na Wu and Di Jin and Chao Deng},
  title = {{JT-Safe: Intrinsically Enhancing the Safety and Trustworthiness of LLMs}},
  year = {2025},
  eprint = {2510.17918},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.17918},
}
Figure