JT-SAFE-V2: Safety-by-Design Foundation Model with World-Context Data

Junlan Feng, Fanyu Meng, Chong Long, Pengyu Cong, Duqing Wang, Yan Zheng, Yuyao Zhang, Xuanchang Gao, Ye Yuan, Yunfei Ma, Zhijie Ren, Fan Yang, Na Wu, Di Jin, Chao DengarXiv 2026 · arXiv

TL;DR

JT-Safe-V2 introduces a "safety-by-design" paradigm for large language models by embedding a three-layer "World Context" directly into pre-training data and utilizing a deterministic, high-certainty training procedure. Alongside this safety-enhanced 35B foundation model, the authors present Safe-MoMA, an intelligent multi-agent orchestration framework that slashes enterprise inference costs by over 30% while maintaining top-tier performance.

What Problem Does This Paper Address?

Traditional LLM safety strategies operate like an external guardrail system applied after a model is fully trained (e.g., via RLHF or safety classifiers). However, many reliability issues—such as hallucinations, factual contradictions, and vulnerability to adversarial prompts—originate much earlier in the pipeline.

Standard web-scale pre-training corpora consist of "flattened" text that lacks temporal context, author intent, or causal structures. When models train purely on next-token prediction over this ambiguous data, they internalize uncertainty. JT-Safe-V2 addresses this by shifting safety from a reactive post-training patch to an intrinsic design objective embedded across the entire model lifecycle: data construction, pre-training, and alignment.

Methods

The authors introduce several core innovations to realize their safety-by-design paradigm:

  • Data with World Context (DWC): Rather than treating text as isolated token sequences, the DWC framework enriches data with a structured, three-layer annotation architecture:
  • Factual Layer: Anchors text with explicit spatiotemporal metadata, domain classifications, and source credibility indicators.
  • Logical Layer: Exposes underlying reasoning structures, including causal relations, conditional dependencies, and chain-of-thought traces.
  • Cognitive Layer: Captures the informational intent, reading purpose, and value perspectives.

    Figure 1: The three-layer annotation architecture of DWC.
  • High-Certainty Pre-training (WSM Paradigm): Traditional cosine learning rate decay schedules can cause optimization instability. The Warmup-Stable-Merge (WSM) paradigm decouples parameter optimization from learning rate annealing. The learning rate is held at a high constant value for broad parameter space exploration, and checkpoints are merged offline using diversified weight distribution strategies. This prevents catastrophic forgetting during continual pre-training.

    Figure 1: The three-layer annotation architecture of DWC.
  • Prefix-Guided Meta-Information Activation: To prevent the model from failing to elicit its internal safety knowledge when given concise user queries, the authors use post-training fine-tuning to map quality and safety constraints to specific system prompt prefixes. These prefixes act as triggers to awaken latent DWC knowledge during inference.

  • Safe-MoMA (Safe Mixture of Models and Agents): A task-driven orchestration framework formulated as a Markov Decision Process (MDP) and optimized via reinforcement learning. It features a high-level routing policy that selects the best solving paradigm (single model, single agent, or multi-agent) and a low-level policy that dynamically allocates specific models and tools based on task complexity and resource constraints.

Datasets

  • Pre-training & Alignment: The model utilizes the newly developed DWC corpus and a seed cold-start dataset of approximately $10^5$ high-quality instruction samples for model-driven self-distillation.
  • Safety Benchmarks: Evaluated across 20 benchmarks spanning six categories, including CNsafe, CValues, Flames, Sweeval, Airbench, AgentHarm, SafetyPrompts, JADE, SimpleSafetyTest, Forbidden, DoNotAnswer, TechHazardQA, SaladBench, BBQ, JailbreakBench, StrongReject, Jbdistill, CSSbench, ChineseSafetyQA, TruthfulQA, and RagTruth.
  • General Capabilities Benchmarks: Evaluated using standard benchmarks including HumanEval(+), MBPP+, CruxEval, MultiPL-E (coding); GSM8K, MATH500, MGSM-zh, MathBench (math); MRCR, RULER, Leval (long context); PIQA, HellaSwag, StrategyQA, AutoLogi (reasoning); CryptoBench, FinEval, Xiezhi, GaokaoBench (knowledge); and BFCL, $\tau^2$-Bench, SEAL, GAIA-text (agentic capabilities).

Key Insights and Ideas

  • Safety Enhances Capability: A common assumption is that strict safety alignment degrades an LLM's general capabilities. JT-Safe-V2 proves the opposite: structuring pre-training data with factual, logical, and cognitive context layers actually boosts performance in downstream reasoning, knowledge retrieval, and long-context tasks.
  • Plug-and-Play Meta-Information: The DWC meta-information exhibits strong plug-and-play compatibility. It can be injected into an already well-trained checkpoint during continued pre-training without requiring architectural changes or re-initialization, significantly accelerating data efficiency.
  • Dynamic Orchestration Over Monolithic Scaling: Deploying a single massive model for every enterprise task is highly inefficient. By framing multi-model routing as a hierarchical reinforcement learning problem, systems can achieve state-of-the-art performance by dynamically combining lightweight models, specialized agents, and heavy reasoning models only when necessary.

Key Results

  • Safety Dominance: JT-Safe-V2-35B achieved a near-perfect score of 100.00 on CValues, 99.66 on CNsafe, and 99.79 on StrongReject, consistently outperforming or matching much larger models like Qwen3-235B and DeepSeek-V3.2.
  • General Capabilities: The safety-first training did not degrade performance; the model achieved 96.30 on HumanEval (compared to the 94.50 baseline SOTA) and dramatically improved on GaokaoBench (92.28 vs. 27.76) and MathBench (94.87 vs. 74.26).
  • Safe-MoMA Efficiency: Compared to using the optimal standalone large model (Qwen3-235B-A22B), the Safe-MoMA routing framework achieved a 2.9% score improvement while reducing inference costs by 31.46% under performance-priority settings, and up to 42% cost reduction overall.

Limitations and Future Work

  • Agentic Capabilities: While JT-Safe-V2 showed strong planning improvements on $\tau^2$-Bench, it lagged behind on benchmarks like SEAL and GAIA-text. The authors explicitly call out agentic capability as a key area for future improvement.
  • Routing in Unconstrained Categories: The SFT-based approach for task routing performs exceptionally well under limited, well-defined data categories, but its performance tends to degrade when applied to highly diverse, unconstrained real-world enterprise scenarios.

Summary updated

Abstract

We introduce JT-Safe-V2, a large language model designed to advance the safety and trustworthiness of foundation models, extending our previous JT-Safe model toward a more comprehensive safety-by-design paradigm. JT-Safe-V2 emphasizes the joint optimization of general intelligence and safety-by-design through several key innovations: enriching pre-training data with contextual world knowledge, high-certainty pre-training procedures, and safety strengthening post-training mechanisms for enterprise-oriented agentic capabilities. Building on these safety-enhanced foundation models, we propose Safe-MoMA (Safe Mixture of Models and Agents), a framework that enables traceable and efficient inference through the orchestrated deployment of multiple models and agents. Extensive evaluations demonstrate that JT-Safe-V2 achieves state-of-the-art performance across both general intelligence and safety benchmarks. Moreover, Safe-MoMA reduces inference costs by more than 30\% compared to using the largest standalone model baseline while maintaining comparable performance. To facilitate future research on safety-by-design foundation models, we publicly release the post-trained JT-Safe-V2-35B model checkpoint.
BibTeX
@misc{feng2026jt,
  author = {Junlan Feng and Fanyu Meng and Chong Long and Pengyu Cong and Duqing Wang and Yan Zheng and Yuyao Zhang and Xuanchang Gao and Ye Yuan and Yunfei Ma and Zhijie Ren and Fan Yang and Na Wu and Di Jin and Chao Deng},
  title = {{JT-SAFE-V2: Safety-by-Design Foundation Model with World-Context Data}},
  year = {2026},
  eprint = {2605.24414},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.24414},
}
Figure