Oyster-I: Beyond Refusal - Constructive Safety Alignment for Responsible Language Models

Ranjie Duan, Jiexi Liu, Xiaojun Jia, Shiji Zhao, Ruoxi Cheng, Fengxiang Wang, Cheng Wei, Yong Xie, Chang Liu, Defeng Li, Yinpeng Dong, Yichi Zhang, Yuefeng Chen, Chongwen Wang, Xingjun Ma, Xingxing Wei, Yang Liu, Hang Su, Jun Zhu, Xinfeng Li, Yitong Sun, Jie Zhang, Jinzhao Hu, Sha Xu, Wenchao Yang, Yitong Yang, Xingyao Zhang, Yingshui Tan, Jialing Tao, Hui XuearXiv 2025 · DOI · arXiv

Introduction

Current Large Language Model (LLM) safety training often relies on a binary "refusal" mechanism. When a query touches a sensitive topic, models typically issue a hard refusal, such as "I cannot assist with that." While this protects against malicious actors, it often fails non-malicious users—like a distressed parent seeking urgent medical advice or a researcher investigating dual-use chemicals—by providing no helpful path forward and potentially driving them toward less-regulated platforms.

Alibaba AAIG introduces Oyster-I, a model designed to move beyond this "refusal-default" mindset toward Constructive Safety Alignment (CSA). The goal is to find the "pearl point": a response strategy that maintains rigorous safety boundaries while remaining genuinely helpful and guiding the user toward beneficial outcomes.

The Constructive Safety Alignment Paradigm

The researchers propose CSA as a way to frame safety as a dynamic, game-theoretic interaction rather than a static label. Unlike previous methods like Constitutional AI or Deliberative Alignment, CSA focuses on safety guidelines tailored to the specific query and user intent.

Comparison of safety alignment methods: Constitutional AI, Deliberative Alignment, and the proposed Constructive Alignment (Ours).

CSA rests on three technical pillars:

  1. Game-Theoretic Modeling: Interaction is framed as a Stackelberg game where the model (the leader) anticipates potential user responses (the follower) to steer the dialogue toward a safe, constructive equilibrium.
  2. Structured Reasoning: Instead of a "black-box" chain-of-thought, the model follows an explicit reasoning graph, analyzing user intent, risk category, and safety guidelines before formulating a strategy.
  3. Linguistic Backpropagation (Lingo-BP): This novel optimization method uses linguistic feedback from specialized evaluators to iteratively refine reasoning strategies. By treating high-level strategies as modular policies, the system can "backpropagate" errors through the reasoning chain to fix specific logic flaws.

Structured Reasoning in Action

Oyster-I differentiates between various levels of risk, from safe inquiries to malicious instructions. Instead of a generic refusal, it employs a structured "Thinking" process to determine the best course of action for the specific context.

Oyster-I's structured reasoning and response strategies across different risk levels, from safe inquiries to adversarial attacks.

As shown in the figure above, for an "Ethical Risk" (e.g., a distressed parent), the model identifies the user's exhaustion and lack of knowledge, refuses dangerous medical operations, but expresses empathy and directs the user to professional pediatric guidance. For "Adversarial Risk," it identifies the malicious intent and cuts off the attacker's expected benefits while maintaining a firm refusal.

Key Results and Insights

Oyster-I (Oy1) demonstrates state-of-the-art performance among open-source models:

  • Constructive Superiority: On a new Constructive Benchmark, Oyster-I achieved a score of 0.5627, surpassing all open-source models and approaching commercial leaders like GPT-5 (0.6075).
  • Robust Defense: On the Strata-Sword jailbreak dataset, it scored 92.54, closely matching GPT-o1 (95.84) and significantly outperforming its base models.
  • Safety-Capability Synergy: Interestingly, the authors found that safety alignment actually improved performance in mathematical and coding tasks, suggesting that the logical rigor required for safety can reduce search space entropy in technical domains.

Conclusion and Future Work

Oyster-I represents a shift from "just saying no" to providing constructive guidance. While the model still occasionally over-refuses in ambiguous scenarios, it sets a new standard for balancing safety with helpfulness. Future work will focus on developing a Constructive Safety Reward Model (CSRM) for real-time feedback and expanding the framework to multi-turn dialogues and multimodal agents.

Summary updated

Abstract

Large language models (LLMs) typically deploy safety mechanisms to prevent harmful content generation. Most current approaches focus narrowly on risks posed by malicious actors, often framing risks as adversarial events and relying on defensive refusals. However, in real-world settings, risks also come from non-malicious users seeking help while under psychological distress (e.g., self-harm intentions). In such cases, the model's response can strongly influence the user's next actions. Simple refusals may lead them to repeat, escalate, or move to unsafe platforms, creating worse outcomes. We introduce Constructive Safety Alignment (CSA), a human-centric paradigm that protects against malicious misuse while actively guiding vulnerable users toward safe and helpful results. Implemented in Oyster-I (Oy1), CSA combines game-theoretic anticipation of user reactions, fine-grained risk boundary discovery, and interpretable reasoning control, turning safety into a trust-building process. Oy1 achieves state-of-the-art safety among open models while retaining high general capabilities. On our Constructive Benchmark, it shows strong constructive engagement, close to GPT-5, and unmatched robustness on the Strata-Sword jailbreak dataset, nearing GPT-o1 levels. By shifting from refusal-first to guidance-first safety, CSA redefines the model-user relationship, aiming for systems that are not just safe, but meaningfully helpful. We release Oy1, code, and the benchmark to support responsible, user-centered AI.
BibTeX
@misc{duan2025oyster,
  author = {Ranjie Duan and Jiexi Liu and Xiaojun Jia and Shiji Zhao and Ruoxi Cheng and Fengxiang Wang and Cheng Wei and Yong Xie and Chang Liu and Defeng Li and Yinpeng Dong and Yichi Zhang and Yuefeng Chen and Chongwen Wang and Xingjun Ma and Xingxing Wei and Yang Liu and Hang Su and Jun Zhu and Xinfeng Li and Yitong Sun and Jie Zhang and Jinzhao Hu and Sha Xu and Wenchao Yang and Yitong Yang and Xingyao Zhang and Yingshui Tan and Jialing Tao and Hui Xue},
  title = {{Oyster-I: Beyond Refusal - Constructive Safety Alignment for Responsible Language Models}},
  year = {2025},
  eprint = {2509.01909},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.01909},
}
Figure