Strong Empowered and Aligned Weak Mastered Annotation for Weak-to-Strong Generalization

Yongqi Li, Xin Miao, Mayi Xu, Tieyun QianAAAI 2025 · DOI

Introduction

As artificial intelligence capabilities approach and exceed human levels, the "super-alignment" problem becomes a central challenge in AI safety: how can relatively "weak" human supervisors effectively guide and control "strong" super-human AI? Researchers typically study this through Weak-to-Strong Generalization (W2SG), a paradigm where a smaller, aligned "weak" model provides supervision (labels) for a larger "strong" model.

However, current W2SG methods suffer from two major flaws. First, the Knowledge Gap: the weak model often lacks the specific domain knowledge or reasoning depth required to provide high-quality labels for complex tasks. Second, the Risky Corrector: previous methods often allowed the strong model to "correct" the weak model's errors. Because the strong model is unaligned, it might "correct" a safe, helpful response into one that is more informative but violates safety or ethical boundaries—for instance, providing a biased opinion because it appears more helpful.

The SEAM Framework

To resolve these tensions, the authors propose SEAM (Strong Empowered and Aligned Weak Mastered). This framework re-positions the strong model as a knowledge enabler and the weak model as the annotation master. Instead of the strong model overruling the weak one, it provides structured context that the weak model uses to make a more informed, yet still aligned, decision.

The SEAM workflow: From searching-while-thinking to final annotation.

The SEAM process follows three main steps:

  1. Principle Definition: The authors define 11 human-centric principles (e.g., Informative, Law-abiding, No NSFW, Objective) to guide the annotation process.
  2. Principle FF-Tree Generation: The strong model performs a "searching-while-thinking" process. It explores different combinations of these principles to generate "thoughts" about a specific query. It builds a Fast-and-Frugal Tree (FF-Tree) where each node represents a principle-aware thought. The tree is pruned based on Information Gain (measured via entropy reduction), ensuring only the most relevant knowledge is surfaced.
  3. Annotation Decision: The aligned weak model reviews the thoughts generated in the FF-Tree. It identifies the deepest node that provides a significant information gain and uses that specific "thought" to make the final annotation. This ensures the final label is "empowered" by the strong model's knowledge but "mastered" by the weak model's alignment.

Experimental Evaluation

The framework was evaluated on six diverse preference datasets, representing tasks where a model must choose the better of two candidate responses. These included helpfulness-focused (AF, HS), harmlessness-focused (AM, CH), and mixed-objective datasets (AHH, SR).

Key Results

SEAM achieved significant performance gains across all metrics:

  • Annotation Quality: SEAM achieved a 6.5% average improvement in weak annotation accuracy over the best baseline.
  • W2SG Performance: In the weak-to-strong fine-tuning phase, SEAM showed a 13% increase in Performance Gap Recovered (PGR) compared to previous state-of-the-art methods.
  • Safety Superiority: On harmlessness datasets (AM and CH), where traditional "strong-corrector" methods often failed, SEAM significantly outperformed baselines, proving it is much safer for alignment tasks.
  • Alignment Success: When used to align a strong model via Direct Preference Optimization (DPO), the SEAM-supervised model achieved a 73.5% win rate against a standard SFT model. This performance is comparable to, and in some cases slightly exceeds, the "strong ceiling" (72.7%) achieved by training on ground-truth human labels.
Win rates of SEAM-supervised models compared to baselines.

Conclusion

The core "aha moment" of SEAM is that alignment is better preserved when the weak model uses the strong model's knowledge rather than being overruled by it. By using entropy to calculate information gain at each node of the search tree, the system quantitatively determines when "thinking" clarifies the decision versus when it adds noise. While generating search trees is more computationally expensive than simple inference, SEAM provides a robust and safe path for supervising super-human AI without sacrificing human-centric values.

Summary updated

BibTeX
@inproceedings{li2025strong,
  author = {Yongqi Li and Xin Miao and Mayi Xu and Tieyun Qian},
  title = {{Strong Empowered and Aligned Weak Mastered Annotation for Weak-to-Strong Generalization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i26.34955},
}
Figure