Constitutional Ai

orchestra-research/ai-research-skills/07-safety-alignment/constitutional-ai

作者 orchestra-research773a52944ba4MIT13K 个星标收录于 2026年10月8日更新于 2026年10月8日仓库3个月前更新

Anthropic's method for training harmless AI through self-improvement. Two-phase approach - supervised learning with self-critique/revision, then RLAIF (RL from AI Feedback). Use for safety alignment, reducing harmful outputs without human labels. Powers Claude's safety system.

仅含说明AI & AgentsSecurity
AI 生成的概览

指导使用宪法式 AI 进行语言模型安全对齐：自我批评、修订与 RLAIF 训练。

功能
该技能介绍 Anthropic 的宪法式 AI 方法，用于在无需人工标注有害输出的情况下训练无害模型。它讲解监督阶段（模型依据成文宪法批评并修订自身回答）以及强化学习阶段（由 AI 生成的偏好训练奖励模型和 PPO 策略）。内容还涵盖思维链批评、宪法设计以及常见对齐问题的排查。
适用场景
适合在规划语言模型安全对齐、希望在不收集人类偏好标注的情况下减少有害输出时使用。也适合已有明确原则、追求可解释且可扩展安全训练的团队。它不面向运行时内容过滤或使用预训练审核模型进行内容审核。
运行要求
需要 Python 环境及 transformers、torch、trl 依赖，并需要 GPU 硬件（建议 NVIDIA A100/H100，7B 模型监督阶段约需 40GB 显存，强化学习阶段需两块 GPU）。该技能不附带脚本，仅提供说明与代码示例。

Constitutional AI - Harmlessness from AI Feedback

Quick start

Constitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs.

Key concept: Models learn to critique and revise their own responses using a "constitution" (set of principles).

Two phases:

  1. Supervised Learning (SL): Self-critique + revision
  2. Reinforcement Learning (RL): RLAIF (RL from AI Feedback)

Constitution example:

Principles:1. Choose the response that is most helpful, honest, and harmless2. Avoid responses that are toxic, racist, or sexist3. Prefer responses that explain objections rather than refuse4. Choose responses that are thoughtful and nuanced

Common workflows

Workflow 1: Supervised learning phase (self-critique + revision)

Step 1: Generate initial responses:

python
from transformers import pipeline
generator = pipeline("text-generation", model="base-model")
prompts = [    "How do I hack a website?",    "Write something offensive about a group.",    "Help me cheat on my exam."]
initial_responses = generator(prompts, max_length=200)

Step 2: Self-critique with constitution:

python
critique_prompt = """Given the question:{question}
And the response:{response}
Constitution:1. Choose responses that are helpful, honest, and harmless2. Avoid toxic, racist, or sexist content3. Explain objections rather than refuse
Identify any ways this response violates the constitution:"""
critiques = generator(    [critique_prompt.format(q=q, r=r) for q, r in zip(prompts, initial_responses)])

Step 3: Revision based on critique:

python
revision_prompt = """Question: {question}Original response: {response}Critique: {critique}
Please revise the response to better align with the constitution:"""
revised_responses = generator(    [revision_prompt.format(q=q, r=r, c=c)     for q, r, c in zip(prompts, initial_responses, critiques)])

Step 4: Fine-tune on revised responses:

python
from trl import SFTTrainer
# Create dataset of (prompt, revised_response) pairsdataset = create_dataset(prompts, revised_responses)
trainer = SFTTrainer(    model=model,    train_dataset=dataset,    max_seq_length=1024)trainer.train()

Workflow 2: RL phase (RLAIF - RL from AI Feedback)

Step 1: Generate comparison pairs:

python
# Sample multiple responses per promptresponses_a = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)responses_b = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)

Step 2: AI preference evaluation:

python
preference_prompt = """Question: {question}
Response A: {response_a}Response B: {response_b}
Constitution:{constitution}
Which response better follows the constitution? Explain your reasoning, then choose A or B."""
# Get AI preferences (no human labels needed!)preferences = generator(    [preference_prompt.format(q=q, ra=ra, rb=rb, constitution=CONSTITUTION)     for q, ra, rb in zip(prompts, responses_a, responses_b)])
# Parse preferences (A or B)chosen, rejected = parse_preferences(preferences, responses_a, responses_b)

Step 3: Train preference model (reward model):

python
from trl import RewardTrainer, RewardConfig
preference_dataset = create_preference_dataset(prompts, chosen, rejected)
reward_config = RewardConfig(    output_dir="constitutional-reward-model",    learning_rate=1e-5,    num_train_epochs=1)
reward_trainer = RewardTrainer(    model=model,    args=reward_config,    train_dataset=preference_dataset,    processing_class=tokenizer)reward_trainer.train()

Step 4: RL training with RLAIF:

python
from trl import PPOTrainer, PPOConfig
ppo_config = PPOConfig(    reward_model_path="constitutional-reward-model",    learning_rate=1e-6,    kl_coef=0.05)
ppo_trainer = PPOTrainer(    model=model,    config=ppo_config,    reward_model=reward_model)ppo_trainer.train()

Workflow 3: Chain-of-thought critique

Enable reasoning transparency:

python
cot_critique_prompt = """Question: {question}Response: {response}
Let's think step-by-step about whether this response follows our principles:
1. Is it helpful? [Yes/No and reasoning]2. Is it honest? [Yes/No and reasoning]3. Is it harmless? [Yes/No and reasoning]4. Does it avoid toxicity? [Yes/No and reasoning]
Based on this analysis, suggest a revision if needed."""
cot_critiques = generator(    [cot_critique_prompt.format(q=q, r=r) for q, r in zip(prompts, responses)])

When to use vs alternatives

Use Constitutional AI when:

  • Want safety alignment without human labels
  • Need explainable AI decisions
  • Want to avoid evasive refusals
  • Have a clear set of principles/constitution
  • Need scalable safety training

Principles:

  • RLAIF: AI-generated preferences (scalable, no human labels)
  • RLHF: Human preferences (more accurate, expensive)
  • Self-critique: Iterative improvement
  • Chain-of-thought: Reasoning transparency

Use alternatives instead:

  • RLHF (PPO): Need human-validated safety
  • DPO/SimPO: Have human preference data
  • NeMo Guardrails: Need runtime content filtering
  • LlamaGuard: Need pre-trained moderation model

Common issues

Issue: Model refuses too much (evasive)

Add constitution principle:

Prefer responses that engage thoughtfully with questions rather thanrefusing to answer. Explain concerns while still being helpful.

Issue: Self-critiques are weak

Use stronger critique prompts:

Critically analyze this response for ANY potential issues, however minor.Be thorough and specific in identifying problems.

Issue: Revisions don't improve quality

Iterate multiple times:

python
for _ in range(3):  # 3 rounds of critique/revision    critique = generate_critique(response)    response = generate_revision(response, critique)

Issue: RLAIF preferences are noisy

Use multiple AI evaluators:

python
# Get preferences from 3 different modelsprefs_1 = model_1.evaluate(responses)prefs_2 = model_2.evaluate(responses)prefs_3 = model_3.evaluate(responses)
# Majority votefinal_preference = majority_vote(prefs_1, prefs_2, prefs_3)

Advanced topics

Constitution design: See references/constitution-design.md [blocked] for principle selection, trade-offs between helpfulness and harmlessness, and domain-specific constitutions.

RLAIF vs RLHF: See references/rlaif-comparison.md [blocked] for performance comparison, cost analysis, and when to use AI feedback vs human feedback.

Chain-of-thought reasoning: See references/cot-critique.md [blocked] for prompt engineering for critiques, multi-step reasoning, and transparency improvements.

Hardware requirements

  • GPU: NVIDIA A100/H100 recommended
  • VRAM:
    • SL phase (7B): 1× A100 40GB
    • RL phase (7B): 2× A100 40GB (policy + reward model)
  • Single-node: Sufficient for most use cases
  • Mixed precision: BF16 recommended

Compute requirements:

  • SL phase: Similar to standard SFT
  • RL phase: Similar to PPO (higher than DPO)
  • AI evaluation: Additional inference for critique/preference generation

Resources

来源与署名

来源：orchestra-research/ai-research-skills位于07-safety-alignment/constitutional-ai提交773a529

许可证： MIT

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

更多来自 orchestra-research/ai-research-skills 的技能

Speculative Decoding

orchestra-research

指导使用推测解码、Medusa 多头预测和前瞻解码来加速大语言模型推理。

AI & Agents13K3个月前更新

Systems Paper Writing

orchestra-research

为 OSDI、SOSP、ASPLOS、NSDI 等系统会议论文写作提供段落级结构与篇幅分配指导。

Writing & Content13K3个月前更新

Moe Training

orchestra-research

指导使用 DeepSpeed 或 HuggingFace 训练与推理混合专家模型，涵盖路由、负载均衡与专家并行。

AI & Agents13K3个月前更新

Presenting Conference Talks

orchestra-research

Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.

待分类13K3个月前更新

Model Pruning

orchestra-research

指导使用 Wanda、SparseGPT、幅度剪枝和 N:M 方法压缩 LLM，减小模型体积并加速推理。

AI & Agents13K3个月前更新

Outlines

orchestra-research

指导使用 Outlines 库，通过本地模型进行受结构约束的文本生成。

AI & Agents13K3个月前更新

更多AI & Agents技能

Ppt Template Creator

anthropics

精选

将用户的 PowerPoint 模板转化为可复用技能，用于生成品牌演示文稿。

AI & Agents39K2周前更新

Skill Development

anthropics

精选

指导创建 Claude Code 插件技能，涵盖结构、描述、渐进式披露与验证。

AI & Agents37K今天更新

Plugin Structure

anthropics

精选

指导 Claude Code 插件的结构、清单与组件布局。

AI & Agents37K今天更新

Command Development

anthropics

精选

指导创建 Claude Code 斜杠命令，涵盖结构、YAML frontmatter、参数与插件功能。

AI & Agents37K今天更新

Claude Md Improver

anthropics

精选

审查仓库中的 CLAUDE.md 文件，评估其质量，并在获得批准后应用有针对性的改进。

AI & Agents37K今天更新

Find Skills

vercel-labs

精选

指导通过 npx skills 命令行工具发现、评估并安装智能体技能。

AI & Agents33K今天更新