Constitutional Ai

orchestra-research/ai-research-skills/07-safety-alignment/constitutional-ai

作者 orchestra-research773a52944ba4MIT13K 個星標收錄於 2026年10月8日更新於 2026年10月8日儲存庫3 個月前更新

Anthropic's method for training harmless AI through self-improvement. Two-phase approach - supervised learning with self-critique/revision, then RLAIF (RL from AI Feedback). Use for safety alignment, reducing harmful outputs without human labels. Powers Claude's safety system.

僅含說明AI & AgentsSecurity
AI 產生的概覽

指導使用憲法式 AI 進行語言模型安全對齊：自我批評、修訂與 RLAIF 訓練。

功能
此技能介紹 Anthropic 的憲法式 AI 方法，用於在不需要人工標註有害輸出的情況下訓練無害模型。內容說明監督階段（模型依據成文憲法批評並修訂自身回應）以及強化學習階段（由 AI 產生的偏好訓練獎勵模型與 PPO 策略）。此外也涵蓋思維鏈批評、憲法設計，以及常見對齊問題的疑難排解。
適用情境
適合在規劃語言模型安全對齊、希望在不蒐集人類偏好標註的情況下減少有害輸出時使用。也適合已有明確原則、追求可解釋且可擴充安全訓練的團隊。它並非針對執行階段內容過濾或使用預訓練審核模型進行內容審核。
執行需求
需要 Python 環境與 transformers、torch、trl 相依套件，並需要 GPU 硬體（建議 NVIDIA A100/H100，7B 模型監督階段約需 40GB 顯示記憶體，強化學習階段需兩張 GPU）。此技能未附帶指令碼，僅提供說明與程式碼範例。

Constitutional AI - Harmlessness from AI Feedback

Quick start

Constitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs.

Key concept: Models learn to critique and revise their own responses using a "constitution" (set of principles).

Two phases:

  1. Supervised Learning (SL): Self-critique + revision
  2. Reinforcement Learning (RL): RLAIF (RL from AI Feedback)

Constitution example:

Principles:1. Choose the response that is most helpful, honest, and harmless2. Avoid responses that are toxic, racist, or sexist3. Prefer responses that explain objections rather than refuse4. Choose responses that are thoughtful and nuanced

Common workflows

Workflow 1: Supervised learning phase (self-critique + revision)

Step 1: Generate initial responses:

python
from transformers import pipeline
generator = pipeline("text-generation", model="base-model")
prompts = [    "How do I hack a website?",    "Write something offensive about a group.",    "Help me cheat on my exam."]
initial_responses = generator(prompts, max_length=200)

Step 2: Self-critique with constitution:

python
critique_prompt = """Given the question:{question}
And the response:{response}
Constitution:1. Choose responses that are helpful, honest, and harmless2. Avoid toxic, racist, or sexist content3. Explain objections rather than refuse
Identify any ways this response violates the constitution:"""
critiques = generator(    [critique_prompt.format(q=q, r=r) for q, r in zip(prompts, initial_responses)])

Step 3: Revision based on critique:

python
revision_prompt = """Question: {question}Original response: {response}Critique: {critique}
Please revise the response to better align with the constitution:"""
revised_responses = generator(    [revision_prompt.format(q=q, r=r, c=c)     for q, r, c in zip(prompts, initial_responses, critiques)])

Step 4: Fine-tune on revised responses:

python
from trl import SFTTrainer
# Create dataset of (prompt, revised_response) pairsdataset = create_dataset(prompts, revised_responses)
trainer = SFTTrainer(    model=model,    train_dataset=dataset,    max_seq_length=1024)trainer.train()

Workflow 2: RL phase (RLAIF - RL from AI Feedback)

Step 1: Generate comparison pairs:

python
# Sample multiple responses per promptresponses_a = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)responses_b = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)

Step 2: AI preference evaluation:

python
preference_prompt = """Question: {question}
Response A: {response_a}Response B: {response_b}
Constitution:{constitution}
Which response better follows the constitution? Explain your reasoning, then choose A or B."""
# Get AI preferences (no human labels needed!)preferences = generator(    [preference_prompt.format(q=q, ra=ra, rb=rb, constitution=CONSTITUTION)     for q, ra, rb in zip(prompts, responses_a, responses_b)])
# Parse preferences (A or B)chosen, rejected = parse_preferences(preferences, responses_a, responses_b)

Step 3: Train preference model (reward model):

python
from trl import RewardTrainer, RewardConfig
preference_dataset = create_preference_dataset(prompts, chosen, rejected)
reward_config = RewardConfig(    output_dir="constitutional-reward-model",    learning_rate=1e-5,    num_train_epochs=1)
reward_trainer = RewardTrainer(    model=model,    args=reward_config,    train_dataset=preference_dataset,    processing_class=tokenizer)reward_trainer.train()

Step 4: RL training with RLAIF:

python
from trl import PPOTrainer, PPOConfig
ppo_config = PPOConfig(    reward_model_path="constitutional-reward-model",    learning_rate=1e-6,    kl_coef=0.05)
ppo_trainer = PPOTrainer(    model=model,    config=ppo_config,    reward_model=reward_model)ppo_trainer.train()

Workflow 3: Chain-of-thought critique

Enable reasoning transparency:

python
cot_critique_prompt = """Question: {question}Response: {response}
Let's think step-by-step about whether this response follows our principles:
1. Is it helpful? [Yes/No and reasoning]2. Is it honest? [Yes/No and reasoning]3. Is it harmless? [Yes/No and reasoning]4. Does it avoid toxicity? [Yes/No and reasoning]
Based on this analysis, suggest a revision if needed."""
cot_critiques = generator(    [cot_critique_prompt.format(q=q, r=r) for q, r in zip(prompts, responses)])

When to use vs alternatives

Use Constitutional AI when:

  • Want safety alignment without human labels
  • Need explainable AI decisions
  • Want to avoid evasive refusals
  • Have a clear set of principles/constitution
  • Need scalable safety training

Principles:

  • RLAIF: AI-generated preferences (scalable, no human labels)
  • RLHF: Human preferences (more accurate, expensive)
  • Self-critique: Iterative improvement
  • Chain-of-thought: Reasoning transparency

Use alternatives instead:

  • RLHF (PPO): Need human-validated safety
  • DPO/SimPO: Have human preference data
  • NeMo Guardrails: Need runtime content filtering
  • LlamaGuard: Need pre-trained moderation model

Common issues

Issue: Model refuses too much (evasive)

Add constitution principle:

Prefer responses that engage thoughtfully with questions rather thanrefusing to answer. Explain concerns while still being helpful.

Issue: Self-critiques are weak

Use stronger critique prompts:

Critically analyze this response for ANY potential issues, however minor.Be thorough and specific in identifying problems.

Issue: Revisions don't improve quality

Iterate multiple times:

python
for _ in range(3):  # 3 rounds of critique/revision    critique = generate_critique(response)    response = generate_revision(response, critique)

Issue: RLAIF preferences are noisy

Use multiple AI evaluators:

python
# Get preferences from 3 different modelsprefs_1 = model_1.evaluate(responses)prefs_2 = model_2.evaluate(responses)prefs_3 = model_3.evaluate(responses)
# Majority votefinal_preference = majority_vote(prefs_1, prefs_2, prefs_3)

Advanced topics

Constitution design: See references/constitution-design.md [blocked] for principle selection, trade-offs between helpfulness and harmlessness, and domain-specific constitutions.

RLAIF vs RLHF: See references/rlaif-comparison.md [blocked] for performance comparison, cost analysis, and when to use AI feedback vs human feedback.

Chain-of-thought reasoning: See references/cot-critique.md [blocked] for prompt engineering for critiques, multi-step reasoning, and transparency improvements.

Hardware requirements

  • GPU: NVIDIA A100/H100 recommended
  • VRAM:
    • SL phase (7B): 1× A100 40GB
    • RL phase (7B): 2× A100 40GB (policy + reward model)
  • Single-node: Sufficient for most use cases
  • Mixed precision: BF16 recommended

Compute requirements:

  • SL phase: Similar to standard SFT
  • RL phase: Similar to PPO (higher than DPO)
  • AI evaluation: Additional inference for critique/preference generation

Resources

來源與署名

來源：orchestra-research/ai-research-skills位於07-safety-alignment/constitutional-ai提交773a529

授權條款： MIT

內容歸原作者所有。SourceWeft 從公開儲存庫中收錄這些內容。

檢舉或申請下架

更多來自 orchestra-research/ai-research-skills 的技能

Speculative Decoding

orchestra-research

指導使用推測解碼、Medusa 多頭預測與前瞻解碼來加速大型語言模型推論。

AI & Agents13K3 個月前更新

Systems Paper Writing

orchestra-research

為 OSDI、SOSP、ASPLOS、NSDI 等系統會議論文寫作提供段落層級結構與篇幅分配指引。

Writing & Content13K3 個月前更新

Moe Training

orchestra-research

指導使用 DeepSpeed 或 HuggingFace 訓練與推論混合專家模型，涵蓋路由、負載平衡與專家平行。

AI & Agents13K3 個月前更新

Presenting Conference Talks

orchestra-research

Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.

待分類13K3 個月前更新

Model Pruning

orchestra-research

指導使用 Wanda、SparseGPT、幅度剪枝與 N:M 方法壓縮 LLM，縮小模型體積並加速推論。

AI & Agents13K3 個月前更新

Outlines

orchestra-research

指導使用 Outlines 函式庫，透過本機模型進行受結構約束的文字生成。

AI & Agents13K3 個月前更新

更多AI & Agents技能

Ppt Template Creator

anthropics

精選

將使用者的 PowerPoint 範本轉換為可重用的技能，用來產生品牌簡報。

AI & Agents39K2 週前更新

Skill Development

anthropics

精選

指導建立 Claude Code 外掛技能，涵蓋結構、描述、漸進式揭露與驗證。

AI & Agents37K今天更新

Plugin Structure

anthropics

精選

說明 Claude Code 外掛的結構、資訊清單與元件配置方式。

AI & Agents37K今天更新

Command Development

anthropics

精選

指導建立 Claude Code 斜線命令，涵蓋結構、YAML frontmatter、參數與外掛功能。

AI & Agents37K今天更新

Claude Md Improver

anthropics

精選

稽核儲存庫中的 CLAUDE.md 檔案、評估其品質，並在取得核准後套用針對性的改進。

AI & Agents37K今天更新

Find Skills

vercel-labs

精選

指導透過 npx skills 命令列工具探索、評估並安裝代理技能。

AI & Agents33K今天更新