Phoenix Evals

arize-ai/phoenix/.agents/skills/phoenix-evals

作者 arize-ai4b1fa58bc7d6Apache-2.011K 个星标收录于 2026年10月9日更新于 2026年10月9日仓库今天更新

Build and run evaluators for AI/LLM applications using Phoenix.

仅含说明AI & Agents

AI 生成的概览

指导使用 Phoenix 为 AI/LLM 应用构建和运行评估器，涵盖代码与 LLM 评判、数据集、实验和 CI 门禁。

功能: 该技能提供使用 Phoenix 为 AI 和 LLM 应用构建评估器的参考文档。内容涵盖 Python 与 TypeScript 环境搭建、评判模型选择、预置及自定义的代码或 LLM 评估器、RAG 评估、数据集与合成数据、运行实验、依据人工标注验证评估器准确度、追踪与采样、错误分析以及生产防护措施。它仅为说明文档，本身不生成脚本或文件。
适用场景: 当你需要为 LLM 应用创建、运行或验证评估器，搭建 Phoenix 追踪与错误分析，构建评估数据集或实验，或依据评估结果设置 CI 门禁时使用。
运行要求: 需要 Phoenix 服务器。Python 相关工作需要 phoenix 和 openai 包；TypeScript 相关工作需要 @arizeai/phoenix-client。不附带脚本，仅为参考文档。

Phoenix Evals

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

Quick Reference

Task	Files
Setup	setup-python [blocked], setup-typescript [blocked]
Decide what to evaluate	evaluators-overview [blocked]
Choose a judge model	fundamentals-model-selection [blocked]
Use pre-built evaluators	evaluators-pre-built [blocked]
Build code evaluator	evaluators-code-python [blocked], evaluators-code-typescript [blocked]
Build LLM evaluator	evaluators-llm-python [blocked], evaluators-llm-typescript [blocked], evaluators-custom-templates [blocked]
Batch evaluate DataFrame	evaluate-dataframe-python [blocked]
Run experiment	experiments-running-python [blocked], experiments-running-typescript [blocked]
Run evals in a test runner (CI gate)	integrations-pytest [blocked], integrations-vitest-jest [blocked]
Create dataset	experiments-datasets-python [blocked], experiments-datasets-typescript [blocked]
Generate synthetic data	experiments-synthetic-python [blocked], experiments-synthetic-typescript [blocked]
Validate evaluator accuracy	validation [blocked], validation-evaluators-python [blocked], validation-evaluators-typescript [blocked]
Export spans	observe-tracing-setup [blocked]
Write a span filter (`SpanQuery().where`)	filter-expressions [blocked]
Sample traces for review	observe-sampling-python [blocked], observe-sampling-typescript [blocked]
Analyze errors	error-analysis [blocked], error-analysis-multi-turn [blocked], axial-coding [blocked]
RAG evals	evaluators-rag [blocked]
Avoid common mistakes	common-mistakes-python [blocked], fundamentals-anti-patterns [blocked]
Production	production-overview [blocked], production-guardrails [blocked], production-continuous [blocked]

Workflows

Starting Fresh: observe-tracing-setup [blocked] → error-analysis [blocked] → axial-coding [blocked] → evaluators-overview [blocked]

Building Evaluator: fundamentals [blocked] → common-mistakes-python [blocked] → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

RAG Systems: evaluators-rag [blocked] → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)

Gating CI: evaluators-{code|llm}-{python|typescript} → integrations-{pytest|vitest-jest} → production-continuous [blocked]

Production: production-overview [blocked] → production-guardrails [blocked] → production-continuous [blocked]

Reference Categories

Prefix	Description
`fundamentals-*`	Types, scores, anti-patterns
`observe-*`	Tracing, sampling
`error-analysis-*`	Finding failures
`axial-coding-*`	Categorizing failures
`evaluators-*`	Code, LLM, RAG evaluators
`experiments-*`	Datasets, running experiments
`integrations-*`	Run evals from test runners (pytest, Vitest, Jest) as a CI gate
`validation-*`	Validating evaluator accuracy against human labels
`production-*`	CI/CD, monitoring

Key Principles

Principle	Action
Error analysis first	Can't automate what you haven't observed
Custom > generic	Build from your failures
Code first	Deterministic before LLM
Validate judges	>80% TPR/TNR
Binary > Likert	Pass/fail, not 1-5
Invariants gate, signals trend	`assert`/`expect` hard invariants (CI red); log LLM-judge quality signals and gate the aggregate (acceptance criteria), not every case

来源与署名

来源：arize-ai/phoenix位于.agents/skills/phoenix-evals提交4b1fa58

许可证： Apache-2.0

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

Phoenix Evals

Phoenix Evals

Quick Reference

Workflows

Reference Categories

Key Principles

来源与署名

更多AI & Agents技能

Ppt Template Creator

Skill Development

Plugin Structure

Command Development

Claude Md Improver

Find Skills