Phoenix Evals

Guides building and running evaluators for AI/LLM applications with Phoenix, covering code and LLM judges, datasets, experiments, and CI gating.

What it does

This skill provides reference documentation for building evaluators for AI and LLM applications using Phoenix. It covers setup in Python and TypeScript, choosing judge models, pre-built and custom code or LLM evaluators, RAG evaluation, datasets and synthetic data, running experiments, validating evaluator accuracy against human labels, tracing and sampling, error analysis, and production guardrails. It is instructions only and produces no scripts or files itself.

When to use it

Use it when you need to create, run, or validate evaluators for an LLM application, set up Phoenix tracing and error analysis, build evaluation datasets or experiments, or gate CI on evaluation results.

Requirements

Requires a Phoenix server. Python work needs the phoenix and openai packages; TypeScript work needs @arizeai/phoenix-client. Ships no scripts; it is reference documentation only.