Eval Audit
ai-evals-course/evals-skills/skills/eval-auditby ai-evals-course80d5f7b0127c7572ed9e9339937adbfd7240ffebNo license1.4K starsListed Oct 9, 2026Updated Oct 9, 2026Repository updated 2 weeks ago
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).
- 80d5f7b0127c7572ed9e9339937adbfd7240ffebCurrentcommit 80d5f7bPublished Oct 9, 2026
Source and attribution
Source:ai-evals-course/evals-skillsinskills/eval-auditat commit80d5f7b
License: No license
Content belongs to its original authors. SourceWeft indexes it from a public repository.
More from ai-evals-course/evals-skills
Validate Evaluator
ai-evals-course
Calibrates an LLM judge against human labels using data splits, TPR/TNR metrics, and bias correction.
Generate Synthetic Data
ai-evals-course
Generates diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation.
Evaluate Rag
ai-evals-course
Guides evaluation of RAG retrieval and generation quality with metrics, datasets and chunking optimization.
Evals Start
ai-evals-course
Routes eval-related requests to the appropriate specialized skill in the same plugin.
More in AI & Agents
Discernment Nudge
anthropics
Appends 2-3 specific follow-up questions to substantive answers so users can check facts, reasoning and missing context.
Ppt Template Creator
anthropics
Turns a user's PowerPoint template into a reusable skill that generates branded presentations.
Skill Development
anthropics
Guides creation of Claude Code plugin skills, covering structure, descriptions, progressive disclosure and validation.
Plugin Structure
anthropics
Guides the structure, manifest, and component layout of Claude Code plugins.
Command Development
anthropics
Guides creation of Claude Code slash commands, covering structure, YAML frontmatter, arguments and plugin features.
Claude Md Improver
anthropics
Audits CLAUDE.md files in a repository, scores their quality, and applies approved targeted improvements.