Output Dev Evaluator Function

作者 growthxaia4f6bd40ab0e无许可证440 个星标收录于 2026年10月8日更新于 2026年10月8日仓库今天更新

Create evaluator functions in evaluators.ts for Output SDK workflows. Use when implementing quality assessment, validation logic, or content evaluation.

仅含说明AI & AgentsSoftware Development
AI 生成的概览

说明如何在 Output SDK 工作流的 evaluators.ts 中编写评估器函数，包括由 LLM 驱动的质量检查。

功能
该技能是一份参考指南，介绍如何在 Output SDK 工作流的 evaluators.ts 文件中创建评估器函数。内容涵盖文件组织方式、导入规则与活动隔离限制、评估器的必需属性，以及带置信度的布尔、数值和字符串结果类型。它还展示了由 LLM 驱动的评估模式、反馈对象和多维结果，并附有代码示例与验证清单。
适用场景
在为 Output SDK 工作流实现质量评估、验证逻辑或内容评估时使用。也适用于添加带置信度的判断、基于 LLM 的内容评估或可复用的评估组件。
运行要求
仅为说明文档，不附带脚本。需要 Output SDK 项目以及 @outputai/core 和 @outputai/llm 包，使用 TypeScript 的 ES 模块导入，并为 LLM 驱动的评估器准备提示词文件。

Creating Evaluator Functions

Overview

This skill documents how to create evaluator functions in evaluators.ts for Output SDK workflows. Evaluators are used to assess quality, validate outputs, and provide confidence-scored judgments about workflow results.

When to Use This Skill

  • Implementing quality assessment for workflow outputs
  • Adding validation logic with confidence scores
  • Creating LLM-powered content evaluation
  • Building reusable evaluation components

File Organization

Option 1: Flat File (Default)

For smaller workflows, use a single evaluators.ts file:

src/workflows/{workflow-name}/├── workflow.ts├── steps.ts├── evaluators.ts    # All evaluators in one file├── types.ts└── ...

Option 2: Folder-Based (Large workflows)

For larger workflows with many evaluators, use an evaluators/ folder:

src/workflows/{workflow-name}/├── workflow.ts├── steps.ts├── evaluators/      # Evaluators split into individual files│   ├── quality.ts│   ├── accuracy.ts│   └── completeness.ts├── types.ts└── ...

Component Location Rules

Important: evaluator() calls MUST be in files containing 'evaluators' in the path:

  • src/workflows/my_workflow/evaluators.ts ✓
  • src/workflows/my_workflow/evaluators/quality.ts ✓
  • src/shared/evaluators/common_evaluators.ts ✓
  • src/workflows/my_workflow/helpers.ts ✗ (cannot contain evaluator() calls)

Activity Isolation Constraints

Evaluators are Temporal activities with strict import rules to ensure deterministic replay.

Evaluators CAN import from:

  • Local workflow files: ./utils.js, ./types.js, ./helpers.js
  • Local subdirectories: ./lib/helpers.js
  • Shared utilities: ../../shared/utils/*.js
  • Shared clients: ../../shared/clients/*.js
  • Shared services: ../../shared/services/*.js

Evaluators CANNOT import:

  • Other evaluator files (activity isolation)
  • Step files
  • Workflow files

Example of WRONG imports:

typescript
// WRONG - evaluators cannot import other evaluatorsimport { otherEvaluator } from '../../shared/evaluators/other.js'; // ✗import { anotherEvaluator } from './other_evaluators.js'; // ✗

Critical Import Patterns

Core Imports

typescript
// CORRECT - Import from @outputai/coreimport {  evaluator,  z,  EvaluationBooleanResult,  EvaluationNumberResult,  EvaluationStringResult,  EvaluationFeedback} from '@outputai/core';
// WRONG - Never import z from zodimport { z } from 'zod';

LLM Client Import (for LLM-powered evaluators)

typescript
// CORRECT - Use @outputai/llm wrapperimport { generateText, aiSdk } from '@outputai/llm';
// WRONG - Never call LLM providers directlyimport OpenAI from 'openai';

ES Module Imports

All imports MUST use .js extension:

typescript
// CORRECTimport { BlogContent } from './types.js';
// WRONG - Missing .js extensionimport { BlogContent } from './types';

Basic Structure

typescript
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const myEvaluator = evaluator( {  name: 'my_evaluator',  description: 'Description of what this evaluator assesses',  inputSchema: z.object( { /* input schema */ } ),  fn: async input => {    // Evaluation logic    return new EvaluationBooleanResult( {      value: true,      confidence: 0.95    } );  }} );

Required Properties

name (string)

Unique identifier for the evaluator. Use snake_case.

typescript
name: 'evaluate_content_quality'

description (string)

Human-readable description of what the evaluator assesses.

typescript
description: 'Evaluate the quality and completeness of generated content'

inputSchema (Zod schema)

Schema for validating evaluator input.

typescript
inputSchema: z.object( {  content: z.string(),  expectedLength: z.number()} )

fn (async function)

The evaluator execution function. Returns an evaluation result with value and confidence.

typescript
fn: async input => {  const isValid = input.content.length >= input.expectedLength;  return new EvaluationBooleanResult( {    value: isValid,    confidence: 0.95  } );}

Result Types

EvaluationBooleanResult

Use for pass/fail or true/false evaluations:

typescript
import { EvaluationBooleanResult } from '@outputai/core';
return new EvaluationBooleanResult( {  value: true,           // boolean result  confidence: 0.95,      // 0.0 to 1.0  reasoning: 'Optional explanation of the evaluation'} );

EvaluationNumberResult

Use for numeric scores or ratings:

typescript
import { EvaluationNumberResult } from '@outputai/core';
return new EvaluationNumberResult( {  value: 85,             // numeric result (e.g., 0-100 score)  confidence: 0.85,      // 0.0 to 1.0  reasoning: 'Optional explanation of the score'} );

EvaluationStringResult

Use for categorical or text-based evaluations:

typescript
import { EvaluationStringResult } from '@outputai/core';
return new EvaluationStringResult( {  value: 'positive',     // string result (e.g., category, sentiment, label)  confidence: 0.9,       // 0.0 to 1.0  reasoning: 'Optional explanation of the classification'} );

Result Properties

PropertyTypeRequiredDescription
valueboolean, number, or stringYesThe evaluation result
confidencenumber (0.0-1.0)YesConfidence in the evaluation
reasoningstringNoExplanation of the evaluation
namestringNoName for this specific result (useful in dimensions)
feedbackEvaluationFeedback[]NoArray of feedback objects with issues and suggestions
dimensionsEvaluationResult[]NoNested results for multi-dimensional evaluation

Simple Evaluator Examples

Boolean Evaluator - Content Validation

typescript
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const evaluateCompleteness = evaluator( {  name: 'evaluate_completeness',  description: 'Check if content meets minimum length requirements',  inputSchema: z.object( {    content: z.string(),    minLength: z.number().default( 100 )  } ),  fn: async ( { content, minLength } ) => {    const isComplete = content.length >= minLength;
    return new EvaluationBooleanResult( {      value: isComplete,      confidence: 1.0,      reasoning: isComplete ?        `Content has ${content.length} characters, meets minimum of ${minLength}` :        `Content has ${content.length} characters, below minimum of ${minLength}`    } );  }} );

Boolean Evaluator - Pattern Detection

typescript
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const evaluateGibberish = evaluator( {  name: 'evaluate_gibberish',  description: 'Check if a given string is gibberish',  inputSchema: z.string(),  fn: async content => {    const gibberishPatterns = [ 'foo', 'bar', 'lorem', 'ipsum' ];    const isGibberish = gibberishPatterns.some( p => content.toLowerCase().includes( p ) );
    return new EvaluationBooleanResult( {      value: !isGibberish,      confidence: 0.95    } );  }} );

Number Evaluator - Quality Score

typescript
import { evaluator, z, EvaluationNumberResult } from '@outputai/core';
export const evaluateReadability = evaluator( {  name: 'evaluate_readability',  description: 'Calculate readability score based on sentence structure',  inputSchema: z.object( {    content: z.string()  } ),  fn: async ( { content } ) => {    const sentences = content.split( /[.!?]+/ ).filter( s => s.trim() );    const words = content.split( /\s+/ ).filter( w => w.trim() );    const avgWordsPerSentence = words.length / Math.max( sentences.length, 1 );
    // Simple readability score (lower avg words = more readable)    const score = Math.max( 0, Math.min( 100, 100 - ( avgWordsPerSentence - 15 ) * 5 ) );
    return new EvaluationNumberResult( {      value: Math.round( score ),      confidence: 0.8,      reasoning: `Average ${avgWordsPerSentence.toFixed( 1 )} words per sentence`    } );  }} );

String Evaluator - Sentiment Classification

typescript
import { evaluator, z, EvaluationStringResult } from '@outputai/core';
export const evaluateSentiment = evaluator( {  name: 'evaluate_sentiment',  description: 'Classify the sentiment of content',  inputSchema: z.object( {    content: z.string()  } ),  fn: async ( { content } ) => {    const positiveWords = [ 'great', 'excellent', 'amazing', 'good', 'love' ];    const negativeWords = [ 'bad', 'terrible', 'awful', 'hate', 'poor' ];
    const lowerContent = content.toLowerCase();    const positiveCount = positiveWords.filter( w => lowerContent.includes( w ) ).length;    const negativeCount = negativeWords.filter( w => lowerContent.includes( w ) ).length;
    const { sentiment, confidence } = positiveCount > negativeCount ?      { sentiment: 'positive', confidence: Math.min( 0.95, 0.6 + positiveCount * 0.1 ) } :      negativeCount > positiveCount ?        { sentiment: 'negative', confidence: Math.min( 0.95, 0.6 + negativeCount * 0.1 ) } :        { sentiment: 'neutral', confidence: 0.7 };
    return new EvaluationStringResult( {      value: sentiment,      confidence,      reasoning: `Found ${positiveCount} positive and ${negativeCount} negative indicators`    } );  }} );

LLM-Powered Evaluator Examples

Note: Evaluators are self-contained components that don't share schemas across steps, so defining aiSdk.Output.object() schemas inline is acceptable here. For workflow steps that share schemas, define them in types.ts instead.

generateText arguments: prompt, promptDir, variables, tools, output, toolChoice, stopWhen, abortSignal.

Using generateText with aiSdk.Output.object() for Evaluation

typescript
import { evaluator, z, EvaluationNumberResult } from '@outputai/core';import { generateText, aiSdk } from '@outputai/llm';
export const evaluateSignalToNoise = evaluator( {  name: 'evaluate_signal_to_noise',  description: 'Evaluate the signal-to-noise ratio of content',  inputSchema: z.object( {    title: z.string(),    content: z.string()  } ),  fn: async ( { title, content } ) => {    const { output } = await generateText( {      prompt: 'signal_noise@v1',  // References prompts/[email protected]      variables: {        title,        content      },      output: aiSdk.Output.object( {        schema: z.object( {          score: z.number().describe( 'Signal-to-noise score 0-100' )        } )      } )    } );
    return new EvaluationNumberResult( {      value: output.score,      confidence: 0.85    } );  }} );

LLM Boolean Evaluation

typescript
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';import { generateText, aiSdk } from '@outputai/llm';
export const evaluateFactualAccuracy = evaluator( {  name: 'evaluate_factual_accuracy',  description: 'Check if content contains factual claims that can be verified',  inputSchema: z.object( {    content: z.string(),    topic: z.string()  } ),  fn: async ( { content, topic } ) => {    const { output } = await generateText( {      prompt: 'factual_check@v1',      variables: { content, topic },      output: aiSdk.Output.object( {        schema: z.object( {          isFactual: z.boolean().describe( 'Whether content appears factually accurate' ),          confidence: z.number().describe( 'Confidence in assessment 0-1' ),          issues: z.array( z.string() ).optional().describe( 'Any factual issues found' )        } )      } )    } );
    return new EvaluationBooleanResult( {      value: output.isFactual,      confidence: output.confidence,      reasoning: output.issues?.length ?        `Issues found: ${output.issues.join( ', ' )}` :        'No factual issues detected'    } );  }} );

LLM String Evaluation - Content Classification

typescript
import { evaluator, z, EvaluationStringResult } from '@outputai/core';import { generateText, aiSdk } from '@outputai/llm';
export const evaluateContentCategory = evaluator( {  name: 'evaluate_content_category',  description: 'Classify content into a category',  inputSchema: z.object( {    content: z.string(),    categories: z.array( z.string() )  } ),  fn: async ( { content, categories } ) => {    const { output } = await generateText( {      prompt: 'categorize_content@v1',      variables: {        content,        categories: categories.join( ', ' )      },      output: aiSdk.Output.object( {        schema: z.object( {          category: z.string().describe( 'The best matching category' ),          confidence: z.number().describe( 'Confidence in classification 0-1' ),          explanation: z.string().describe( 'Why this category was chosen' )        } )      } )    } );
    return new EvaluationStringResult( {      value: output.category,      confidence: output.confidence,      reasoning: output.explanation    } );  }} );

EvaluationResult with Feedback

Use the feedback field to provide actionable improvement suggestions alongside your evaluation result. Import EvaluationFeedback from @outputai/core to create feedback objects.

typescript
import { evaluator, z, EvaluationStringResult, EvaluationFeedback } from '@outputai/core';
export const evaluateWithFeedback = evaluator( {  name: 'evaluate_with_feedback',  description: 'Evaluate content quality and provide actionable feedback',  inputSchema: z.string(),  fn: async response => {    const feedback = [];
    if ( response.length < 50 ) {      feedback.push( new EvaluationFeedback( {        issue: 'Response is too short',        suggestion: 'Expand the response with more detail',        priority: 'medium'      } ) );    }
    return new EvaluationStringResult( {      value: feedback.length === 0 ? 'good' : 'needs_improvement',      confidence: 0.85,      feedback: feedback    } );  }} );

EvaluationFeedback Properties

PropertyTypeDescription
issuestringThe problem identified
suggestionstringRecommended fix
prioritystringPriority level (e.g., 'low', 'medium', 'high')

Multi-Dimensional Evaluation

Use the dimensions field to nest EvaluationResult instances for sub-scores. Each dimension should use the name field to identify it.

typescript
import { evaluator, z, EvaluationStringResult, EvaluationNumberResult } from '@outputai/core';
export const evaluateMultiDimensional = evaluator( {  name: 'evaluate_multi_dimensional',  description: 'Evaluate content across multiple quality dimensions',  inputSchema: z.string(),  fn: async response => {    const coherenceScore = calculateCoherence( response );    const relevanceScore = calculateRelevance( response );    const overallScore = ( coherenceScore + relevanceScore ) / 2;
    return new EvaluationStringResult( {      value: overallScore > 0.7 ? 'high_quality' : 'low_quality',      confidence: 0.9,      dimensions: [        new EvaluationNumberResult( {          value: coherenceScore,          confidence: 0.85,          name: 'coherence'        } ),        new EvaluationNumberResult( {          value: relevanceScore,          confidence: 0.88,          name: 'relevance'        } )      ]    } );  }} );

Complete Example

Based on a real workflow evaluator file:

typescript
import { evaluator, z, EvaluationBooleanResult, EvaluationNumberResult } from '@outputai/core';import { generateText, aiSdk } from '@outputai/llm';import { blogContentSchema } from './types.js';import type { BlogContent, QualityMetrics } from './types.js';
// Simple boolean evaluatorexport const evaluateMinimumLength = evaluator( {  name: 'evaluate_minimum_length',  description: 'Check if blog content meets minimum length requirements',  inputSchema: blogContentSchema,  fn: async ( input: BlogContent ) => {    const MIN_TOKENS = 500;    const meetsRequirement = input.tokenCount >= MIN_TOKENS;
    return new EvaluationBooleanResult( {      value: meetsRequirement,      confidence: 1.0,      reasoning: `Content has ${input.tokenCount} tokens (minimum: ${MIN_TOKENS})`    } );  }} );
// LLM-powered number evaluatorexport const evaluateSignalToNoise = evaluator( {  name: 'evaluate_signal_to_noise',  description: 'Evaluate the signal-to-noise ratio of blog content',  inputSchema: blogContentSchema,  fn: async ( input: BlogContent ) => {    const { output } = await generateText( {      prompt: 'signal_noise@v1',      variables: {        title: input.title,        content: input.content      },      output: aiSdk.Output.object( {        schema: z.object( {          score: z.number().describe( 'Signal-to-noise score 0-100' )        } )      } )    } );
    return new EvaluationNumberResult( {      value: output.score,      confidence: 0.85    } );  }} );
// LLM-powered boolean evaluatorexport const evaluateRelevance = evaluator( {  name: 'evaluate_relevance',  description: 'Check if content is relevant to the stated topic',  inputSchema: z.object( {    content: z.string(),    topic: z.string(),    keywords: z.array( z.string() )  } ),  fn: async ( { content, topic, keywords } ) => {    const { output } = await generateText( {      prompt: 'relevance_check@v1',      variables: { content, topic, keywords: keywords.join( ', ' ) },      output: aiSdk.Output.object( {        schema: z.object( {          isRelevant: z.boolean(),          relevanceScore: z.number().describe( 'Relevance score 0-1' ),          explanation: z.string()        } )      } )    } );
    return new EvaluationBooleanResult( {      value: output.isRelevant,      confidence: output.relevanceScore,      reasoning: output.explanation    } );  }} );

Best Practices

1. Use Appropriate Result Types

typescript
// Boolean for pass/fail decisionsreturn new EvaluationBooleanResult( { value: true, confidence: 0.9 } );
// Number for scores and ratingsreturn new EvaluationNumberResult( { value: 85, confidence: 0.85 } );
// String for categories, labels, or classificationsreturn new EvaluationStringResult( { value: 'positive', confidence: 0.9 } );

2. Provide Meaningful Confidence Scores

typescript
// High confidence for deterministic checksconfidence: 1.0  // e.g., length checks, pattern matching
// Medium confidence for heuristic-based evaluationsconfidence: 0.85  // e.g., LLM-based assessments
// Lower confidence for uncertain evaluationsconfidence: 0.7  // e.g., subjective quality judgments

3. Include Reasoning for Transparency

typescript
return new EvaluationBooleanResult( {  value: false,  confidence: 0.95,  reasoning: `Content contains ${errorCount} grammatical errors, exceeding threshold of ${maxErrors}`} );

4. Keep Evaluators Focused

typescript
// Good - single responsibilityexport const evaluateGrammar = evaluator( { ... } );export const evaluateReadability = evaluator( { ... } );export const evaluateTone = evaluator( { ... } );
// Avoid - doing too much in one evaluatorexport const evaluateEverything = evaluator( { ... } );

5. Use Descriptive Names

typescript
// Good - clear what is being evaluatedname: 'evaluate_content_originality'name: 'evaluate_factual_accuracy'name: 'evaluate_sentiment_alignment'
// Avoid - vague namesname: 'check'name: 'validate'name: 'evaluate_stuff'

6. Use Feedback for Actionable Improvements

typescript
feedback: [  new EvaluationFeedback( {    issue: 'Missing conclusion paragraph',    suggestion: 'Add a summary paragraph at the end',    priority: 'high'  } )]

7. Use Dimensions for Multi-Criteria Evaluation

typescript
dimensions: [  new EvaluationNumberResult( { value: 8, confidence: 0.9, name: 'coherence' } ),  new EvaluationNumberResult( { value: 6, confidence: 0.85, name: 'relevance' } )]

Verification Checklist

  • evaluator, z, result types imported from @outputai/core
  • generateText and aiSdk imported from @outputai/llm if using LLM (not direct provider)
  • LLM output schemas use .describe() instead of .min()/.max() on z.number()
  • All imports use .js extension
  • Named exports used for each evaluator
  • Each evaluator has name, description, inputSchema, fn
  • Evaluator name uses snake_case
  • Returns appropriate result type (EvaluationBooleanResult, EvaluationNumberResult, or EvaluationStringResult)
  • Confidence score between 0.0 and 1.0
  • Evaluators only import allowed dependencies (local files, shared code)
  • No imports of other evaluators, steps, or workflows
  • EvaluationFeedback imported from @outputai/core when using feedback
  • Feedback objects include issue, suggestion, and priority
  • Dimensions use the name field to identify sub-evaluations

Related Skills

  • output-dev-workflow-function - Orchestrating evaluators in workflow.ts
  • output-dev-step-function - Creating step functions
  • output-dev-types-file - Defining evaluator input schemas
  • output-dev-prompt-file - Creating prompt files for LLM-powered evaluators
  • output-dev-folder-structure - Understanding project layout
  • output-eval-error-analysis — Identify what to evaluate before writing evaluators

来源与署名

来源：growthxai/output位于coding_assistants/claude/plugins/outputai/skills/output-dev-evaluator-function提交a4f6bd4

许可证： 无许可证

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

更多来自 growthxai/output 的技能

Output Workflow Trace File

growthxai

将本地 Output SDK 工作流追踪文件的最终输出渲染为易读的 Markdown。

Productivity & Workflow440今天更新

Output Plan Workflow

growthxai

引导智能体为新的 Output.ai 工作流生成一份 Markdown 实施计划。

AI & Agents440今天更新

Output Meta Post Flight

growthxai

用于 Output SDK 工作流执行后的校验清单，涵盖步骤完成、规范遵循、质量与交付物验证。

AI & Agents440今天更新

Output Dev Types File

growthxai

说明如何为 Output SDK 工作流编写包含 Zod 模式的 types.ts 文件。

Software Development440今天更新

Output Dev Step Function

growthxai

说明如何在 Output SDK 工作流的 steps.ts 中编写步骤函数，涵盖 I/O、HTTP、LLM 调用与错误处理。

Software Development440今天更新

Output Dev Scenario File

growthxai

说明如何为 Output SDK 工作流创建测试场景 JSON 文件，并使其匹配工作流的输入 schema。

Software Development440今天更新

更多AI & Agents技能

Skill Development

anthropics

精选

指导创建 Claude Code 插件技能，涵盖结构、描述、渐进式披露与验证。

AI & Agents37K今天更新

Plugin Structure

anthropics

精选

指导 Claude Code 插件的结构、清单与组件布局。

AI & Agents37K今天更新

Command Development

anthropics

精选

指导创建 Claude Code 斜杠命令，涵盖结构、YAML frontmatter、参数与插件功能。

AI & Agents37K今天更新

Claude Md Improver

anthropics

精选

审查仓库中的 CLAUDE.md 文件，评估其质量，并在获得批准后应用有针对性的改进。

AI & Agents37K今天更新

Smb Onboard

anthropics

精选

引导小微企业主完成首次设置：连接工具、运行一次体现价值的配方、记录业务背景并设定每周检查节奏。

AI & Agents27K今天更新

Smb Router

anthropics

精选

将小企业主的需求转接到合适的插件技能或命令，并说明可用功能。

AI & Agents27K今天更新