Output Dev Evaluator Function

作者 growthxaia4f6bd40ab0e無授權條款440 個星標收錄於 2026年10月8日更新於 2026年10月8日儲存庫今天更新

Create evaluator functions in evaluators.ts for Output SDK workflows. Use when implementing quality assessment, validation logic, or content evaluation.

僅含說明AI & AgentsSoftware Development
AI 產生的概覽

說明如何在 Output SDK 工作流程的 evaluators.ts 中撰寫評估器函式，包括由 LLM 驅動的品質檢查。

功能
這項技能是一份參考指南，介紹如何在 Output SDK 工作流程的 evaluators.ts 檔案中建立評估器函式。內容涵蓋檔案組織方式、匯入規則與活動隔離限制、評估器的必要屬性，以及帶有信賴度的布林、數值和字串結果類型。它也展示由 LLM 驅動的評估模式、回饋物件和多維度結果，並附有程式碼範例與驗證清單。
適用情境
在為 Output SDK 工作流程實作品質評估、驗證邏輯或內容評估時使用。也適用於加入帶信賴度的判斷、以 LLM 為基礎的內容評估，或可重複使用的評估元件。
執行需求
僅為說明文件，不附帶指令碼。需要 Output SDK 專案以及 @outputai/core 和 @outputai/llm 套件，使用 TypeScript 的 ES 模組匯入，並為 LLM 驅動的評估器準備提示詞檔案。

Creating Evaluator Functions

Overview

This skill documents how to create evaluator functions in evaluators.ts for Output SDK workflows. Evaluators are used to assess quality, validate outputs, and provide confidence-scored judgments about workflow results.

When to Use This Skill

  • Implementing quality assessment for workflow outputs
  • Adding validation logic with confidence scores
  • Creating LLM-powered content evaluation
  • Building reusable evaluation components

File Organization

Option 1: Flat File (Default)

For smaller workflows, use a single evaluators.ts file:

src/workflows/{workflow-name}/├── workflow.ts├── steps.ts├── evaluators.ts    # All evaluators in one file├── types.ts└── ...

Option 2: Folder-Based (Large workflows)

For larger workflows with many evaluators, use an evaluators/ folder:

src/workflows/{workflow-name}/├── workflow.ts├── steps.ts├── evaluators/      # Evaluators split into individual files│   ├── quality.ts│   ├── accuracy.ts│   └── completeness.ts├── types.ts└── ...

Component Location Rules

Important: evaluator() calls MUST be in files containing 'evaluators' in the path:

  • src/workflows/my_workflow/evaluators.ts ✓
  • src/workflows/my_workflow/evaluators/quality.ts ✓
  • src/shared/evaluators/common_evaluators.ts ✓
  • src/workflows/my_workflow/helpers.ts ✗ (cannot contain evaluator() calls)

Activity Isolation Constraints

Evaluators are Temporal activities with strict import rules to ensure deterministic replay.

Evaluators CAN import from:

  • Local workflow files: ./utils.js, ./types.js, ./helpers.js
  • Local subdirectories: ./lib/helpers.js
  • Shared utilities: ../../shared/utils/*.js
  • Shared clients: ../../shared/clients/*.js
  • Shared services: ../../shared/services/*.js

Evaluators CANNOT import:

  • Other evaluator files (activity isolation)
  • Step files
  • Workflow files

Example of WRONG imports:

typescript
// WRONG - evaluators cannot import other evaluatorsimport { otherEvaluator } from '../../shared/evaluators/other.js'; // ✗import { anotherEvaluator } from './other_evaluators.js'; // ✗

Critical Import Patterns

Core Imports

typescript
// CORRECT - Import from @outputai/coreimport {  evaluator,  z,  EvaluationBooleanResult,  EvaluationNumberResult,  EvaluationStringResult,  EvaluationFeedback} from '@outputai/core';
// WRONG - Never import z from zodimport { z } from 'zod';

LLM Client Import (for LLM-powered evaluators)

typescript
// CORRECT - Use @outputai/llm wrapperimport { generateText, aiSdk } from '@outputai/llm';
// WRONG - Never call LLM providers directlyimport OpenAI from 'openai';

ES Module Imports

All imports MUST use .js extension:

typescript
// CORRECTimport { BlogContent } from './types.js';
// WRONG - Missing .js extensionimport { BlogContent } from './types';

Basic Structure

typescript
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const myEvaluator = evaluator( {  name: 'my_evaluator',  description: 'Description of what this evaluator assesses',  inputSchema: z.object( { /* input schema */ } ),  fn: async input => {    // Evaluation logic    return new EvaluationBooleanResult( {      value: true,      confidence: 0.95    } );  }} );

Required Properties

name (string)

Unique identifier for the evaluator. Use snake_case.

typescript
name: 'evaluate_content_quality'

description (string)

Human-readable description of what the evaluator assesses.

typescript
description: 'Evaluate the quality and completeness of generated content'

inputSchema (Zod schema)

Schema for validating evaluator input.

typescript
inputSchema: z.object( {  content: z.string(),  expectedLength: z.number()} )

fn (async function)

The evaluator execution function. Returns an evaluation result with value and confidence.

typescript
fn: async input => {  const isValid = input.content.length >= input.expectedLength;  return new EvaluationBooleanResult( {    value: isValid,    confidence: 0.95  } );}

Result Types

EvaluationBooleanResult

Use for pass/fail or true/false evaluations:

typescript
import { EvaluationBooleanResult } from '@outputai/core';
return new EvaluationBooleanResult( {  value: true,           // boolean result  confidence: 0.95,      // 0.0 to 1.0  reasoning: 'Optional explanation of the evaluation'} );

EvaluationNumberResult

Use for numeric scores or ratings:

typescript
import { EvaluationNumberResult } from '@outputai/core';
return new EvaluationNumberResult( {  value: 85,             // numeric result (e.g., 0-100 score)  confidence: 0.85,      // 0.0 to 1.0  reasoning: 'Optional explanation of the score'} );

EvaluationStringResult

Use for categorical or text-based evaluations:

typescript
import { EvaluationStringResult } from '@outputai/core';
return new EvaluationStringResult( {  value: 'positive',     // string result (e.g., category, sentiment, label)  confidence: 0.9,       // 0.0 to 1.0  reasoning: 'Optional explanation of the classification'} );

Result Properties

PropertyTypeRequiredDescription
valueboolean, number, or stringYesThe evaluation result
confidencenumber (0.0-1.0)YesConfidence in the evaluation
reasoningstringNoExplanation of the evaluation
namestringNoName for this specific result (useful in dimensions)
feedbackEvaluationFeedback[]NoArray of feedback objects with issues and suggestions
dimensionsEvaluationResult[]NoNested results for multi-dimensional evaluation

Simple Evaluator Examples

Boolean Evaluator - Content Validation

typescript
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const evaluateCompleteness = evaluator( {  name: 'evaluate_completeness',  description: 'Check if content meets minimum length requirements',  inputSchema: z.object( {    content: z.string(),    minLength: z.number().default( 100 )  } ),  fn: async ( { content, minLength } ) => {    const isComplete = content.length >= minLength;
    return new EvaluationBooleanResult( {      value: isComplete,      confidence: 1.0,      reasoning: isComplete ?        `Content has ${content.length} characters, meets minimum of ${minLength}` :        `Content has ${content.length} characters, below minimum of ${minLength}`    } );  }} );

Boolean Evaluator - Pattern Detection

typescript
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';
export const evaluateGibberish = evaluator( {  name: 'evaluate_gibberish',  description: 'Check if a given string is gibberish',  inputSchema: z.string(),  fn: async content => {    const gibberishPatterns = [ 'foo', 'bar', 'lorem', 'ipsum' ];    const isGibberish = gibberishPatterns.some( p => content.toLowerCase().includes( p ) );
    return new EvaluationBooleanResult( {      value: !isGibberish,      confidence: 0.95    } );  }} );

Number Evaluator - Quality Score

typescript
import { evaluator, z, EvaluationNumberResult } from '@outputai/core';
export const evaluateReadability = evaluator( {  name: 'evaluate_readability',  description: 'Calculate readability score based on sentence structure',  inputSchema: z.object( {    content: z.string()  } ),  fn: async ( { content } ) => {    const sentences = content.split( /[.!?]+/ ).filter( s => s.trim() );    const words = content.split( /\s+/ ).filter( w => w.trim() );    const avgWordsPerSentence = words.length / Math.max( sentences.length, 1 );
    // Simple readability score (lower avg words = more readable)    const score = Math.max( 0, Math.min( 100, 100 - ( avgWordsPerSentence - 15 ) * 5 ) );
    return new EvaluationNumberResult( {      value: Math.round( score ),      confidence: 0.8,      reasoning: `Average ${avgWordsPerSentence.toFixed( 1 )} words per sentence`    } );  }} );

String Evaluator - Sentiment Classification

typescript
import { evaluator, z, EvaluationStringResult } from '@outputai/core';
export const evaluateSentiment = evaluator( {  name: 'evaluate_sentiment',  description: 'Classify the sentiment of content',  inputSchema: z.object( {    content: z.string()  } ),  fn: async ( { content } ) => {    const positiveWords = [ 'great', 'excellent', 'amazing', 'good', 'love' ];    const negativeWords = [ 'bad', 'terrible', 'awful', 'hate', 'poor' ];
    const lowerContent = content.toLowerCase();    const positiveCount = positiveWords.filter( w => lowerContent.includes( w ) ).length;    const negativeCount = negativeWords.filter( w => lowerContent.includes( w ) ).length;
    const { sentiment, confidence } = positiveCount > negativeCount ?      { sentiment: 'positive', confidence: Math.min( 0.95, 0.6 + positiveCount * 0.1 ) } :      negativeCount > positiveCount ?        { sentiment: 'negative', confidence: Math.min( 0.95, 0.6 + negativeCount * 0.1 ) } :        { sentiment: 'neutral', confidence: 0.7 };
    return new EvaluationStringResult( {      value: sentiment,      confidence,      reasoning: `Found ${positiveCount} positive and ${negativeCount} negative indicators`    } );  }} );

LLM-Powered Evaluator Examples

Note: Evaluators are self-contained components that don't share schemas across steps, so defining aiSdk.Output.object() schemas inline is acceptable here. For workflow steps that share schemas, define them in types.ts instead.

generateText arguments: prompt, promptDir, variables, tools, output, toolChoice, stopWhen, abortSignal.

Using generateText with aiSdk.Output.object() for Evaluation

typescript
import { evaluator, z, EvaluationNumberResult } from '@outputai/core';import { generateText, aiSdk } from '@outputai/llm';
export const evaluateSignalToNoise = evaluator( {  name: 'evaluate_signal_to_noise',  description: 'Evaluate the signal-to-noise ratio of content',  inputSchema: z.object( {    title: z.string(),    content: z.string()  } ),  fn: async ( { title, content } ) => {    const { output } = await generateText( {      prompt: 'signal_noise@v1',  // References prompts/[email protected]      variables: {        title,        content      },      output: aiSdk.Output.object( {        schema: z.object( {          score: z.number().describe( 'Signal-to-noise score 0-100' )        } )      } )    } );
    return new EvaluationNumberResult( {      value: output.score,      confidence: 0.85    } );  }} );

LLM Boolean Evaluation

typescript
import { evaluator, z, EvaluationBooleanResult } from '@outputai/core';import { generateText, aiSdk } from '@outputai/llm';
export const evaluateFactualAccuracy = evaluator( {  name: 'evaluate_factual_accuracy',  description: 'Check if content contains factual claims that can be verified',  inputSchema: z.object( {    content: z.string(),    topic: z.string()  } ),  fn: async ( { content, topic } ) => {    const { output } = await generateText( {      prompt: 'factual_check@v1',      variables: { content, topic },      output: aiSdk.Output.object( {        schema: z.object( {          isFactual: z.boolean().describe( 'Whether content appears factually accurate' ),          confidence: z.number().describe( 'Confidence in assessment 0-1' ),          issues: z.array( z.string() ).optional().describe( 'Any factual issues found' )        } )      } )    } );
    return new EvaluationBooleanResult( {      value: output.isFactual,      confidence: output.confidence,      reasoning: output.issues?.length ?        `Issues found: ${output.issues.join( ', ' )}` :        'No factual issues detected'    } );  }} );

LLM String Evaluation - Content Classification

typescript
import { evaluator, z, EvaluationStringResult } from '@outputai/core';import { generateText, aiSdk } from '@outputai/llm';
export const evaluateContentCategory = evaluator( {  name: 'evaluate_content_category',  description: 'Classify content into a category',  inputSchema: z.object( {    content: z.string(),    categories: z.array( z.string() )  } ),  fn: async ( { content, categories } ) => {    const { output } = await generateText( {      prompt: 'categorize_content@v1',      variables: {        content,        categories: categories.join( ', ' )      },      output: aiSdk.Output.object( {        schema: z.object( {          category: z.string().describe( 'The best matching category' ),          confidence: z.number().describe( 'Confidence in classification 0-1' ),          explanation: z.string().describe( 'Why this category was chosen' )        } )      } )    } );
    return new EvaluationStringResult( {      value: output.category,      confidence: output.confidence,      reasoning: output.explanation    } );  }} );

EvaluationResult with Feedback

Use the feedback field to provide actionable improvement suggestions alongside your evaluation result. Import EvaluationFeedback from @outputai/core to create feedback objects.

typescript
import { evaluator, z, EvaluationStringResult, EvaluationFeedback } from '@outputai/core';
export const evaluateWithFeedback = evaluator( {  name: 'evaluate_with_feedback',  description: 'Evaluate content quality and provide actionable feedback',  inputSchema: z.string(),  fn: async response => {    const feedback = [];
    if ( response.length < 50 ) {      feedback.push( new EvaluationFeedback( {        issue: 'Response is too short',        suggestion: 'Expand the response with more detail',        priority: 'medium'      } ) );    }
    return new EvaluationStringResult( {      value: feedback.length === 0 ? 'good' : 'needs_improvement',      confidence: 0.85,      feedback: feedback    } );  }} );

EvaluationFeedback Properties

PropertyTypeDescription
issuestringThe problem identified
suggestionstringRecommended fix
prioritystringPriority level (e.g., 'low', 'medium', 'high')

Multi-Dimensional Evaluation

Use the dimensions field to nest EvaluationResult instances for sub-scores. Each dimension should use the name field to identify it.

typescript
import { evaluator, z, EvaluationStringResult, EvaluationNumberResult } from '@outputai/core';
export const evaluateMultiDimensional = evaluator( {  name: 'evaluate_multi_dimensional',  description: 'Evaluate content across multiple quality dimensions',  inputSchema: z.string(),  fn: async response => {    const coherenceScore = calculateCoherence( response );    const relevanceScore = calculateRelevance( response );    const overallScore = ( coherenceScore + relevanceScore ) / 2;
    return new EvaluationStringResult( {      value: overallScore > 0.7 ? 'high_quality' : 'low_quality',      confidence: 0.9,      dimensions: [        new EvaluationNumberResult( {          value: coherenceScore,          confidence: 0.85,          name: 'coherence'        } ),        new EvaluationNumberResult( {          value: relevanceScore,          confidence: 0.88,          name: 'relevance'        } )      ]    } );  }} );

Complete Example

Based on a real workflow evaluator file:

typescript
import { evaluator, z, EvaluationBooleanResult, EvaluationNumberResult } from '@outputai/core';import { generateText, aiSdk } from '@outputai/llm';import { blogContentSchema } from './types.js';import type { BlogContent, QualityMetrics } from './types.js';
// Simple boolean evaluatorexport const evaluateMinimumLength = evaluator( {  name: 'evaluate_minimum_length',  description: 'Check if blog content meets minimum length requirements',  inputSchema: blogContentSchema,  fn: async ( input: BlogContent ) => {    const MIN_TOKENS = 500;    const meetsRequirement = input.tokenCount >= MIN_TOKENS;
    return new EvaluationBooleanResult( {      value: meetsRequirement,      confidence: 1.0,      reasoning: `Content has ${input.tokenCount} tokens (minimum: ${MIN_TOKENS})`    } );  }} );
// LLM-powered number evaluatorexport const evaluateSignalToNoise = evaluator( {  name: 'evaluate_signal_to_noise',  description: 'Evaluate the signal-to-noise ratio of blog content',  inputSchema: blogContentSchema,  fn: async ( input: BlogContent ) => {    const { output } = await generateText( {      prompt: 'signal_noise@v1',      variables: {        title: input.title,        content: input.content      },      output: aiSdk.Output.object( {        schema: z.object( {          score: z.number().describe( 'Signal-to-noise score 0-100' )        } )      } )    } );
    return new EvaluationNumberResult( {      value: output.score,      confidence: 0.85    } );  }} );
// LLM-powered boolean evaluatorexport const evaluateRelevance = evaluator( {  name: 'evaluate_relevance',  description: 'Check if content is relevant to the stated topic',  inputSchema: z.object( {    content: z.string(),    topic: z.string(),    keywords: z.array( z.string() )  } ),  fn: async ( { content, topic, keywords } ) => {    const { output } = await generateText( {      prompt: 'relevance_check@v1',      variables: { content, topic, keywords: keywords.join( ', ' ) },      output: aiSdk.Output.object( {        schema: z.object( {          isRelevant: z.boolean(),          relevanceScore: z.number().describe( 'Relevance score 0-1' ),          explanation: z.string()        } )      } )    } );
    return new EvaluationBooleanResult( {      value: output.isRelevant,      confidence: output.relevanceScore,      reasoning: output.explanation    } );  }} );

Best Practices

1. Use Appropriate Result Types

typescript
// Boolean for pass/fail decisionsreturn new EvaluationBooleanResult( { value: true, confidence: 0.9 } );
// Number for scores and ratingsreturn new EvaluationNumberResult( { value: 85, confidence: 0.85 } );
// String for categories, labels, or classificationsreturn new EvaluationStringResult( { value: 'positive', confidence: 0.9 } );

2. Provide Meaningful Confidence Scores

typescript
// High confidence for deterministic checksconfidence: 1.0  // e.g., length checks, pattern matching
// Medium confidence for heuristic-based evaluationsconfidence: 0.85  // e.g., LLM-based assessments
// Lower confidence for uncertain evaluationsconfidence: 0.7  // e.g., subjective quality judgments

3. Include Reasoning for Transparency

typescript
return new EvaluationBooleanResult( {  value: false,  confidence: 0.95,  reasoning: `Content contains ${errorCount} grammatical errors, exceeding threshold of ${maxErrors}`} );

4. Keep Evaluators Focused

typescript
// Good - single responsibilityexport const evaluateGrammar = evaluator( { ... } );export const evaluateReadability = evaluator( { ... } );export const evaluateTone = evaluator( { ... } );
// Avoid - doing too much in one evaluatorexport const evaluateEverything = evaluator( { ... } );

5. Use Descriptive Names

typescript
// Good - clear what is being evaluatedname: 'evaluate_content_originality'name: 'evaluate_factual_accuracy'name: 'evaluate_sentiment_alignment'
// Avoid - vague namesname: 'check'name: 'validate'name: 'evaluate_stuff'

6. Use Feedback for Actionable Improvements

typescript
feedback: [  new EvaluationFeedback( {    issue: 'Missing conclusion paragraph',    suggestion: 'Add a summary paragraph at the end',    priority: 'high'  } )]

7. Use Dimensions for Multi-Criteria Evaluation

typescript
dimensions: [  new EvaluationNumberResult( { value: 8, confidence: 0.9, name: 'coherence' } ),  new EvaluationNumberResult( { value: 6, confidence: 0.85, name: 'relevance' } )]

Verification Checklist

  • evaluator, z, result types imported from @outputai/core
  • generateText and aiSdk imported from @outputai/llm if using LLM (not direct provider)
  • LLM output schemas use .describe() instead of .min()/.max() on z.number()
  • All imports use .js extension
  • Named exports used for each evaluator
  • Each evaluator has name, description, inputSchema, fn
  • Evaluator name uses snake_case
  • Returns appropriate result type (EvaluationBooleanResult, EvaluationNumberResult, or EvaluationStringResult)
  • Confidence score between 0.0 and 1.0
  • Evaluators only import allowed dependencies (local files, shared code)
  • No imports of other evaluators, steps, or workflows
  • EvaluationFeedback imported from @outputai/core when using feedback
  • Feedback objects include issue, suggestion, and priority
  • Dimensions use the name field to identify sub-evaluations

Related Skills

  • output-dev-workflow-function - Orchestrating evaluators in workflow.ts
  • output-dev-step-function - Creating step functions
  • output-dev-types-file - Defining evaluator input schemas
  • output-dev-prompt-file - Creating prompt files for LLM-powered evaluators
  • output-dev-folder-structure - Understanding project layout
  • output-eval-error-analysis — Identify what to evaluate before writing evaluators

來源與署名

來源：growthxai/output位於coding_assistants/claude/plugins/outputai/skills/output-dev-evaluator-function提交a4f6bd4

授權條款： 無授權條款

內容歸原作者所有。SourceWeft 從公開儲存庫中收錄這些內容。

檢舉或申請下架

更多來自 growthxai/output 的技能

Output Workflow Trace File

growthxai

將本機 Output SDK 工作流追蹤檔案的最終輸出渲染成易讀的 Markdown。

Productivity & Workflow440今天更新

Output Plan Workflow

growthxai

引導代理為新的 Output.ai 工作流程產出一份 Markdown 實作計畫。

AI & Agents440今天更新

Output Meta Post Flight

growthxai

用於 Output SDK 工作流程執行後的驗證清單，涵蓋步驟完成、規範遵循、品質與交付項目驗證。

AI & Agents440今天更新

Output Dev Types File

growthxai

說明如何為 Output SDK 工作流程撰寫內含 Zod 結構定義的 types.ts 檔案。

Software Development440今天更新

Output Dev Step Function

growthxai

說明如何在 Output SDK 工作流程的 steps.ts 中撰寫步驟函式，涵蓋 I/O、HTTP、LLM 呼叫與錯誤處理。

Software Development440今天更新

Output Dev Scenario File

growthxai

說明如何為 Output SDK 工作流程建立測試情境 JSON 檔案，並使其符合工作流程的輸入 schema。

Software Development440今天更新

更多AI & Agents技能

Skill Development

anthropics

精選

指導建立 Claude Code 外掛技能，涵蓋結構、描述、漸進式揭露與驗證。

AI & Agents37K今天更新

Plugin Structure

anthropics

精選

說明 Claude Code 外掛的結構、資訊清單與元件配置方式。

AI & Agents37K今天更新

Command Development

anthropics

精選

指導建立 Claude Code 斜線命令，涵蓋結構、YAML frontmatter、參數與外掛功能。

AI & Agents37K今天更新

Claude Md Improver

anthropics

精選

稽核儲存庫中的 CLAUDE.md 檔案、評估其品質，並在取得核准後套用針對性的改進。

AI & Agents37K今天更新

Google Cloud Solution Agentic Ai Data Science Workflow

google

精選

指導在 Google Cloud 上設計多產品代理式資料科學架構，涵蓋需求、部署與驗證。

AI & Agents21K今天更新

Google Cloud Solution Agentic Ai Borderless Data Lakehouse

google

精選

指導在 Google Cloud 上設計無邊界多雲開放資料湖倉，涵蓋需求探索、部署與驗證。

AI & Agents21K今天更新