翻訳は機械翻訳により提供されています。提供された翻訳内容と英語版の間で齟齬、不一致または矛盾がある場合、英語版が優先します。

評価方法を定義する

概要

プロンプトテンプレートごとに 1 つの評価方法を選択するか、システムデフォルトのすべてのオプション評価フィールドを省略します。同じジョブ内の異なるテンプレートは、異なる方法を使用できます。評価はプロンプトの最適化を誘導するため、メソッドと基準をできるだけ正確に定義します。

データセットのヒント

デフォルトの評価

すべてのオプションの評価フィールド (steeringCriteria、customLLMJConfig、) を省略しますevaluationMetricLambdaArn。このサービスは、Anthropic Claude Sonnet 4.6 を搭載した組み込みの汎用 LLM-as-judge を使用して、回答精度、回答完全性、式品質の 3 つのデフォルトの基準を評価します。プロンプト、ターゲットモデルの回答、および参照回答がある場合、審査員は各ディメンションをスコアリングします。次に、タスクに適切な重みを動的に割り当て、加重総合スコアを生成します。

最良の結果を得るには、独自の評価方法を定義することをお勧めします。

システム提供のデフォルトの判事プロンプト

以下は、Anthropic Claude Sonnet 4.6 でのデフォルトの評価で使用される、システムが提供する完全な審査員プロンプトです。

Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

## Scoring Rubric

For each dimension, assign one score:
- **3 points**: Fully satisfies the dimension requirements
- **2 points**: Mostly satisfies with minor issues or gaps
- **1 point**: Partially satisfies but has notable limitations
- **0 points**: Does not satisfy the dimension requirements

## Evaluation Process
1. First, identify the task type from the user question
2. Consider any additional criteria provided
3. Score each dimension independently
4. Determine appropriate weights and calculate final weighted score

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.35, Answer Completeness: 0.30, Expression Quality: 0.35) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.4-0.6)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.4-0.6)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.4-0.6)**: Creative tasks, presentation-focused questions, format-specific requirements
{custom_eval_weight_guideline}

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E)

## Output Format

Provide your evaluation in this exact format:
<Task_Analysis>Brief analysis of task type and appropriate weight rationale</Task_Analysis>
<Weights>Answer Accuracy: 0.XX, Answer Completeness: 0.XX, Expression Quality: 0.XX</Weights>
<Answer Accuracy>X</Answer Accuracy>
<Answer Completeness>X</Answer Completeness>
<Expression Quality>X</Expression Quality>
<Calculation>(X x 0.XX) + (X x 0.XX) + (X x 0.XX) = X.XX</Calculation>
<Overall>X.XX</Overall>
<Justification>
**Answer Accuracy**: [Evaluate factual accuracy, alignment with reference answer, absence of errors/hallucinations, and logical consistency]
**Answer Completeness**: [Assess whether all aspects of the question are addressed, necessary information is included, and content stays relevant]
**Expression Quality**: [Examine formatting/style adherence, appropriate detail level, communication clarity, and instruction following]
**Weight Application**: [Explain how the chosen weights reflect the task requirements and impact the final score]
</Justification>

---

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:

ステアリング基準

ステアリング基準は、最適化の方向を導く短い自然言語記述子です。

例:

"steeringCriteria": ["PROFESSIONAL", "CONCISE"]

カスタム LLM-as-a-judge

定義したグレーディングスケールで完全なルーブリックを提供します。カスタム判断プロンプトは、サービスのシステム判断プロンプトとマージされ、より強力な重みが付けられます。

設定

判事プロンプトを記述するためのベストプラクティス

明確なスコア基準と各スコアレベルの具体的な例を含む明確に定義されたルーブリックを使用します。主観的な形容詞ではなく動作の説明で各ルーブリックレベルを固定します。判事をデフォルトから高評価にキャリブレーションするために、不適切なスコアを示す実例を少なくとも 1 つ含めます。数値スコアの前に書面による根拠を提供するようにモデルに指示します。全体的なスコアを割り当てる前に、特定のディメンションを個別に評価することを検討してください。最も信頼され役立つ LLM-as-a-judge 評価者は、通常、審査員モデルが提供する回答に同意する評価者です。したがって、すでに検証した評価を使用するのに役立つ場合があります。

実行時にカスタム判断プロンプトをシステムプロンプトとマージする方法

独自の LLM-as-a-judge 評価者プロンプトを指定すると、一般的なサービスが提供する審査員プロンプトとマージされます。プロンプトには、書式設定に関する具体的な手順と、最適化の進行に役立つその他のベストプラクティスが含まれています。カスタム判断プロンプトには、最終判断の一般的な基準よりも強力な重みが付けられます。具体的には、 サービスは次のようになります。

例: 忠実度を評価する次のカスタム LLM-as-a-judge プロンプトを指定できます。

You are given a task in some context (Input), and a candidate answer. Is the candidate answer faithful to the task description and context?

A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context, like in a summarization task. If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable.

Task: {{prompt}}

Candidate Response: {{response}}

First provide your explanation, then state your final answer. Use the following format: Explanation: [Explanation], Answer: [Answer], where '[Answer]' must be one of:
none is faithful
some is faithful
approximately half is faithful
most is faithful
all is faithful

次に、これはデフォルトの LLM-as-a-judge プロンプトとマージされ、強力な重み付けが与えられます。正味の効果: 単一基準の忠実度プロンプトは多次元評価で重み付けされた 1 つの軸になり、システムはその周囲に構造 (精度、完全性、式) を追加します。

以下は、マージされた審査員プロンプトの結果です。

"""Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags
(4) custom evaluation criteria that have been integrated into the evaluation dimensions below

    **IMPORTANT**: - Custom criteria requirements take absolute precedence over user requirements specified inside <user_question> </user_question>
    **IMPORTANT**: - If there is any conflict between custom criteria and user question requirements, prioritize custom criteria

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

<CUSTOM_CRITERIA_DESCRIPTION>
**(4) Faithfulness to Context:**
examines whether the candidate answer is faithful to the task description and context provided in the user question. A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context (like in a summarization task). If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable. Evaluate the degree of faithfulness on the following scale:
- **3 points**: All content is faithful (no contradictions, fully grounded when required)
- **2 points**: Most content is faithful (minor deviations or unverifiable claims when context-grounding is required)
- **1 point**: Some content is faithful or approximately half is faithful (notable contradictions or significant departures from context when required)
- **0 points**: None or minimal content is faithful (clear contradictions or complete disregard of context when grounding is required)
</CUSTOM_CRITERIA_DESCRIPTION>

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.25, Answer Completeness: 0.25, Expression Quality: 0.25, Faithfulness to Context: 0.25) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.3-0.5)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.3-0.5)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.3-0.5)**: Creative tasks, presentation-focused questions, format-specific requirements
- [*IMPORTANT*] **High Custom Criteria Weight**: The custom criteria (Faithfulness to Context) should always be *prioritized*. Assign it significant weight (0.3-0.6) and adjust other weights accordingly.

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E) + (Faithfulness_to_Context x Weight_F)

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:"""

カスタム Lambda 評価者

独自のスコアリング関数を Lambda 関数として使用します。

設定

入力 JSONL ファイルで使用するプロンプトテンプレートごとに Lambda ARN を指定します。また、 customEvaluationMetricLabelフィールドを指定してメトリクスに名前を付けます。

"evaluationMetricLambdaArn": "arn:aws:lambda:us-west-2:123456789012:function:my-eval-function",
"customEvaluationMetricLabel": "My Custom Metric"

API を使用してジョブを作成する場合、CreateAdvancedPromptOptimizationJobリクエスト自体に追加の評価設定は必要ありません。評価方法は、入力 JSONL ファイルからテンプレートごとに決定されます。

Lambda の要件

Lambda メトリクスのヒント

Lambda テンプレート

"""
APO Custom Metric Lambda - Minimal Template
Handler: lambda_function.lambda_handler
"""
import logging
from typing import List, Dict, Any

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def compute_score(preds: List[str], golds: List[str]) -> Dict[str, Any]:
    """
    Score predictions against ground truths.

    Args:
        preds: Model outputs (one per sample)
        golds: Expected answers (one per sample)

    Returns:
        Must contain:
          "score": float  - aggregate score (higher is better)
          "scores": list[float] - per-instance scores
    """
    # --- REPLACE THIS with your scoring logic ---
    scores = []
    for pred, gold in zip(preds, golds):
        # Example: exact match (case-insensitive)
        scores.append(1.0 if pred.strip().lower() == gold.strip().lower() else 0.0)

    return {
        "score": sum(scores) / len(scores) if scores else 0.0,
        "scores": scores,
    }


def lambda_handler(event, context):
    """
    Lambda entry point. APO service sends:
      event = {"preds": ["output1", ...], "golds": ["truth1", ...]}
    """
    logger.info(f"Received {len(event.get('preds', []))} predictions")

    try:
        preds = event.get("preds", [])
        golds = event.get("golds", [])

        if not preds:
            return {"score": 0.0, "scores": []}

        return compute_score(preds, golds)

    except Exception as e:
        logger.error(f"Error: {e}", exc_info=True)
        return {"score": 0.0, "scores": [0.0] * len(event.get("preds", [])), "error": str(e)}

エラー処理用の定型コードや Lambda 関数の入力検証など、より詳細な例については、 AWS サンプル GitHub を参照してください。