기계 번역으로 제공되는 번역입니다. 제공된 번역과 원본 영어의 내용이 상충하는 경우에는 영어 버전이 우선합니다.

평가 방법 정의

개요

프롬프트 템플릿당 평가 방법 하나를 선택하거나 시스템 기본값에 대한 모든 선택적 평가 필드를 생략합니다. 동일한 작업의 템플릿마다 다른 메서드를 사용할 수 있습니다. 평가는 프롬프트 최적화를 안내하므로 메서드와 기준을 최대한 정확하게 정의합니다.

데이터 세트 팁

기본 평가

모든 선택적 평가 필드(steeringCriteria, customLLMJConfig, evaluationMetricLambdaArn)를 생략합니다. 이 서비스는 응답 정확도, 응답 완전성 및 표현식 품질이라는 세 가지 기본 기준을 평가하는 Anthropic Claude Sonnet 4.6으로 구동되는 내장 일반 LLM-as-judge를 사용합니다. 프롬프트, 대상 모델의 답변 및 참조 답변이 주어지면 판사는 각 차원의 점수를 매깁니다. 그런 다음 작업에 적절한 가중치를 동적으로 할당하고 가중치가 적용된 전체 점수를 생성합니다.

최상의 결과를 얻으려면 자체 평가 방법을 정의하는 것이 좋습니다.

기본 시스템 제공 판사 프롬프트

다음은 Anthropic Claude Sonnet 4.6을 사용한 기본 평가에 사용되는 전체 시스템 제공 판단 프롬프트입니다.

Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

## Scoring Rubric

For each dimension, assign one score:
- **3 points**: Fully satisfies the dimension requirements
- **2 points**: Mostly satisfies with minor issues or gaps
- **1 point**: Partially satisfies but has notable limitations
- **0 points**: Does not satisfy the dimension requirements

## Evaluation Process
1. First, identify the task type from the user question
2. Consider any additional criteria provided
3. Score each dimension independently
4. Determine appropriate weights and calculate final weighted score

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.35, Answer Completeness: 0.30, Expression Quality: 0.35) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.4-0.6)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.4-0.6)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.4-0.6)**: Creative tasks, presentation-focused questions, format-specific requirements
{custom_eval_weight_guideline}

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E)

## Output Format

Provide your evaluation in this exact format:
<Task_Analysis>Brief analysis of task type and appropriate weight rationale</Task_Analysis>
<Weights>Answer Accuracy: 0.XX, Answer Completeness: 0.XX, Expression Quality: 0.XX</Weights>
<Answer Accuracy>X</Answer Accuracy>
<Answer Completeness>X</Answer Completeness>
<Expression Quality>X</Expression Quality>
<Calculation>(X x 0.XX) + (X x 0.XX) + (X x 0.XX) = X.XX</Calculation>
<Overall>X.XX</Overall>
<Justification>
**Answer Accuracy**: [Evaluate factual accuracy, alignment with reference answer, absence of errors/hallucinations, and logical consistency]
**Answer Completeness**: [Assess whether all aspects of the question are addressed, necessary information is included, and content stays relevant]
**Expression Quality**: [Examine formatting/style adherence, appropriate detail level, communication clarity, and instruction following]
**Weight Application**: [Explain how the chosen weights reflect the task requirements and impact the final score]
</Justification>

---

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:

조향 기준

조향 기준은 최적화 방향을 안내하는 짧은 자연어 설명자입니다.

예:

"steeringCriteria": ["PROFESSIONAL", "CONCISE"]

사용자 지정 LLM-as-a-judge

정의한 그레이딩 스케일로 전체 마찰을 제공합니다. 사용자 지정 판단 프롬프트가 서비스의 시스템 판단 프롬프트와 병합되고 더 강력한 가중치가 부여됩니다.

구성

판사 프롬프트 작성 모범 사례

명확한 평가 기준과 각 점수 수준의 구체적인 예와 함께 명확하게 정의된 마찰을 사용합니다. 주관적 형용사가 아닌 동작 설명으로 각 마찰 레벨을 앵커링합니다. 판단자를 기본값에서 높은 평점으로 조정하기 위해 불완전한 점수를 보여주는 작업 예제를 하나 이상 포함합니다. 숫자 점수 앞에 서면 정당화를 제공하도록 모델에 지시합니다. 전체 점수를 할당하기 전에 특정 차원을 독립적으로 평가하는 것이 좋습니다. 가장 신뢰할 수 있고 유용한 LLM-as-a-judge 평가자는 일반적으로 판사 모델이 제공하는 답변에 동의하는 평가자입니다. 따라서 이미 심사한 평가를 사용하는 것이 도움이 될 수 있습니다.

런타임 시 사용자 지정 판단 프롬프트가 시스템 프롬프트와 병합되는 방법

자체 LLM-as-a-judge 평가자 프롬프트를 제공하면 일반 서비스 제공 판사 프롬프트와 병합됩니다. 프롬프트에는 형식 지정에 대한 구체적인 지침과 최적화 진행에 도움이 되는 기타 모범 사례가 포함되어 있습니다. 사용자 지정 판단 프롬프트에는 최종 판단의 일반 기준보다 더 강력한 가중치가 부여됩니다. 특히 서비스는 다음과 같습니다.

예: 충실도를 평가하는 다음과 같은 사용자 지정 LLM-as-a-judge 프롬프트를 제공할 수 있습니다.

You are given a task in some context (Input), and a candidate answer. Is the candidate answer faithful to the task description and context?

A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context, like in a summarization task. If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable.

Task: {{prompt}}

Candidate Response: {{response}}

First provide your explanation, then state your final answer. Use the following format: Explanation: [Explanation], Answer: [Answer], where '[Answer]' must be one of:
none is faithful
some is faithful
approximately half is faithful
most is faithful
all is faithful

그런 다음 기본 LLM-as-a-judge 프롬프트와 병합되고 강력한 가중치가 부여됩니다. 순 효과: 단일 기준 충실도 프롬프트는 다차원 평가에서 가중치가 큰 축 하나가 되는 반면, 시스템은 그 주위에 구조(정확성, 완전성, 표현식)를 추가합니다.

다음은 결과 병합된 판사 프롬프트입니다.

"""Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags
(4) custom evaluation criteria that have been integrated into the evaluation dimensions below

    **IMPORTANT**: - Custom criteria requirements take absolute precedence over user requirements specified inside <user_question> </user_question>
    **IMPORTANT**: - If there is any conflict between custom criteria and user question requirements, prioritize custom criteria

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

<CUSTOM_CRITERIA_DESCRIPTION>
**(4) Faithfulness to Context:**
examines whether the candidate answer is faithful to the task description and context provided in the user question. A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context (like in a summarization task). If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable. Evaluate the degree of faithfulness on the following scale:
- **3 points**: All content is faithful (no contradictions, fully grounded when required)
- **2 points**: Most content is faithful (minor deviations or unverifiable claims when context-grounding is required)
- **1 point**: Some content is faithful or approximately half is faithful (notable contradictions or significant departures from context when required)
- **0 points**: None or minimal content is faithful (clear contradictions or complete disregard of context when grounding is required)
</CUSTOM_CRITERIA_DESCRIPTION>

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.25, Answer Completeness: 0.25, Expression Quality: 0.25, Faithfulness to Context: 0.25) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.3-0.5)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.3-0.5)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.3-0.5)**: Creative tasks, presentation-focused questions, format-specific requirements
- [*IMPORTANT*] **High Custom Criteria Weight**: The custom criteria (Faithfulness to Context) should always be *prioritized*. Assign it significant weight (0.3-0.6) and adjust other weights accordingly.

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E) + (Faithfulness_to_Context x Weight_F)

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:"""

사용자 지정 Lambda 평가자

자체 채점 함수를 Lambda 함수로 가져옵니다.

구성

에서 사용해야 하는 각 프롬프트 템플릿에 대해 입력 JSONL 파일에 Lambda ARN을 지정합니다. 또한 필드에 지표 이름을 지정합니다customEvaluationMetricLabel.

"evaluationMetricLambdaArn": "arn:aws:lambda:us-west-2:123456789012:function:my-eval-function",
"customEvaluationMetricLabel": "My Custom Metric"

API를 통해 작업을 생성할 때 CreateAdvancedPromptOptimizationJob 요청 자체에 추가 평가 구성이 필요하지 않습니다. 평가 방법은 입력 JSONL 파일에서 템플릿별로 결정됩니다.

Lambda 요구 사항

Lambda 지표 팁

Lambda 템플릿

"""
APO Custom Metric Lambda - Minimal Template
Handler: lambda_function.lambda_handler
"""
import logging
from typing import List, Dict, Any

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def compute_score(preds: List[str], golds: List[str]) -> Dict[str, Any]:
    """
    Score predictions against ground truths.

    Args:
        preds: Model outputs (one per sample)
        golds: Expected answers (one per sample)

    Returns:
        Must contain:
          "score": float  - aggregate score (higher is better)
          "scores": list[float] - per-instance scores
    """
    # --- REPLACE THIS with your scoring logic ---
    scores = []
    for pred, gold in zip(preds, golds):
        # Example: exact match (case-insensitive)
        scores.append(1.0 if pred.strip().lower() == gold.strip().lower() else 0.0)

    return {
        "score": sum(scores) / len(scores) if scores else 0.0,
        "scores": scores,
    }


def lambda_handler(event, context):
    """
    Lambda entry point. APO service sends:
      event = {"preds": ["output1", ...], "golds": ["truth1", ...]}
    """
    logger.info(f"Received {len(event.get('preds', []))} predictions")

    try:
        preds = event.get("preds", [])
        golds = event.get("golds", [])

        if not preds:
            return {"score": 0.0, "scores": []}

        return compute_score(preds, golds)

    except Exception as e:
        logger.error(f"Error: {e}", exc_info=True)
        return {"score": 0.0, "scores": [0.0] * len(event.get("preds", [])), "error": str(e)}

오류 처리 및 Lambda 함수 입력 검증을 위한 표준 문안 코드를 포함한 자세한 예제는 AWS 샘플 GitHub를 참조하세요.