翻訳は機械翻訳により提供されています。提供された翻訳内容と英語版の間で齟齬、不一致または矛盾がある場合、英語版が優先します。

ルーブリックベースの審査員

概要:

Rubric Judge は、Nova 2.LLM-as-a-judge 評価モデルです。優先判断のみを提供する元の判事モデル (A>B、B>A、またはタイ) とは異なり、Rubric Judge は各プロンプトに合わせたカスタム評価基準を動的に生成し、複数のディメンションにわたってきめ細かなスコアを割り当てます。

主要な機能

モデルによって生成された条件の例

price_validation:  
  description: "The response includes validation to ensure price is a positive value."  
  type: "scale"  
  weight: 0.3

このモデルは、生成されたすべての基準に対して両方のレスポンスを評価し、これらの基準レベルのスコアを使用して最終的な優先決定を通知します。

レシピ設定

Rubric Judge レシピ

レシピtask: rubric_llm_judgeで を設定して Rubric Judge を有効にします。

run:  
  name: nova-eval-job-name                              # [MODIFIABLE] Unique identifier for your evaluation job  
  model_type: amazon.nova-2-lite-v1:0:256k              # [FIXED] Rubric Judge model type  
  model_name_or_path: "nova-lite-2/prod"                # [FIXED] Path to model checkpoint or identifier  
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job  
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job  
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job  
    
evaluation:  
  task: rubric_llm_judge                                # [FIXED] Evaluation task - enables Rubric Judge  
  strategy: judge                                       # [FIXED] Evaluation strategy  
  metric: all                                           # [FIXED] Metric calculation method  
    
inference:  
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate  
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter  
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter  
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

ジャッジレシピとしての元の LLM (比較用)

元の判事モデルは を使用しますtask: llm_judge。

run:  
  name: eval-job-name                                   # [MODIFIABLE] Unique identifier for your evaluation job  
  model_type: amazon.nova-micro-v1:0:128k               # [FIXED] Model type   
  model_name_or_path: "nova-micro/prod"                 # [FIXED] Path to model checkpoint or identifier  
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job  
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job  
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job  
    
evaluation:  
  task: llm_judge                                       # [FIXED] Original judge task  
  strategy: judge                                       # [FIXED] Evaluation strategy  
  metric: all                                           # [FIXED] Metric calculation method  
  
inference:  
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate  
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter  
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter  
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

入力データセット形式

入力データセットの形式は元の判事モデルと同じです。

必須フィールド

データセットの例 (JSONL 形式)

{"prompt": "What is the most effective way to combat climate change?", "response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.", "response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."}  
{"prompt": "Explain how a computer's CPU works", "response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.", "response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."}  
{"prompt": "How does photosynthesis work?", "response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.", "response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."}

フォーマット要件

評価出力

出力構造

Rubric Judge は、元の判事モデルと比較して強化された評価メトリクスを生成します。

{  
  "config_general": {  
    "lighteval_sha": "string",  
    "num_fewshot_seeds": "int",  
    "max_samples": "int | null",  
    "job_id": "int",  
    "start_time": "float",  
    "end_time": "float",  
    "total_evaluation_time_secondes": "string",  
    "model_name": "string",  
    "model_sha": "string",  
    "model_dtype": "string | null",  
    "model_size": "string"  
  },  
  "results": {  
    "custom|rubric_llm_judge_judge|0": {  
      "a_scores": "float",  
      "a_scores_stderr": "float",  
      "b_scores": "float",  
      "b_scores_stderr": "float",  
      "ties": "float",  
      "ties_stderr": "float",  
      "inference_error": "float",  
      "inference_error_stderr": "float",  
      "score": "float",  
      "score_stderr": "float",  
      "weighted_score_A": "float",  
      "weighted_score_A_stderr": "float",  
      "weighted_score_B": "float",  
      "weighted_score_B_stderr": "float",  
      "score_margin": "float",  
      "score_margin_stderr": "float",  
      "winrate": "float",  
      "lower_rate": "float",  
      "upper_rate": "float"  
    }  
  },  
  "versions": {  
    "custom|rubric_llm_judge_judge|0": "int"  
  }  
}

Rubric Judge の新しいメトリクス

次の 6 つのメトリクスは Rubric Judge に固有であり、きめ細かな品質評価を提供します。

加重スコアメトリクスについて

目的: 加重スコアは、バイナリプリファレンスの判定を補完する継続的な品質測定値を提供し、モデルのパフォーマンスをより深く洞察できるようにします。

元の判事との主な違い

score_margin の解釈

ユースケース

計算方法

加重スコアは、次のプロセスを通じて計算されます。

例: response_A の加重合計が 0.65 で response_B の加重合計が 0.78 score_marginの場合、 は -0.13 になり、すべての加重基準で response_B の品質が 13 パーセントポイント高いことを示します。