直接偏好优化（DPO）

概述

直接偏好优化（DPO）是一种对齐技术，利用成对对比数据对基础模型进行微调，使模型输出与人类偏好保持一致。与强化学习方法不同，DPO 直接基于人类反馈（即哪些响应更优）来优化模型行为，是一种更稳定、更具可扩展性的方案。

为何选择 DPO

基础模型生成的输出虽然在事实层面可能是正确的，但往往无法与特定用户需求、组织价值观或安全要求相符合。DPO 可通过以下功能解决该问题：

DPO 的工作原理

DPO 使用配对的示例，由人类评估者指出在两种可能的响应中偏好哪一种响应。该模型通过学习来最大化生成偏好响应的可能性，同时最大程度减少非预期的响应。

DPO 的适用场景

将 DPO 用于以下方案：

支持的模型与技术

DPO 同时支持全参数微调和 LoRA（低秩适配）：

训练方法

对于大多数使用案例，LoRA 方案具备足够的适配能力，且效率显著提升。

数据格式

DPO 训练数据格式与 SFT 一致，区别仅在于最后一轮助手响应必须包含带有 preferred 和 non-preferred 标签的偏好对。

基本结构

最终轮次的助手响应使用 candidates 数组而非 content 字段：

{
  "role": "assistant",
  "candidates": [
    {
      "content": [
        {
          "text": "This is the preferred response."
        }
      ],
      "preferenceLabel": "preferred"
    },
    {
      "content": [
        {
          "text": "This is the non-preferred response."
        }
      ],
      "preferenceLabel": "non-preferred"
    }
  ]
}

完整文本示例

{
  "schemaVersion": "bedrock-conversation-2024",
  "system": [
    {
      "text": "You are a helpful assistant."
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "text": "What is the capital of France?"
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "text": "The capital of France is Paris."
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "text": "Tell me more about it."
        }
      ]
    },
    {
      "role": "assistant",
      "candidates": [
        {
          "content": [
            {
              "text": "Paris is the capital and largest city of France, known for the Eiffel Tower, world-class museums like the Louvre, and its rich cultural heritage."
            }
          ],
          "preferenceLabel": "preferred"
        },
        {
          "content": [
            {
              "text": "Paris is a city in France."
            }
          ],
          "preferenceLabel": "non-preferred"
        }
      ]
    }
  ]
}

带图像示例

{
  "schemaVersion": "bedrock-conversation-2024",
  "system": [
    {
      "text": "You are a helpful assistant."
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "text": "Describe this image."
        },
        {
          "image": {
            "format": "jpeg",
            "source": {
              "s3Location": {
                "uri": "s3://your-bucket/your-path/image.jpg",
                "bucketOwner": "your-aws-account-id"
              }
            }
          }
        }
      ]
    },
    {
      "role": "assistant",
      "candidates": [
        {
          "content": [
            {
              "text": "The image shows a detailed description with relevant context and observations."
            }
          ],
          "preferenceLabel": "preferred"
        },
        {
          "content": [
            {
              "text": "This is a picture."
            }
          ],
          "preferenceLabel": "non-preferred"
        }
      ]
    }
  ]
}

数据集要求

上传数据

aws s3 cp /path/to/training-data/ s3://your-bucket/train/ --recursive
aws s3 cp /path/to/validation-data/ s3://your-bucket/val/ --recursive

配方配置

常规运行配置

run:
  name: "my-dpo-run"
  model_type: "amazon.nova-lite-v1:0:300k"
  model_name_or_path: "nova-lite/prod"
  replicas: 4

训练配置

training_config:
  max_length: 16384
  global_batch_size: 32

  trainer:
    max_epochs: 3

  model:
    hidden_dropout: 0.0
    attention_dropout: 0.0
    ffn_dropout: 0.0

优化器配置

model:
  optim:
    lr: 1e-5
    name: distributed_fused_adam
    adam_w_mode: true
    eps: 1e-08
    weight_decay: 0.0
    betas:
      - 0.9
      - 0.999
    sched:
      warmup_steps: 10
      constant_steps: 0
      min_lr: 1e-6

DPO 专属配置

model:
  dpo_cfg:
    beta: 0.1

建议：初始设置 beta: 0.1；若发现偏好学习效果不足，再向下调整。

LoRA PEFT 配置

model:
  peft:
    peft_scheme: "lora"
    lora_tuning:
      loraplus_lr_ratio: 64.0
      alpha: 32
      adapter_dropout: 0.01

启动训练作业

容器映像

708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-fine-tune-repo:SM-TJ-DPO-latest

示例代码

from sagemaker.pytorch import PyTorch
from sagemaker.inputs import TrainingInput

instance_type = "ml.p5.48xlarge"
instance_count = 4

image_uri = "708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-fine-tune-repo:SM-TJ-DPO-latest"

recipe_overrides = {
    "training_config": {
        "trainer": {"max_epochs": 2},
        "model": {
            "dpo_cfg": {"beta": 0.1},
            "peft": {
                "peft_scheme": "lora",
                "lora_tuning": {
                    "loraplus_lr_ratio": 64.0,
                    "alpha": 32,
                    "adapter_dropout": 0.01,
                },
            },
        },
    },
}

estimator = PyTorch(
    output_path=f"s3://{bucket_name}/{job_name}",
    base_job_name=job_name,
    role=role,
    instance_count=instance_count,
    instance_type=instance_type,
    training_recipe="fine-tuning/nova/nova_lite_p5_gpu_lora_dpo",
    recipe_overrides=recipe_overrides,
    max_run=18000,
    sagemaker_session=sagemaker_session,
    image_uri=image_uri,
    disable_profiler=True,
    debugger_hook_config=False,
)

train_input = TrainingInput(
    s3_data=train_dataset_s3_path,
    distribution="FullyReplicated",
    s3_data_type="Converse",
)

val_input = TrainingInput(
    s3_data=val_dataset_s3_path,
    distribution="FullyReplicated",
    s3_data_type="Converse",
)

estimator.fit(inputs={"train": train_input, "validation": val_input}, wait=True)

部署模型

训练完成后，可通过自定义模型导入功能，将自定义模型部署到 Amazon Bedrock。该模型同时支持预置吞吐量与按需推理。经 LoRA 训练的模型仅支持按需推理。

有关部署说明，请参阅部署自定义模型。

限制