Scheduler audit logs in AWS PCS

Scheduler audit logs record Remote Procedure Call (RPC) operations processed by your cluster's Slurm controller (slurmctld) and database daemon (slurmdbd). The AUDIT_RPCS: prefix in the log message identifies these logs. They support security auditing and compliance use cases.

For clusters running Slurm 25.11 and later, AWS PCS delivers audit logs separately through the PCS_SCHEDULER_AUDIT_LOGS log type. This separation lets you control audit log ingestion and storage costs independently from your operational logs, because audit logs can make up to 90% of scheduler log volume.

Prerequisites

Before you can set up scheduler audit logs, you must meet the following requirements:

The following example IAM policy grants the required permissions.

JSON

{
   "Version":"2012-10-17",
   "Statement": [
      {
         "Sid": "PcsAllowVendedLogsDelivery",
         "Effect": "Allow",
         "Action": ["pcs:AllowVendedLogDeliveryForResource"],
         "Resource": [
            "arn:aws:pcs:*::cluster/*"
         ]
      }
   ]
}

Set up scheduler audit logs

You can set up scheduler audit logs for your AWS PCS cluster with the AWS Management Console or AWS CLI. Scheduler audit logs are opt-in. AWS PCS does not deliver them until you subscribe.

AWS Management Console

To set up scheduler audit logs with the console

1. Open the AWS PCS console.

2. In the navigation pane, choose Clusters.

3. Choose the cluster where you want to add scheduler audit logs.

4. On the cluster details page, choose the Logs tab.

5. Under Scheduler Audit Logs, choose Add to add up to 3 log delivery destinations from among CloudWatch Logs, Amazon S3, and Firehose.

6. Choose Update log deliveries.

AWS CLI

To set up scheduler audit logs with the AWS CLI

1. Create a log delivery destination:

   aws logs put-delivery-destination --region region \
     --name pcs-audit-logs-destination \
     --delivery-destination-configuration \
     destinationResourceArn=resource-arn

   Replace:

   • region — The AWS Region where you want to create the destination, such as us-east-1

   • pcs-audit-logs-destination — A name for the destination

   • resource-arn — The Amazon Resource Name (ARN) of a CloudWatch Logs log group, S3 bucket, or Firehose delivery stream.

   For more information, see PutDeliveryDestination in the Amazon CloudWatch Logs API Reference.

2. Set the PCS cluster as a log delivery source:

   aws logs put-delivery-source --region region \
     --name cluster-audit-logs-source-name \
     --resource-arn cluster-arn \
     --log-type PCS_SCHEDULER_AUDIT_LOGS

   Replace:

   • region — The AWS Region of your cluster, such as us-east-1

   • cluster-audit-logs-source-name — A name for the source

   • cluster-arn — the ARN of your AWS PCS cluster

   For more information, see PutDeliverySource in the Amazon CloudWatch Logs API Reference.

3. Connect the delivery source to the delivery destination:

   aws logs create-delivery --region region \
     --delivery-source-name cluster-audit-logs-source \
     --delivery-destination-arn destination-arn

   Replace:

   • region — The AWS Region, such as us-east-1

   • cluster-audit-logs-source — The name of your delivery source

   • destination-arn — The ARN of your delivery destination

   For more information, see CreateDelivery in the Amazon CloudWatch Logs API Reference.

Scheduler audit log stream paths and names

The path and name for AWS PCS scheduler audit logs depend on the destination type.

Example scheduler audit log record

AWS PCS scheduler audit logs are structured. They use the same schema as scheduler logs, with the log message containing the AUDIT_RPCS: prefix. Here is an example from slurmctld.

{
    "resource_id": "pcs_bu93qsds2j",
    "resource_type": "PCS_CLUSTER",
    "event_timestamp": 1774481175953,
    "log_level": "info",
    "log_name": "slurmctld",
    "scheduler_type": "slurm",
    "scheduler_major_version": "25.11",
    "scheduler_patch_version": "2",
    "node_type": "controller_primary",
    "message": "[2026-01-21T08:19:26.692+00:00] AUDIT_RPCS: [slurmctld-primary:6817(fd:18)] msg_type=REQUEST_PARTITION_INFO uid=0 client=[10.0.76.95:56918]\n"
}

Here is an example from slurmdbd.

{
    "resource_id": "pcs_bu93qsds2j",
    "resource_type": "PCS_CLUSTER",
    "event_timestamp": 1774485082772,
    "log_level": "info",
    "log_name": "slurmdbd",
    "scheduler_type": "slurm",
    "scheduler_major_version": "25.11",
    "scheduler_patch_version": "2",
    "node_type": "slurmdbd_primary",
    "message": "[2026-01-21T08:19:26.692+00:00] AUDIT_RPCS: msg_type=DBD_GET_CLUSTERS uid=0 client=[28.5.0.18:36658] protocol=11008\n"
}

Audit log behavior by Slurm version

The following table describes how audit logs are delivered depending on the Slurm version running on your cluster.