本文為英文版的機器翻譯版本，如內容有任何歧義或不一致之處，概以英文版為準。 # 對叢集部署問題進行故障診斷 如果您的叢集無法建立並復原堆疊建立，您可以查看日誌檔案來診斷問題。失敗訊息可能看起來像下列輸出： ``` $ pcluster create-cluster --cluster-name mycluster --region eu-west-1 \ --cluster-configuration cluster-config.yaml { "cluster": { "clusterName": "mycluster", "cloudformationStackStatus": "CREATE_IN_PROGRESS", "cloudformationStackArn": "arn:aws:cloudformation:eu-west-1:xxx:stack/mycluster/1bf6e7c0-0f01-11ec-a3b9-024fcc6f3387", "region": "eu-west-1", "version": "3.15.0", "clusterStatus": "CREATE_IN_PROGRESS" } } $ pcluster describe-cluster --cluster-name mycluster --region eu-west-1 { "creationTime": "2021-09-06T11:03:47.696Z", ... "cloudFormationStackStatus": "ROLLBACK_IN_PROGRESS", "clusterName": "mycluster", "computeFleetStatus": "UNKNOWN", "cloudformationStackArn": "arn:aws:cloudformation:eu-west-1:xxx:stack/mycluster/1bf6e7c0-0f01-11ec-a3b9-024fcc6f3387", "lastUpdatedTime": "2021-09-06T11:03:47.696Z", "region": "eu-west-1", "clusterStatus": "CREATE_FAILED" } ``` **Topics** + [在 上檢視 CloudFormation 事件 `CREATE_FAILED`](#troubleshooting-v3-cluster-deployment-events) + [使用 CLI 來檢視日誌串流](#troubleshooting-v3-cluster-deployment-cli-logstreams) + [使用 重新建立失敗的叢集 `rollback-on-failure`](#troubleshooting-v3-cluster-deployment-cli-fail-rollback) ## 在 上檢視 CloudFormation 事件 `CREATE_FAILED` 您可以使用 主控台或 AWS ParallelCluster CLI 檢視`CREATE_FAILED`錯誤時的 CloudFormation 事件，以協助尋找根本原因。 **Topics** + [在 CloudFormation 主控台中檢視事件](#troubleshooting-v3-cluster-deployment-cloudformation) + [使用 CLI 在 上檢視和篩選 CloudFormation 事件 `CREATE_FAILED`](#troubleshooting-v3-cluster-deployment-cli-events) ### 在 CloudFormation 主控台中檢視事件 若要查看導致`"CREATE_FAILED"`狀態的原因的詳細資訊，您可以使用 CloudFormation 主控台。 **從主控台檢視 CloudFormation 錯誤訊息。** 1. 登入 AWS 管理主控台 並導覽至 https：//[https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/)。 1. 選取名為 *cluster\$1name* 的堆疊。 1. 選擇**事件**索引標籤。 1. 透過**按邏輯 ID** 捲動資源事件清單，檢查無法建立之資源**的狀態**。如果子任務無法建立，請向後工作以尋找失敗的資源事件。 1. 例如，如果您看到下列狀態訊息，您必須使用不超過目前 vCPU 限制或請求更多 vCPU 容量的執行個體類型。 ``` 2022-02-04 16:09:44 UTC-0800 HeadNode CREATE_FAILED You have requested more vCPU capacity than your current vCPU limit of 0 allows for the instance bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit. (Service: AmazonEC2; Status Code: 400; Error Code: VcpuLimitExceeded; Request ID: a9876543-b321-c765-d432-dcba98766789; Proxy: null). ``` ### 使用 CLI 在 上檢視和篩選 CloudFormation 事件 `CREATE_FAILED` 若要診斷叢集建立問題，您可以透過篩選`CREATE_FAILED`狀態來使用 [`pcluster get-cluster-stack-events`](pcluster.get-cluster-stack-events-v3.md)命令。如需詳細資訊，請參閱*AWS Command Line Interface 《 使用者指南*》中的[篩選 AWS CLI 輸出](https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html)。 ``` $ pcluster get-cluster-stack-events --cluster-name mycluster --region eu-west-1 \ --query 'events[?resourceStatus==`CREATE_FAILED`]' [ { "eventId": "3ccdedd0-0f03-11ec-8c06-02c352fe2ef9", "physicalResourceId": "arn:aws:cloudformation:eu-west-1:xxx:stack/mycluster/1bf6e7c0-0f02-11ec-a3b9-024fcc6f3387", "resourceStatus": "CREATE_FAILED", "resourceStatusReason": "The following resource(s) failed to create: [HeadNode]. ", "stackId": "arn:aws:cloudformation:eu-west-1:xxx:stack/mycluster/1bf6e7c0-0f02-11ec-a3b9-024fcc6f3387", "stackName": "mycluster", "logicalResourceId": "mycluster", "resourceType": "AWS::CloudFormation::Stack", "timestamp": "2021-09-06T11:11:51.780Z" }, { "eventId": "HeadNode-CREATE_FAILED-2021-09-06T11:11:50.127Z", "physicalResourceId": "i-04e91cc1f4ea796fe", "resourceStatus": "CREATE_FAILED", "resourceStatusReason": "Received FAILURE signal with UniqueId i-04e91cc1f4ea796fe", "resourceProperties": "{\"LaunchTemplate\":{\"Version\":\"1\",\"LaunchTemplateId\":\"lt-057d2b1e687f05a62\"}}", "stackId": "arn:aws:cloudformation:eu-west-1:xxx:stack/mycluster/1bf6e7c0-0f02-11ec-a3b9-024fcc6f3387", "stackName": "mycluster", "logicalResourceId": "HeadNode", "resourceType": "AWS::EC2::Instance", "timestamp": "2021-09-06T11:11:50.127Z" } ] ``` 在先前的範例中，失敗是在前端節點設定中。 ## 使用 CLI 來檢視日誌串流 若要偵錯這類問題，您可以[`pcluster list-cluster-log-streams`](pcluster.list-cluster-log-streams-v3.md)篩選 ，`node-type`然後分析日誌串流內容，以使用 列出前端節點可用的日誌串流。 ``` $ pcluster list-cluster-log-streams --cluster-name mycluster --region eu-west-1 \ --filters 'Name=node-type,Values=HeadNode' { "logStreams": [ { "logStreamArn": "arn:aws:logs:eu-west-1:xxx:log-group:/aws/parallelcluster/mycluster-202109061103:log-stream:ip-10-0-0-13.i-04e91cc1f4ea796fe.cfn-init", "logStreamName": "ip-10-0-0-13.i-04e91cc1f4ea796fe.cfn-init", ... }, { "logStreamArn": "arn:aws:logs:eu-west-1:xxx:log-group:/aws/parallelcluster/mycluster-202109061103:log-stream:ip-10-0-0-13.i-04e91cc1f4ea796fe.chef-client", "logStreamName": "ip-10-0-0-13.i-04e91cc1f4ea796fe.chef-client", ... }, { "logStreamArn": "arn:aws:logs:eu-west-1:xxx:log-group:/aws/parallelcluster/mycluster-202109061103:log-stream:ip-10-0-0-13.i-04e91cc1f4ea796fe.cloud-init", "logStreamName": "ip-10-0-0-13.i-04e91cc1f4ea796fe.cloud-init", ... }, ... ] } ``` 您可以用來尋找初始化錯誤的兩個主要日誌串流如下： + `cfn-init` 是`cfn-init`指令碼的日誌。首先檢查此日誌串流。您可能會在此日誌中看到`Command chef failed`錯誤。查看此行前面的行，以取得與錯誤訊息連線的更多詳細資訊。如需詳細資訊，請參閱 [cfn-init](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.html)。 + `cloud-init` 是 [cloud-init](https://cloudinit.readthedocs.io/) 的日誌。如果您在 中沒有看到任何內容`cfn-init`，請嘗試接下來檢查此日誌。 您可以使用 擷取日誌串流的內容 [`pcluster get-cluster-log-events`](pcluster.get-cluster-log-events-v3.md)（請注意限制擷取事件數量`--limit 5`的選項）： ``` $ pcluster get-cluster-log-events --cluster-name mycluster \ --region eu-west-1 --log-stream-name ip-10-0-0-13.i-04e91cc1f4ea796fe.cfn-init \ --limit 5 { "nextToken": "f/36370880979637159565202782352491087067973952362220945409/s", "prevToken": "b/36370880752972385367337528725601470541902663176996585497/s", "events": [ { "message": "2021-09-06 11:11:39,049 [ERROR] Unhandled exception during build: Command runpostinstall failed", "timestamp": "2021-09-06T11:11:39.049Z" }, { "message": "Traceback (most recent call last):\n File \"/opt/aws/bin/cfn-init\", line 176, in \n worklog.build(metadata, configSets)\n File \"/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py\", line 135, in build\n Contractor(metadata).build(configSets, self)\n File \"/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py\", line 561, in build\n self.run_config(config, worklog)\n File \"/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py\", line 573, in run_config\n CloudFormationCarpenter(config, self._auth_config).build(worklog)\n File \"/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py\", line 273, in build\n self._config.commands)\n File \"/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py\", line 127, in apply\n raise ToolError(u\"Command %s failed\" % name)", "timestamp": "2021-09-06T11:11:39.049Z" }, { "message": "cfnbootstrap.construction_errors.ToolError: Command runpostinstall failed", "timestamp": "2021-09-06T11:11:39.049Z" }, { "message": "2021-09-06 11:11:49,212 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.eu-west-1.amazonaws.com", "timestamp": "2021-09-06T11:11:49.212Z" }, { "message": "2021-09-06 11:11:49,213 [DEBUG] Signaling resource HeadNode in stack mycluster with unique ID i-04e91cc1f4ea796fe and status FAILURE", "timestamp": "2021-09-06T11:11:49.213Z" } ] } ``` 在先前的範例中，失敗是由`runpostinstall`失敗所造成，因此與 組態`OnNodeConfigured`參數中使用的自訂引導指令碼內容嚴格相關[`CustomActions`](HeadNode-v3.md#HeadNode-v3-CustomActions)。 ## 使用 重新建立失敗的叢集 `rollback-on-failure` AWS ParallelCluster 在日誌群組中建立叢集 CloudWatch 日誌串流。您可以在 CloudWatch 主控台**自訂儀表板**或日誌**群組中檢視這些日誌**。如需詳細資訊，請參閱[與 Amazon CloudWatch Logs 的整合](cloudwatch-logs-v3.md)及[Amazon CloudWatch 儀表板](cloudwatch-dashboard-v3.md)。如果沒有可用的日誌串流，則失敗可能是由[`CustomActions`](HeadNode-v3.md#HeadNode-v3-CustomActions)自訂引導指令碼或 AMI 相關問題所造成。若要診斷在此情況下的建立問題，請使用 再次建立叢集[`pcluster create-cluster`](pcluster.create-cluster-v3.md)，包括將 `--rollback-on-failure` 參數設定為 `false`。然後，使用 SSH 檢視叢集，如下所示： ``` $ pcluster create-cluster --cluster-name mycluster --region eu-west-1 \ --cluster-configuration cluster-config.yaml --rollback-on-failure false { "cluster": { "clusterName": "mycluster", "cloudformationStackStatus": "CREATE_IN_PROGRESS", "cloudformationStackArn": "arn:aws:cloudformation:eu-west-1:xxx:stack/mycluster/1bf6e7c0-0f01-11ec-a3b9-024fcc6f3387", "region": "eu-west-1", "version": "3.15.0", "clusterStatus": "CREATE_IN_PROGRESS" } } $ pcluster ssh --cluster-name mycluster ``` 登入前端節點後，您應該會找到三個主要日誌檔案，可用來尋找錯誤。 + `/var/log/cfn-init.log` 是`cfn-init`指令碼的日誌。首先檢查此日誌。您可能會在此日誌`Command chef failed`中看到錯誤，例如 。查看此行前面的行，以取得與錯誤訊息連線的更多詳細資訊。如需詳細資訊，請參閱 [cfn-init](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.html)。 + `/var/log/cloud-init.log` 是 [cloud-init](https://cloudinit.readthedocs.io/) 的日誌。如果您在 中沒有看到任何內容`cfn-init.log`，請嘗試接下來檢查此日誌。 + `/var/log/cloud-init-output.log` 是 [cloud-init](https://cloudinit.readthedocs.io/) 所執行命令的輸出。這包括來自 的輸出`cfn-init`。在大多數情況下，您不需要查看此日誌來疑難排解此類問題。