Troubleshoot degraded workspaces in Amazon Managed Grafana
An Amazon Managed Grafana workspace can enter a degraded state for several reasons, including VPC configuration issues and KMS key problems. When a workspace is degraded, you may experience availability loss, inability to make configuration changes, and missed security updates. The following sections describe each degraded reason and the actions you can take to resolve it.
KMS key disabled (not recoverable)
Your workspace has failed and cannot be recovered because the KMS key used in the workspace has been disabled for more than 7 days, or the KMS grant has been revoked.
You will experience the following issues:
-
A complete availability loss for the workspace, resulting in non-functioning alerts and inaccessible dashboards
-
Inability to make configuration changes to your workspace
-
Your workspace will not be able to receive security updates or patches
-
All workspace data is permanently lost and cannot be recovered
To resolve this issue:
This workspace cannot be recovered. You must create a new workspace. For more information about encryption at rest, see Encryption at rest. For best practices on managing KMS keys, see Best practices for AWS KMS in the AWS KMS Developer Guide.
KMS key disabled (recoverable)
Your workspace is disabled and non-operational because the KMS key used for customer managed key encryption has been disabled.
Until you take action, you will experience the following issues:
-
A complete availability loss for the workspace, resulting in non-functioning alerts and inaccessible dashboards
-
Inability to make configuration changes to your workspace
-
Your workspace will not be able to receive security updates or patches
To resolve this issue:
Re-enable the KMS key and restore Amazon Managed Grafana access in the key policy. For more information, see Enabling and disabling keys in the AWS KMS Developer Guide.
Important
You must re-enable the KMS key within 7 days. After this period, the
workspace transitions to a FAILED state and cannot be
recovered.
If you revoke KMS grants created by Amazon Managed Grafana to access your KMS key, the grants cannot be recreated, and the data in the workspace is lost permanently. For more information about grants, see Grants in AWS KMS in the AWS KMS Developer Guide.
Deleted security group
Your workspace is disabled and non-operational because a security group associated with the workspace VPC configuration has been deleted.
Until you take action, you will experience the following issues:
-
A complete availability loss for the workspace, resulting in non-functioning alerts and inaccessible dashboards
-
Inability to make configuration changes to your workspace
-
Your workspace will not be able to receive security updates or patches
To resolve this issue:
-
Open the Amazon Managed Grafana console and select your workspace.
-
Update the security groups to valid security groups under the Outbound VPC connection setting.
-
Confirm the change and retry the VPC connection.
To avoid this issue in the future, update the security groups in your Amazon Managed Grafana console before deleting them from their VPC.
Deleted subnet
Your workspace is disabled and non-operational because a subnet has been deleted from your Elastic Network Interface (ENI).
Until you take action, you will experience the following issues:
-
A complete availability loss for the workspace, resulting in non-functioning alerts and inaccessible dashboards
-
Inability to make configuration changes to your workspace
-
Your workspace will not be able to receive security updates or patches
To resolve this issue:
-
Open the Amazon Managed Grafana console and select your workspace.
-
Update the subnets to valid subnets under the Outbound VPC connection setting.
-
Confirm the change and retry the VPC connection.
To avoid this issue in the future, update the subnets in your Amazon Managed Grafana console before deleting them from their VPC.
IP address exhaustion
Your workspace is experiencing availability loss because the subnets connected to your workspace do not have enough free IP addresses.
To resolve this issue:
-
Open the Amazon Managed Grafana console. In the left navigation pane, choose All workspaces, then select your workspace.
-
In the Network access control tab, under Outbound VPC connection, choose each subnet to access the Subnet Details page.
-
Verify that each subnet has at least 15 available IPv4 addresses.
-
If a subnet has fewer than 15 free IP addresses, free up IP addresses by releasing addresses associated with instances or deleting unused network interfaces.
-
If you cannot free up IP addresses, replace the subnet with one that has at least 15 free IP addresses. We recommend using dedicated subnets for Amazon Managed Grafana. For step-by-step instructions, see What should I do if I'm unable to update an Amazon Managed Grafana workspace due to insufficient IP addresses?.
We strongly recommend that you configure alarms to monitor IP usage in your VPC subnets. For more information, see Track IP addresses in the Amazon VPC IPAM Guide.
Missing DHCP option set
Your workspace is experiencing availability loss because the VPC connected to your workspace does not have a DHCP option set configured.
To resolve this issue:
-
Open the Amazon Managed Grafana console. In the left navigation pane, choose All workspaces, then select your workspace.
-
In the Network access control tab, under Outbound VPC connection, open the VPC associated with your workspace.
-
In the VPC Details, choose Actions, then choose Edit VPC settings.
-
Under DHCP settings, change the DHCP option set from No DHCP Option set to a valid option set. For more information, see DHCP option sets in the Amazon VPC User Guide.
