# Configure CloudWatch investigations
Configuring CloudWatch investigations creates an investigation group that CloudWatch investigations will use to access telemetry and aggregate data related to the investigation. Before you create the investigation group, you can walk-through a sample investigation to get an overall idea of how they work.
**Topics**
+ [See a sample investigation](Investigations-sample.md)
+ [Set up an investigation group](Investigations-GetStarted-Group.md)
+ [Configure alarms to create investigations](Investigations-configure-alarms.md)
+ [Integration with Amazon EKS](EKS-Integration.md)
# See a sample investigation
If you'd like to see an investigation in action before you configure an investigation group for your account, you can walk through a sample investigation. The sample investigation doesn't use your data and doesn't make data calls or start API operations in your account.
**To view the sample investigation**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the left navigation pane, choose **AI Operations**, **Overview**.
1. Choose **Try a sample investigation**.
The console displays the sample investigation, with suggestions and findings in the right pane. In each popup, choose **Next** to advance to the next part of the sample walkthrough.
# Set up an investigation group
To set up CloudWatch investigations in your account for use with an enhanced investigation, you create an *investigation group*. Creating an investigation group is a one-time setup task, after it's created it is used to conduct other investigations. Settings in the investigation group help you centrally manage the common properties of your investigations, such as the following:
+ Who can access the investigations
+ Cross-account investigation support to access resources in other accounts during the investigation
+ Whether investigation data is encrypted with a customer managed AWS Key Management Service key.
+ How long investigations and their data are retained by default.
You can have one investigation group per account. Each investigation in your account will be part of this investigation group.
To create an investigation group you must be signed in to an IAM principal that has the either the **AIOpsConsoleAdminPolicy** or the **AdministratorAccess** IAM policy attached, or to an account that has similar permissions.
**Note**
To be able to choose the recommended option of creating a new IAM role for CloudWatch investigations operational investigations, you must be signed in to an IAM principal that has the `iam:CreateRole`, `iam:AttachRolePolicy`, and `iam:PutRolePolicy` permissions.
**Important**
CloudWatch investigations uses *cross-Region inference* to distribute traffic across different AWS Regions. For more information, see [Cross-Region inference](Investigations-Security.md#cross-region-inference).
**To create an investigation group in your account**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the left navigation pane, choose **AI Operations**, **Configuration**.
1. Choose **Configure for this account**.
1. Optionally change the retention period for investigations. For more information about what the retention period governs, see [CloudWatch investigations data retention](Investigations-Retention.md).
1. (Optional) To encrypt your investigation data with a customer managed AWS KMS key, choose **Customize encryption settings** and follow the steps to create or specify a key to use. If you don't specify a customer managed key, CloudWatch investigations uses an AWS owned key for encryption. For more information, see [Encryption of investigation data](Investigations-Security.md#Investigations-KMS).
1. Choose how to give CloudWatch investigations permissions to access resources. To be able to choose either of the first two options, you must be signed in to an IAM principal that has the `iam:CreateRole`, `iam:AttachRolePolicy`, and `iam:PutRolePolicy` permissions.
1. (Recommended) Select **Auto-create a new role with default investigation permissions**. This role will be granted permissions using the AWS managed policies for AI Operations.For more information, see [User permissions for your CloudWatch investigations group](Investigations-Security.md#Investigations-Security-IAM).
1. Create a new role yourself and then assign the policy templates.
1. Choose **Assign an existing role** if you already have a role with the permissions that you want to use.
If you choose this option, you must make sure the role includes a trust policy that names `aiops.amazonaws.com` as the service principal. For more information about using service principals in trust policies, see [AWS service principals](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html#principal-services).
We also recommend that you include a `Condition` section with the account number, to prevent a confused deputy situation. The following example trust policy illustrates both the service principal and the `Condition` section.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "aiops.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "123456789012"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:aiops:us-east-1:123456789012:*"
}
}
}
]
}
```
------
1. Choose **Create investigation group**, you can now create an investigation from an alarm, metric, or log insight.
Optionally, you can setup additional recommended configurations to enhance your experience.
1. In the left navigation pane, choose **AI Operations, Configuration**.
1. On the **Optional configuration** tab, choose the enhancements you want to add to CloudWatch investigations.
1. In **Configure cross account access** you can set this account as a monitoring account that collects data from other source accounts in your organization. For more information, see [Cross-account investigations](Investigations-cross-account.md).
1. For **Enhanced integrations**, choose to allow CloudWatch investigations access to additional services in your system, to enable it to gather more data and be more useful.
1. In the **Tags for application boundary detection** section, enter the existing custom tag keys for custom applications in your system. Resource tags help CloudWatch investigations narrow the search space when it is unable to discover definite relationships between resources. For example, to discover that an Amazon ECS service depends on an Amazon RDS database, CloudWatch investigations can discover this relationship using data sources such as X-Ray and CloudWatch Application Signals. However, if you haven't deployed these features, CloudWatch investigations will attempt to identify possible relationships. Tag boundaries can be used to narrow the resources that will be discovered by CloudWatch investigations in these cases.
You don't need to enter tags created by myApplications or CloudFormation, because CloudWatch investigations can automatically detect those tags.
1. CloudTrail records events about changes in your system including deployment events. These events can often be useful to CloudWatch investigations to create hypotheses about root causes of issues in your system. In the **CloudTrail for change event detection** section, you can give CloudWatch investigations some access to the events logged by AWS CloudTrail by enabling **Allow the assistant access to CloudTrail change events through the CloudTrail Event history**. For more information, see [Working with CloudTrail Event history](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html).
1. The **X-Ray for topology mapping** and **Application Signals for health assessment** sections point out other AWS services that can help CloudWatch investigations find information. If you have deployed them and you have granted the **AIOpsAssistantPolicy** IAM policy to CloudWatch investigations, it will be able to access X-Ray and Application Signals telemetry.
For more information about how these services help CloudWatch investigations, see [X-Ray](Investigations-RecommendedServices.md#Investigations-Xray) and [CloudWatch Application Signals](Investigations-RecommendedServices.md#Investigations-ApplicationSignals).
1. If you use Amazon EKS, your CloudWatch investigations investigation group can utilize information directly from your Amazon EKS cluster once you set up access entries. For more information, see [Integration with Amazon EKS](EKS-Integration.md).
1. If you use Amazon RDS, enable the Advanced mode of Database Insights on your database instances. Database Insights monitors database load and provides detailed performance analysis that helps CloudWatch investigations identify database-related issues during investigations. When Advanced Database Insights is enabled, CloudWatch investigations can automatically generate performance analysis reports that include detailed observations, metric anomalies, root cause analysis, and recommendations specific to your database workload. For more information about Database Insights and how to enable Advanced mode, see [Monitoring Amazon RDS databases with CloudWatch Database Insights](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_DatabaseInsights.html).
1. You can integrate CloudWatch investigations with a *chat channel*. This makes it possible to receive notifications about an investigation through the chat channel. CloudWatch investigations support chat channels in the following applications:
+ Slack
+ Microsoft Teams
If you want to integrate with a chat channel, we recommend that you complete some additional steps before enabling this enhancement in your investigation group. For more information, see [Integration with third-party chat systems](Investigations-Integrations.md#Investigations-Integrations-Chat).
Then, perform the following steps to integrate with a chat channel in chat applications:
+ In the **Chat client integration** section, choose **Select SNS topic**.
+ Select the SNS topic to use for sending notifications about your investigations.
# Configure alarms to create investigations
You can configure an existing CloudWatch alarm to automatically create investigations in CloudWatch investigations. When the alarm enters the ALARM state, CloudWatch automatically creates a new investigation or adds to an existing investigation based on the deduplication string.
When configuring an alarm to automatically create investigations, you'll need to specify an Amazon Resource Name (ARN) in the alarm's actionArns. This ARN identifies the investigation group where alarm-triggered investigations will be created. You can optionally include a deduplication string in the ARN to group related alarms.
## ARN format and parameters
The ARN pattern for investigation group alarm actions follows this format:
```
arn:aws:aiops:region:account-id:investigation-group/investigation-group-identifier#DEDUPE_STRING=value
```
The following table describes each ARN component:
| Parameter | Description |
| --- | --- |
| region (required) | The AWS Region where your investigation group is located. For example: us-east-1. |
| account-id (required) | Your 12-digit AWS account ID. For example: 123456789012. |
| investigation-group-identifier (required) | The unique identifier of your investigation group. Fore example, sMwwg1IogXdvL7UZ |
| DEDUPE\$1STRING=value (optional) | A deduplication string that groups related alarms into the same investigation. When multiple alarms use the same deduplication string, they contribute to a single investigation instead of creating separate ones. |
**Example without deduplication string:**
```
arn:aws:aiops:us-east-1:123456789012:investigation-group/sMwwg1IogXdvL7UZ
```
**Example with deduplication string:**
```
arn:aws:aiops:us-east-1:123456789012:investigation-group/sMwwg1IogXdvL7UZ#DEDUPE_STRING=performance
```
### Benefits of deduplication strings
Deduplication strings help you organize related alarms and reduce investigation fragmentation. Use deduplication strings when:
+ **Multiple alarms monitor the same system** - CPU, memory, and disk alarms for the same EC2 instance can share a deduplication string to create one comprehensive investigation.
+ **Cascading failures occur** - When one issue triggers multiple related alarms, the same deduplication string prevents creating separate investigations for each symptom.
+ **You want to categorize by problem type** - Use descriptive strings like "performance", "connectivity", or "security" to group alarms by issue category.
Effective deduplication string examples:
+ `DEDUPE_STRING=webserver-performance` - Groups performance-related alarms for web servers
+ `DEDUPE_STRING=database-connectivity` - Groups database connection issues
+ `DEDUPE_STRING=instance-i-1234567890abcdef0` - Groups all alarms for a specific EC2 instance
**Note**
If no deduplication string is specified, the system uses a default combination of alarm name, account ID, and region to group investigations.
For more information about investigation groups, see [Set up an investigation group](Investigations-GetStarted-Group.md).
# Configure an alarm to create investigations
After you have an investigation group set up in your account, you can configure existing CloudWatch alarms to automatically create investigations when they enter the ALARM state. This eliminates the need to manually start investigations and ensures consistent response to operational issues. You can configure alarms using the AWS Management Console, AWS CLI, CloudFormation, or AWS SDKs.
------
#### [ Console ]
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Alarms**, and select an existing alarm.
1. Choose **Actions**, **Edit**.
1. In the **Alarm actions** section, choose **Add alarm action**.
1. Under the **Configure actions**, **Investigation action** section, choose the investigation group ARN.
1. (Optional) Add a deduplication string to group related alarms.
1. Choose **Update alarm**.
------
#### [ CLI ]
This command requires that you specify an ARN for the `alarm-actions` parameter. For information about how to create the ARN, see [ARN format and parameters](Investigations-configure-alarms.md#Investigations-arn-format).
**To configure a CloudWatch alarm with InvestigationGroup action (AWS CLI)**
1. Install and configure the AWS CLI, if you haven't already. For information, see [Installing or updating the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
1. Run the following command to collect information about the alarm that you want to configure.
```
aws cloudwatch describe-alarms --alarm-names "alarm name"
```
1. Run the following command to update an alarm. Replace each *example resource placeholder* with your own information.
```
aws cloudwatch put-metric-alarm --alarm-name name \
--alarm-description "description" \
--metric-name name --namespace namespace \
--statistic statistic --period value --threshold value \
--comparison-operator value \
--dimensions "dimensions" --evaluation-periods value \
--alarm-actions "arn:aws:aiops:region:{account-id}:investigation-group/{investigationGroupIdentifier}#DEDUPE_STRING={my-dedupe-string}"
```
Here's an example.
```
//Without deduplication string
aws cloudwatch put-metric-alarm --alarm-name cpu-mon \
--alarm-description "Alarm when CPU exceeds 70 percent" \
--metric-name CPUUtilization --namespace AWS/EC2 \
--statistic Average --period 300 --threshold 70 \
--comparison-operator GreaterThanThreshold \
--dimensions "Name=InstanceId,Value=i-12345678" --evaluation-periods 2 \
--alarm-actions arn:aws:aiops:us-east-1:123456789012:investigation-group/sMwwg1IogXdvL7UZ \
--unit Percent
//With deduplication string
aws cloudwatch put-metric-alarm --alarm-name cpu-mon \
--alarm-description "Alarm when CPU exceeds 70 percent" \
--metric-name CPUUtilization --namespace AWS/EC2 \
--statistic Average --period 300 --threshold 70 \
--comparison-operator GreaterThanThreshold \
--dimensions "Name=InstanceId,Value=i-12345678" --evaluation-periods 2 \
--alarm-actions arn:aws:aiops:us-east-1:123456789012:investigation-group/sMwwg1IogXdvL7UZ#DEDUPE_STRING=performance \
--unit Percent
```
------
#### [ CloudFormation ]
This section includes CloudFormation templates that you can use to configure CloudWatch alarms to automatically create or update investigations. Each template requires that you specify an ARN for the `AlarmActions` parameter. For information about how to create the ARN, see [ARN format and parameters](Investigations-configure-alarms.md#Investigations-arn-format).
```
//Without deduplication string
Resources:
MyAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmActions:
- !Sub "arn:aws:aiops:${AWS::Region}:${AWS::AccountId}:investigation-group/{investigationGroupIdentifier}"
//With deduplication string
Resources:
MyAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmActions:
- !Sub "arn:aws:aiops:${AWS::Region}:${AWS::AccountId}:investigation-group/{investigationGroupIdentifier}#DEDUPE_STRING={my-dedupe-string}"
```
------
#### [ SDK ]
This section includes Java code snippets that you can use to configure CloudWatch alarms to automatically create or update investigations. Each snippet requires that you specify an ARN for the `investigationGroupArn` parameter. For information about how to create the ARN, see [ARN format and parameters](Investigations-configure-alarms.md#Investigations-arn-format).
```
import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
import com.amazonaws.services.cloudwatch.AmazonCloudWatchClientBuilder;
import com.amazonaws.services.cloudwatch.model.ComparisonOperator;
import com.amazonaws.services.cloudwatch.model.Dimension;
import com.amazonaws.services.cloudwatch.model.PutMetricAlarmRequest;
import com.amazonaws.services.cloudwatch.model.PutMetricAlarmResult;
import com.amazonaws.services.cloudwatch.model.StandardUnit;
import com.amazonaws.services.cloudwatch.model.Statistic;
//Without deduplication string
private void putMetricAlarmWithCloudWatchInvestigationAction() {
final AmazonCloudWatch cloudWatchClient =
AmazonCloudWatchClientBuilder.defaultClient();
Dimension dimension = new Dimension()
.withName("InstanceId")
.withValue("i-12345678");
String investigationGroupArn = "arn:aws:aiops:us-east-1:123456789012:investigation-group/sMwwg1IogXdvL7UZ";
PutMetricAlarmRequest request = new PutMetricAlarmRequest()
.withAlarmName("cpu-mon")
.withComparisonOperator(
ComparisonOperator.GreaterThanThreshold)
.withEvaluationPeriods(2)
.withMetricName("CPUUtilization")
.withNamespace("AWS/EC2")
.withPeriod(300)
.withStatistic(Statistic.Average)
.withThreshold(70.0)
.withActionsEnabled(true)
.withAlarmDescription("Alarm when CPU exceeds 70 percent")
.withUnit(StandardUnit.Percent)
.withDimensions(dimension)
.withAlarmActions(investigationGroupArn);
PutMetricAlarmResult response = cloudWatchClient.putMetricAlarm(request);
}
//With deduplication string
private void putMetricAlarmWithCloudWatchInvestigationActionWithDedupeString() {
final AmazonCloudWatch cloudWatchClient =
AmazonCloudWatchClientBuilder.defaultClient();
Dimension dimension = new Dimension()
.withName("InstanceId")
.withValue("i-12345678");
String investigationGroupArn = "arn:aws:aiops:us-east-1:123456789012:investigation-group/sMwwg1IogXdvL7UZ#DEDUPE_STRING=performance";
PutMetricAlarmRequest request = new PutMetricAlarmRequest()
.withAlarmName("cpu-mon")
.withComparisonOperator(
ComparisonOperator.GreaterThanThreshold)
.withEvaluationPeriods(2)
.withMetricName("CPUUtilization")
.withNamespace("AWS/EC2")
.withPeriod(300)
.withStatistic(Statistic.Average)
.withThreshold(70.0)
.withActionsEnabled(true)
.withAlarmDescription("Alarm when CPU exceeds 70 percent")
.withUnit(StandardUnit.Percent)
.withDimensions(dimension)
.withAlarmActions(investigationGroupArn);
PutMetricAlarmResult response = cloudWatchClient.putMetricAlarm(request);
}
```
------
# Integration with Amazon EKS
CloudWatch investigations investigation groups can utilize information directly from your Amazon EKS cluster. To get started, first grant access to the `Investigation Group` IAM role. We recommend using the default AWS managed *access policy* `AmazonAIOpsAssistantPolicy` that grants CloudWatch investigations investigation groups access to resources in the cluster. By using this policy you will automatically get policy updates as needed.
**Note**
`AmazonAIOpsAssistantPolicy` is an access policy. The AWS managed identity policy that authorizes the access associated with CloudWatch investigations investigation groups is [https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AIOpsAssistantPolicy.html](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AIOpsAssistantPolicy.html).
Use the **Advanced Configuration** option to scope down the access provided by the access policy to a set of namespaces or the entire cluster. Alternatively, you can further scope access down by associating the access entry to a Kubernetes group RBAC permission. For more information, see [Creating access entries](https://docs.aws.amazon.com/eks/latest/userguide/creating-access-entries.html).
## Configuring the Amazon EKS access entry (Console)
To associate the `AmazonAIOpsAssistantPolicy` to the investigation role using the AWS Management Console, follow these steps:
1. Open the CloudWatch console and navigate to the Investigations Configuration page.
1. In the Amazon EKS Access section, select the option to associate the `AmazonAIOpsAssistantPolicy` with your investigation role.
1. Review the policy details and confirm the association.
To further customize the access scope:
1. Click **Advanced Configuration** in the Amazon EKS Access section.
1. You will be redirected to the Amazon EKS console.
1. In the Amazon EKS console, you can:
1. Scope the policy to specific namespaces
1. Configure the group feature for more granular access control
## Configuring Amazon EKS Access Entries (CDK)
To configure Amazon EKS Access Entries using the AWS CDK, use the following code example:
```
const testAccessEntry = new AccessEntry(this, `test-access-entry`, {
cluster: eksCluster,
principal: investigationsIamRole.roleArn,
accessPolicies: [
AccessPolicy.fromAccessPolicyName('AmazonAIOpsAssistantPolicy', {
accessScopeType: AccessScopeType.CLUSTER
}),
],
});
```
## AmazonAIOpsAssistantPolicy
The Amazon EKS Access Policy, `AmazonAIOpsAssistantPolicy`, provides comprehensive Read Only access to resources in the cluster. Information from each resource may not be currently utilized by CloudWatch investigations.
```
- apiGroups: [""]
resources:
- pods
- pods/log
- services
- nodes
- namespaces
- events
- persistentvolumes
- persistentvolumeclaims
- configmaps
verbs:
- get
- list
- apiGroups: ["apps"]
resources:
- deployments
- replicasets
- statefulsets
- daemonsets
verbs:
- get
- list
- apiGroups: ["batch"]
resources:
- jobs
- cronjobs
verbs:
- get
- list
- apiGroups: ["events.k8s.io"]
resources:
- events
verbs:
- get
- list
- apiGroups: ["networking.k8s.io"]
resources:
- ingresses
- ingressclasses
verbs:
- get
- list
- apiGroups: ["storage.k8s.io"]
resources:
- storageclasses
verbs:
- get
- list
- apiGroups: ["metrics.k8s.io"]
resources:
- pods
- nodes
verbs:
- get
- list
```
## Updates to AmazonAIOpsAssistantPolicy
| Change | Description | Date |
| --- | --- | --- |
| Add policy for CloudWatch investigations | Initial release of AmazonAIOpsAssistantPolicy | August 9, 2025 |