# Create field indexes to improve query performance and reduce scan volume
You can create *field indexes* of fields in your log events for efficient equality-based searches. When you then use a field index in a CloudWatch Logs Insights query, the query attempts to skip processing log events that are known to not include the indexed field. This reduces the scan volume of your queries that use field indexes, making it possible to return results faster. This can help you quickly search petabytes of total logs across thousands of log groups, and hone in on relevant logs faster. Good fields to index are fields that you often need to query for. Fields that have high cardinality of values are also good candidates for field indexes because a query using these field indexes will complete faster because it limits the log events that are being matched to the target value.
For example, suppose you have created a field index for `requestId`. Then, any CloudWatch Logs Insights query on that log group that includes `requestId = value` or `requestId IN [value, value, ...]` will attempt to process only the log events that are known to contain that indexed field and the queried value, and that CloudWatch Logs has detected a value for that field in the past.
You can also leverage your field indexes to create efficient queries of larger numbers of log groups. When you use the `filterIndex` command in your query instead of the `filter` command, the query will run against selected log groups on log events that have field indexes. These queries can scan as many as 10,000 log groups which you choose by specifying as many as five log group name prefixes. If this is a monitoring account in CloudWatch cross-account observability, you can choose all the source accounts or specify individual source accounts to select the log groups”.
Indexed fields are case-sensitive. For example, a field index of `RequestId` won't match a log event containing `requestId`.
Fields indexes are supported only for the structured log formats of JSON and service logs.
CloudWatch Logs provides default field indexes for all log groups in the Standard log class. Default field indexes are automatically available for the following fields:
+ `@logStream`
+ `@aws.region`
+ `@aws.account`
+ `@source.log`
+ `@data_source_name`
+ `@data_source_type`
+ `@data_format`
+ `traceId`
+ `severityText`
+ `attributes.session.id`
CloudWatch Logs provides default field indexes for certain data source name and type combinations as well. Default field indexes are automatically available for the following data source name and type combinations:
| Data Source Name and Type | Default Field Indexes |
| --- | --- |
| `amazon_vpc.flow` | `action` `logStatus` `region` `flowDirection` `type` |
| `amazon_route53.resolver_query` | `query_type` `transport` `rcode` |
| `aws_waf.access` | `action` `httpRequest.country` |
| `aws_cloudtrail.data` ` aws_cloudtrail.management` | `eventSource` `eventName` `awsRegion` `userAgent` `errorCode` `eventType` `managementEvent` `readOnly` `eventCategory` `requestId` |
Default field indexes are in addition to any custom field indexes you define within your policy. Default field indexes are not counted towards your [field index quota](CloudWatchLogs-Field-Indexing-Syntax.md).
CloudWatch Logs indexes only the log events ingested after an index policy is created. It doesn't index log events ingested before the policy was created. After you create a field index, each matching log event remains indexed for 30 days from the log event's ingestion time.
**Note**
If you create a field index policy in a monitoring account, that policy is not used for log groups in linked source accounts. A field index policy applies only in the account where it is created.
The rest of the topics in this section explain how to create field indexes. For information about referring to field indexes in your queries, see [filterIndex](CWL_QuerySyntax-FilterIndex.md) and [filter](CWL_QuerySyntax-Filter.md).
**Topics**
+ [Field index syntax and quotas](CloudWatchLogs-Field-Indexing-Syntax.md)
+ [Create an account-level field index policy](CloudWatchLogs-Field-Indexing-CreateAccountLevel.md)
+ [Create a log-group level field index policy](CloudWatchLogs-Field-Indexing-CreateLogGroupLevel.md)
+ [Log group selection options when creating a query](Field-Indexing-Selection.md)
+ [Effects of deleting a field index policy](CloudWatchLogs-Field-Indexing-Deletion.md)
# Field index syntax and quotas
You create field indexes by creating *field index policies*. You can create account-level index policies that apply to your whole account, and you can also create policies that apply to only a single log group. For account-wide index policies, you can have one that applies to all log groups in the account. You can also create account-level index policies that apply to a subset of log groups in the account, selected by the prefixes of their log group names. If you have multiple account-level policies in the same account, the log group name prefixes for these policies can't overlap. Similarly, you can create account-level index policies that apply to a specific data source name and type combination. Only one account policy can be created per data source name and type combination.
Log group-level field index policies override account-level field index policies: which apply to the log group as a whole (such as, account-level policies with no selection criteria or with log group name prefix based selection criteria). Account-level policies which match at the log event level (such as, for a given data source name and type combination) will apply in addition to policies which match the log group as a whole. If you create log-group level index policy, that log group does not use account-level policies that match at the log group level.
Matches of log events to the names of field indexes are case-sensitive. For example, a field index of `RequestId` won't match a log event containing `requestId`.
You can have as many as 40 account-level index policies, of these policies 20 can use log group name prefix selection criteria and 20 can use data source based selection criteria. If you have multiple account-level index policies filtered to log group name prefixes, no two of them can use the same or overlapping log group name prefixes. For example, if you have one policy filtered to log groups that start with `my-log`, you can't have another field index policy filtered to `my-logpprod` or `my-logging`. Similarly, if you have multiple account-level index policies filtered to data source name and type combinations, no two of them can use the same data source name and type. For example, if you have one policy filtered to the data source name `amazon_vpc` and data source type `flow` you cannot create another policy with this combination.
If you have an account-level index policy that has no name prefixes and applies to all log groups, then no other account-level index policy with log group name prefix filters can be created; you can create account-level index policies which use data source name and type filters.
Each index policy has the following quotas and restrictions:
+ As many as 20 fields can be included in the policy.
+ Each field name can include as many as 100 characters.
+ To create an index of a custom field in your log groups that starts with `@`, you must specify the field with an extra `@` at the beginning of the field name. For example, if your log events include a field named `@userId`, you must specify `@@userId` to create an index for this field.
For account-level index policies with data source name and type based selection criteria an additional restriction applies: all of the fields must be primitive data types, nested primitives are only supported for structs.
**Generated fields and reserved fields**
CloudWatch Logs Insights automatically generates system fields in each log event. These generated fields are prefixed with `@` For more information about generated fields, see [Supported logs and discovered fields](CWL_AnalyzeLogData-discoverable-fields.md).
Of these generated fields, the following are supported for use as field indexes:
+ `@logStream`
+ `@ingestionTime`
+ `@requestId`
+ `@type`
+ `@initDuration`
+ `@duration`
+ `@billedDuration`
+ `@memorySize`
+ `@maxMemoryUsed`
+ `@xrayTraceId`
+ `@xraySegmentId`
To index these generated fields, you don't need to add an extra `@` when specifying them, as you have to do for custom fields that start with `@`. For example, to create a field index for `@logStream`, just specify `@logStream` as the field index.
CloudWatch Logs provides default field indexes for all log groups in the Standard log class. Default field indexes are automatically available for the following fields:
+ `@logStream`
+ `@aws.region`
+ `@aws.account`
+ `@source.log`
+ `@data_source_name`
+ `@data_source_type`
+ `@data_format`
+ `traceId`
+ `severityText`
+ `attributes.session.id`
CloudWatch Logs provides default field indexes for certain data source name and type combinations as well. Default field indexes are automatically available for the following data source name and type combinations:
| Data Source Name and Type | Default Field Indexes |
| --- | --- |
| `amazon_vpc.flow` | `action` `logStatus` `region` `flowDirection` `type` |
| `amazon_route53.resolver_query` | `query_type` `transport` `rcode` |
| `aws_waf.access` | `action` `httpRequest.country` |
| `aws_cloudtrail.data` ` aws_cloudtrail.management` | `eventSource` `eventName` `awsRegion` `userAgent` `errorCode` `eventType` `managementEvent` `readOnly` `eventCategory` `requestId` |
Default field indexes are in addition to any custom field indexes you define within your policy. Default field indexes are not counted towards your [field index quota](#CloudWatchLogs-Field-Indexing-Syntax).
**Child fields and array fields in JSON logs**
You can index fields that are nested child fields or array fields in JSON logs.
For example, you can create an index of the `accessKeyId` child field within the `userIdentity` field within this log:
```
{
"eventVersion": "1.0",
"userIdentity": {
"type": "IAMUser",
"principalId": "EXAMPLE_PRINCIPAL_ID",
"arn": "arn: aws: iam: : 123456789012: user/Alice",
"accessKeyId": "11112222",
"accountId": "123456789012",
"userName": "Alice"
},
"eventTime": "2014-03-06T21: 22: 54Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "StartInstances",
"awsRegion": "us-east-2",
"sourceIPAddress": "192.0.2.255",
"userAgent": "ec2-api-tools1.6.12.2",
"requestParameters": {
"instancesSet": {
"items": [{
"instanceId": "i-abcde123",
"currentState": {
"code": 0,
"name": "pending"
},
"previousState": {
"code": 80,
"name": "stopped"
}
}]
}
}
}
```
To create this field, you refer to it using dot notation (`userIdentity.accessKeyId`) both when creating the field index and when specifying it in a query. The query could look like this:
```
fields @timestamp, @message
| filterIndex userIdentity.accessKeyId = "11112222"
```
In the previous example event, the `instanceId` field is in an array within `requestParameters.instancesSet.items` To represent this field both when creating the field index and when querying, refer to it as `requestParameters.instancesSet.items.0.instanceId` The 0 refers to that field's place in the array.
Then a query for this field could be the following:
```
fields @timestamp, @message
| filterIndex requestParameters.instancesSet.items.0.instanceId="i-abcde123"
```
# Create an account-level field index policy
Use the steps in this section to create a field index policy that applies to all log groups in the account, or to multiple log groups that have log group names that start with the same string.
**To create an account-level field index policy**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the left navigation pane, choose **Settings** and then choose the **Logs** tab.
1. In the **Account level index policies** section, choose **Manage**.
1. Choose **Create index policy**.
1. For **Policy name**, enter a name for your new policy.
1. For **Select scope of policy**, do one of the following:
+ Choose **All standard log groups** to have the index policy apply to all Standard Class log groups in the account.
+ Choose **Log groups by prefix match** to apply the policy to a subset of log groups that all have names that start with the same string. Then, enter the prefix for these log groups in **Enter a prefix name**.
After you enter your prefix, you can choose **Preview prefix matched log groups** to confirm that your prefix matches the log groups that you expected.
Choose **Logs Data by Data source** to apply the policy to a specific data source name and type combination. You can then select the **Data source** and **Data type** from the drop-down menu.
After you select the data source name and type you can select **Get fields** to populate the **Configure field indexes and facets** section with relevant information such as the fields available, included log groups, as well as default and custom field indexes.
1. For **Custom index field configuration**, choose **Add field path** to enter the first field to index.
Then enter the string to use as the value of the field name or select a field from the drop-down menu. This must be an exact case match to what appears in the log events. For example, if your log events include `requestId`, you must enter `requestId` here. `RequestId`, `requestID`, and `request Id` wouldn't match.
If you want to index a custom log field that starts with the `@` character, you must include an extra `@` character when you enter the index string. For example, if you have a custom log field `@emailname`, enter `@@emailname` in the **Add field path** box.
You can also create indexes for the `@ingestionTime` and `@logStream` fields that CloudWatch Logs automatically generates. If you do, you don't need to add an extra `@` when specifying them.
1. (Optional)In addition to specifying the field path, you can select **Set as a facet** to create the field as a facet.
1. Repeat the previous step to add as many as 20 field indexes.
1. When you have finished, choose **Create**.
# Create a log-group level field index policy
Use the steps in this section to create a field index policy that applies to a single log group.
**To create a log-group level field index policy**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the left navigation pane, choose **Logs**, **Log groups**.
1. Choose the name of the log group.
1. Choose the **Field indexes** tab.
1. Choose **Manage field indexes for this log group**
1. For **Manage log group level field indexes**, choose **Add field path** to enter the first field to index.
Then enter the string to use as the value of the field name. This must be an exact case match to what appears in the log events. For example, if your log events include `requestId`, you must enter `requestId` here. `RequestId`, `requestID`, and `request Id` would not match.
If you want to index a custom log field that starts with the `@` character, you must include an extra `@` character when you enter the index string. For example, if you have a custom log field `@emailname`, enter `@@emailname` in the **Add field path** box.
You can also create indexes for the `@ingestionTime` and `@logStream` fields that CloudWatch Logs automatically generates. If you do, do not need to add an extra `@` when specifying them.
1. (Optional) In addition to specifying the field path, you can select **Set as a facet** to create the field as a facet.
1. Repeat the previous step to add as many as 20 field indexes.
1. When you have finished, choose **Save**.
# Log group selection options when creating a query
This section explains the various ways that you can select log groups to include in a query.
**To select log groups for a query in the console**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Logs**, **Logs Insights**.
1. Select the query language you want to use for this query. You can choose either: **Logs Insights QL**, **OpenSearch PPL**, or **OpenSearch SQL**.
1. There are three ways to select log groups for the query:
+ Use the **Log group name** box. This is the default selection method. You can enter as many as 50 log group names with this method. If this is a monitoring account in CloudWatch cross-account observability, you can select log groups in the source accounts as well as the monitoring account. A single query can query logs from different accounts at once.
+ Use the **Log group criteria** section. In this section, you can choose log groups based on the prefix of the log group names. You can include as many as five prefixes in one query. Log groups having these prefixes in their names will be selected. Alternatively, the **All log groups** option selects all the log groups from the account.
+ If this is a monitoring account in CloudWatch cross-account observability, you can select **All accounts** in the account dropdown menu to select the log groups from all linked accounts. Alternatively, you can individually select which accounts should be included for this query.
If your choices match more than 10,000 log groups, you'll see an error that prompts you to narrow your selection.
1. The default log class for a query is **Standard**. You can use **Log class** to change it to **Infrequent access**.
**Using the AWS CLI**
To make these types of selections when you start a query from the command line, you can use the `source` command in your query. For more information and examples, see [SOURCE](CWL_QuerySyntax-Source.md).
# Effects of deleting a field index policy
If you delete a field index policy that has been in effect for a time, the following happens:
+ For up to 30 days after the policy is deleted, queries can still benefit from the indexed log events.
+ If you delete a log-group level index policy, and there is already an account-level policy in place that would apply to that log group, the account-level policy will eventually apply to that log group.