# CloudWatch Logs Insights query language (Logs Insights QL) This section includes full documentation of Logs Insights QL commands and functions. It also includes sample queries for this language. For information on other query languages you can use, see [OpenSearch Service PPL](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_PPL.html), [OpenSearch Service SQL](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_SQL.html), and [CloudWatch Metrics Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/query_with_cloudwatch-metrics-insights.html). **Topics** + [ # CloudWatch Logs Insights language query syntax ](CWL_QuerySyntax.md) + [ # Get started with Logs Insights QL: Query tutorials ](CWL_AnalyzeLogData_Tutorials.md) + [ # Sample queries ](CWL_QuerySyntax-examples.md) + [ # Compare (diff) with previous time ranges ](CWL_AnalyzeLogData_Compare.md) + [ # Visualize log data in graphs ](CWL_Insights-Visualizing-Log-Data.md) # CloudWatch Logs Insights language query syntax This section provides details about the Logs Insights QL. The query syntax supports different functions and operations that include but aren't limited to general functions, arithmetic and comparison operations, and regular expressions. **Important** To avoid incurring excessive charges by running large queries, keep in mind the following best practices: Select only the necessary log groups for each query. Always specify the narrowest possible time range for your queries. When you use the console to run queries, cancel all your queries before you close the CloudWatch Logs Insights console page. Otherwise, queries continue to run until completion. When you add a CloudWatch Logs Insights widget to a dashboard, ensure that the dashboard is not refreshing at a high frequency, because each refresh starts a new query. To create queries that contain multiple commands, separate the commands with the pipe character (**\$1**). To create queries that contain comments, set off the comments with the hash character (**\$1**). **Note** CloudWatch Logs Insights automatically discovers fields for different log types and generates fields that start with the **@** character. For more information about these fields, see [ Supported logs and discovered fields](https://docs.aws.amazon.com/en_us/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html) in the *Amazon CloudWatch User Guide*. The following table briefly describes each command. Following this table is a more comprehensive description of each command, with examples. **Note** All Logs Insights QL query commands are supported on log groups in the Standard log class. Log groups in the Infrequent Access log class support all Logs Insights QL query commands except `pattern`, `diff`, and `unmask`. | | | | --- |--- | | **` anomaly`** | Identifies unusual patterns in your log data using machine learning. | | **` display`** | Displays a specific field or fields in query results. | | **` fields`** | Displays specific fields in query results and supports functions and operations you can use to modify field values and create new fields to use in your query. | | **` filter`** | Filters the query to return only the log events that match one or more conditions. | | **` filterIndex`** | Forces a query to attempt to scan only the log groups that are both indexed on the field mentioned in a field index and also contain a value for that field index. This reduces scanned volume by attempting to scan only log events from these log groups that contain the value specified in the query for this field index. This command is not supported for log groups in the Infrequent Access log class. | | **` pattern`** | Automatically clusters your log data into patterns. A pattern is shared text structure that recurs among your log fields. CloudWatch Logs Insights provides ways for you to analyze the patterns found in your log events. For more information, see [Pattern analysis](CWL_AnalyzeLogData_Patterns.md). | | **` diff`** | Compares the log events found in your requested time period with the log events from a previous time period of equal length, so that you can look for trends and find out if certain log events are new. | | **` parse`** | Extracts data from a log field to create an extracted field that you can process in your query. **`parse`** supports both glob mode using wildcards, and regular expressions. | | **` sort`** | Displays the returned log events in ascending (`asc`) or descending (`desc`) order. | | **` SOURCE`** | Including `SOURCE` in a query is a useful way to specify a large amount of log groups based on log group name prefix, account identifiers, and log group class to include in a query. This command is supported only when you create a query in the AWS CLI or programmatically, not in the CloudWatch console. | | **` stats`** | Calculate aggregate statistics using values in the log fields. | | **` limit`** | Specifies a maximum number of log events that you want your query to return. Useful with **`sort`** to return "top 20" or "most recent 20" results. | | **` dedup`** | Removes duplicate results based on specific values in fields that you specify. | | **` unmask`** | Displays all the content of a log event that has some content masked because of a data protection policy. For more information about data protection in log groups, see [Help protect sensitive log data with masking](mask-sensitive-log-data.md). | | **`[unnest](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-Unnest.html)`** | Flattens a list taken as input to produce multiple records with a single record for each element in the list. | | **` lookup`** | Enriches log events with data from a lookup table by matching field values. Use lookup tables to add reference data such as user details, application names, or product information to your query results. | | ** [Other operations and functions](CWL_QuerySyntax-operations-functions.md)** | CloudWatch Logs Insights also supports many comparison, arithmetic, datetime, numeric, string, IP address, and general functions and operations. | The following sections provide more details about the CloudWatch Logs Insights query commands. **Topics** + [ # Logs Insights QL commands supported in log classes ](CWL_AnalyzeLogData_Classes.md) + [ # anomaly ](CWL_QuerySyntax-Anomaly.md) + [ # **display** ](CWL_QuerySyntax-Display.md) + [ # fields ](CWL_QuerySyntax-Fields.md) + [ # filter ](CWL_QuerySyntax-Filter.md) + [ # filterIndex ](CWL_QuerySyntax-FilterIndex.md) + [ # SOURCE ](CWL_QuerySyntax-Source.md) + [ # pattern ](CWL_QuerySyntax-Pattern.md) + [ # diff ](CWL_QuerySyntax-Diff.md) + [ # parse ](CWL_QuerySyntax-Parse.md) + [ # sort ](CWL_QuerySyntax-Sort.md) + [ # stats ](CWL_QuerySyntax-Stats.md) + [ # limit ](CWL_QuerySyntax-Limit.md) + [ # dedup ](CWL_QuerySyntax-Dedup.md) + [ # unmask ](CWL_QuerySyntax-Unmask.md) + [ # unnest ](CWL_QuerySyntax-Unnest.md) + [ # lookup ](CWL_QuerySyntax-Lookup.md) + [ # Boolean, comparison, numeric, datetime, and other functions ](CWL_QuerySyntax-operations-functions.md) + [ # Fields that contain special characters ](CWL_QuerySyntax-Guidelines.md) + [ # Use aliases and comments in queries ](CWL_QuerySyntax-alias.md) # Logs Insights QL commands supported in log classes All Logs Insights QL query commands are supported on log groups in the Standard log class. Log groups in the Infrequent Access log class support all query commands except `pattern`, `diff`, `filterIndex`, and `unmask`. # anomaly Use `anomaly` to automatically identify unusual patterns and potential issues within your log data using machine learning. The `anomaly` command extends the existing `pattern` functionality and leverages advanced analytics to help identify potential anomalies in log data. You can use `anomaly` to reduce the time it takes to identify and resolve operational issues by automatically surfacing unusual patterns or behaviors in your logs. The `anomaly` command works with the ` pattern` command to first identify log patterns, then detect anomalies within those patterns. You can also combine `anomaly` with the ` filter` or ` sort` commands to focus anomaly detection on specific subsets of your data. **Anomaly Command Input** The `anomaly` command is typically used after the ` pattern` command to analyze the patterns identified in your log data. The command does not require additional parameters and analyzes the output from preceding commands in your query. **Types of Anomalies Identified** The `anomaly` command identifies five distinct types of anomalies: + *Pattern Frequency Anomalies*: Unusual frequencies of specific log patterns, such as when an application starts generating more error messages than usual. + *New Pattern Anomalies*: Previously unseen log patterns that may indicate new types of errors or messages appearing in your logs. + *Token Variation Anomalies*: Unexpected changes in log message contents that may indicate unusual variations in expected log formats. + *Numerical Token Anomalies*: Unusual changes in numerical values within logs that can help detect potential performance issues or unexpected metric variations. + *HTTP Error Code Anomalies*: Patterns related to HTTP error responses, particularly useful when monitoring web applications and APIs. **Anomaly Command Output** The `anomaly` command preserves all fields from the input data and adds anomaly detection results to help identify unusual patterns in your log data. **Examples** The following command identifies patterns in your log data and then detects anomalies within those patterns: ``` fields @timestamp, @message | pattern @message | anomaly ``` The `anomaly` command can be used with filtering to focus on specific log types: ``` fields @timestamp, @message | filter @type = "REPORT" | pattern @message | anomaly ``` The `anomaly` command can be combined with sorting to organize results: ``` fields @timestamp, @message | filter @type = "ERROR" | pattern @message | anomaly | sort @timestamp desc ``` # **display** Use `display` to show a specific field or fields in query results. The `display` command shows only the fields you specify. If your query contains multiple `display` commands, the query results show only the field or fields that you specified in the final `display` command. **Example: Display one field** The code snippet shows an example of a query that uses the parse command to extract data from `@message` to create the extracted fields `loggingType` and `loggingMessage`. The query returns all log events where the values for `loggingType` are **ERROR**. `display` shows only the values for `loggingMessage` in the query results. ``` fields @message | parse @message "[*] *" as loggingType, loggingMessage | filter loggingType = "ERROR" | display loggingMessage ``` **Tip** Use `display` only once in a query. If you use `display` more than once in a query, the query results show the field specified in the last occurrence of `display` command being used. # fields Use `fields` to show specific fields in query results. If your query contains multiple `fields` commands and doesn't include a `display` command, the results display all of the fields that are specified in the `fields` commands. ** Example: Display specific fields ** The following example shows a query that returns 20 log events and displays them in descending order. The values for `@timestamp` and `@message` are shown in the query results. ``` fields @timestamp, @message | sort @timestamp desc | limit 20 ``` Use `fields` instead of `display` when you want to use the different functions and operations supported by `fields` for modifying field values and creating new fields that can be used in queries. You can use the `fields` command with the keyword *as* to create extracted fields that use fields and functions in your log events. For example, `fields ispresent as isRes` creates an extracted field named `isRes`, and the extracted field can be used in the rest of your query. # filter Use `filter` to get log events that match one or more conditions. ** Example: Filter log events using one condition ** The code snippet shows an example of a query that returns all log events where the value for `range` is greater than ***3000***. The query limits the results to 20 log events and sorts the logs events by `@timestamp` and in descending order. ``` fields @timestamp, @message | filter (range>3000) | sort @timestamp desc | limit 20 ``` ** Example: Filter log events using more than one condition ** You can use the keywords `and` and `or` to combine more than one condition. The code snippet shows an example of a query that returns log events where the value for `range` is greater than ***3000*** and value for `accountId` is equal to ***123456789012***. The query limits the results to 20 log events and sorts the logs events by `@timestamp` and in descending order. ``` fields @timestamp, @message | filter (range>3000 and accountId=123456789012) | sort @timestamp desc | limit 20 ``` ## Indexed fields and the filter command If you have created field indexes for a log group, you can leverage those field indexes to make your `filter` queries more efficient and reduce scanned volume. For example, suppose you have created a field index for `requestId`. Then, any CloudWatch Logs Insights query on that log group that includes `filter requestId = value` or `filter requestId IN [value, value, ...]` will attempt to skip processing log events that are known not to include the indexed field. By attempting to scan only the log events that are known to contain that indexed field, scan volume can be reduced and the query is faster. For more information about field indexes and how to create them, see [Create field indexes to improve query performance and reduce scan volume](CloudWatchLogs-Field-Indexing.md). **Important** Only queries with `filter fieldName =...` and `filter fieldName IN...` will benefit from the field index improvements. Queries with `filter fieldName like` don't use indexes and always scan all log events in the selected log groups. ** Example: Find log events that are related to a certain request ID, using indexes ** This example assumes that you have created a field index on `requestId`. For log groups that use this field index, the query will leverage field indexes to attempt to scan the least amount of log events to find events with `requestId` with a value of `123456` ``` fields @timestamp, @message | filter requestId = "1234656" | limit 20 ``` ## Matches and regular expressions in the filter command The filter command supports the use of regular expressions. You can use the following comparison operators (`=`, `!=`, `<`, `<=`, `>`, `>=`) and Boolean operators (`and`, `or`, and `not`). You can use the keyword `in` to test for set membership and check for elements in an array. To check for elements in an array, put the array after `in`. You can use the Boolean operator `not` with `in`. You can create queries that use `in` to return log events where fields are string matches. The fields must be complete strings. For example, the following code snippet shows a query that uses `in` to return log events where the field `logGroup` is the complete string `example_group`. ``` fields @timestamp, @message | filter logGroup in ["example_group"] ``` You can use the keyword phrases `like` and `not like` to match substrings. You can use the regular expression operator `=~` to match substrings. To match a substring with `like` and `not like`, enclose the substring that you want to match in single or double quotation marks. You can use regular expression patterns with `like` and `not like`. To match a substring with the regular expression operator, enclose the substring that you want to match in forward slashes. The following examples contain code snippets that show how you can match substrings using the `filter` command. **Examples: Match substrings** The following examples return log events where `f1` contains the word ***Exception***. All three examples are case sensitive. The first example matches a substring with `like`. ``` fields f1, f2, f3 | filter f1 like "Exception" ``` The second example matches a substring with `like` and a regular expression pattern. ``` fields f1, f2, f3 | filter f1 like /Exception/ ``` The third example matches a substring with a regular expression. ``` fields f1, f2, f3 | filter f1 =~ /Exception/ ``` **Example: Match substrings with wildcards** You can use the period symbol (`.`) as a wildcard in regular expressions to match substrings. In the following example, the query returns matches where the value for `f1` begins with the string `ServiceLog`. ``` fields f1, f2, f3 | filter f1 like /ServiceLog./ ``` You can place the asterisk symbol after the period symbol (`.*`) to create a greedy quantifier that returns as many matches as possible. For example, the following query returns matches where the value for `f1` not only begins with the string `ServiceLog`, but also includes the string `ServiceLog`. ``` fields f1, f2, f3 | filter f1 like /ServiceLog.*/ ``` Possible matches can be formatted like the following: + `ServiceLogSampleApiLogGroup` + `SampleApiLogGroupServiceLog` **Example: Exclude substrings from matches** The following example shows a query that returns log events where `f1` doesn't contain the word ***Exception***. The example is case senstive. ``` fields f1, f2, f3 | filter f1 not like "Exception" ``` **Example: Match substrings with case-insensitive patterns** You can match substrings that are case insensitive with `like` and regular expressions. Place the following parameter (**?i**) before the substring you want to match. The following example shows a query that returns log events where `f1` contains the word ***Exception*** or ***exception***. ``` fields f1, f2, f3 | filter f1 like /(?i)Exception/ ``` # filterIndex Use `filterIndex` to return indexed data only, by forcing a query to scan only log groups that are indexed on a field that you specify in the query. For these log groups that are indexed on this field, it further optimizes the query by skipping the log groups that do not have any log events containing the field specified in the query for the indexed field. It further reduces scanned volume by attempting to scan only log events from these log groups that match the value specified in the query for this field index. For more information about field indexes and how to create them, see [Create field indexes to improve query performance and reduce scan volume](CloudWatchLogs-Field-Indexing.md). Using `filterIndex` with indexed fields can help you query log groups that include petabytes of log data efficiently by limiting the actual search space to log groups and log events that have field indexes. For example, suppose that you have created a field index for `IPaddress` in some of the log groups in your account. You can then create the following query and choose to query all log groups in the account to find log events that include the value `198.51.100.0` in the `IPaddress` field. ``` fields @timestamp, @message | filterIndex IPaddress = "198.51.100.0" | limit 20 ``` The `filterIndex` command causes this query to attempt to skip all log groups that are not indexed for `IPaddress`. Additionally, within the log groups that are indexed, the query skips log events that have an `IPaddress` field but not observed `198.51.100.0` as the value for that field. Use the `IN` operator to expand the results to any of multiple values for the indexed fields. The following example finds logs events that include either the value `198.51.100.0` or `198.51.100.1` in the `IPaddress` field. ``` fields @timestamp, @message | filterIndex IPaddress in ["198.51.100.0", "198.51.100.1"] | limit 20 ``` CloudWatch Logs provides default field indexes for all log groups in the Standard log class. Default field indexes are automatically available for the following fields: + `@logStream` + `@aws.region` + `@aws.account` + `@source.log` + `@data_source_name` + `@data_source_type` + `@data_format` + `traceId` + `severityText` + `attributes.session.id` CloudWatch Logs provides default field indexes for certain data source name and type combinations as well. Default field indexes are automatically available for the following data source name and type combinations: | Data Source Name and Type | Default Field Indexes | | --- | --- | | `amazon_vpc.flow` | `action` `logStatus` `region` `flowDirection` `type` | | `amazon_route53.resolver_query` | `query_type` `transport` `rcode` | | `aws_waf.access` | `action` `httpRequest.country` | | `aws_cloudtrail.data` ` aws_cloudtrail.management` | `eventSource` `eventName` `awsRegion` `userAgent` `errorCode` `eventType` `managementEvent` `readOnly` `eventCategory` `requestId` | Default field indexes are in addition to any custom field indexes you define within your policy. Default field indexes are not counted towards your [field index quota](CloudWatchLogs-Field-Indexing-Syntax.md). ## filterIndex compared to filter To illustrate the difference between `filterIndex` and `filter`, consider the following example queries. Assume that you have created a field index for `IPaddress`, for four of your log groups, but not for a fifth log group. The following query using `filterIndex` will skip scanning the log group that doesn't have the field indexed. For each indexed log group, it attempts to scan only log events that have the indexed field, and it also returns only results from after the field index was created. ``` fields @timestamp, @message | filterIndex IPaddress = "198.51.100.0" | limit 20 ``` In contrast, if you use `filter` instead of `filterIndex` for a query of the same five log groups, the query will attempt to scan not only the log events that contain the value in the indexed log groups, but will also scan the fifth log group that isn't indexed, and it will scan every log event in that fifth log group. ``` fields @timestamp, @message | filter IPaddress = "198.51.100.0" | limit 20 ``` # SOURCE Including `SOURCE` in a query is a useful way to specify the log groups and/or data sources to include in a query when you are using the AWS CLI or API to create a query. The `SOURCE` command is supported only in the AWS CLI and API, not in the CloudWatch console. When you use the CloudWatch console to start a query, you use the console interface to specify the log groups. Query log groups To use `SOURCE` to specify the log groups to query, you can use the following keywords: + `namePrefix` runs the query against log groups that have names that start with the string that you specify. If you omit this, all log groups are queried. You can include as many as five prefixes in the list. + `accountIdentifier` runs the query against log groups in the specified AWS account. This works only when you run the query in a monitoring account. If you omit this, the default is to query all linked source accounts and the current monitoring account. For more information about cross-account observability, see [CloudWatch cross-account observability](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html). You can include as many as 20 account identifiers in the list. + `logGroupClass` runs the query against log groups that are in the specified log class, either Standard or Infrequent Access. If you omit this, the default of Standard log class is used. For more information about log classes, see [Log classes](CloudWatch_Logs_Log_Classes.md). Because you can specify large numbers of log groups to query this way, we recommend that you use `SOURCE` only in queries that leverage field indexes that you have created. For more information about indexing fields in log groups, see [Create field indexes to improve query performance and reduce scan volume](CloudWatchLogs-Field-Indexing.md) The following example selects all log groups in the account. If this is a monitoring account then the log groups across monitoring and all the source accounts will be selected. If the total number of log groups exceed 10,000 then you will see an error prompting you to reduce the number of log groups by using a different log group selection method. ``` SOURCE logGroups() ``` The following example selects the log groups in the `111122223333` source account. If you start a query in a monitoring account in CloudWatch cross-account observability, log groups in all source accounts and in the monitoring account are selected by default. ``` SOURCE logGroups(accountIdentifiers:['111122223333']) ``` The next example selects log groups based on name prefixes. ``` SOURCE logGroups(namePrefix: ['namePrefix1', 'namePrefix2']) ``` The following example selects all log groups in the Infrequent Access log class. If you don't include the `class` identifier, the query applies only to log groups in the Standard log class, which is the default. ``` SOURCE logGroups(class: ['INFREQUENT_ACCESS']) ``` The next example selects log groups in the 111122223333 account that start with specific name prefixes and are in the Standard log class. The class is not mentioned in the command because Standard is the default log class value. ``` SOURCE logGroups(accountIdentifiers:['111122223333'], namePrefix: ['namePrefix1', 'namePrefix2'] ``` The final example displays how to use the `SOURCE` command with the `start-query` AWS CLI command. ``` aws logs start-query --region us-east-1 --start-time 1729728200 --end-time 1729728215 --query-string "SOURCE logGroups(namePrefix: ['Query']) | fields @message | limit 5" ``` Query data sources To use `SOURCE` to specify the data sources to query, you can use the `dataSource` keyword. You can include as many as ten data sources in the list. The following example selects the `amazon_vpc.flow` data source. ``` SOURCE dataSource(['amazon_vpc.flow']) ``` The following example selects the `amazon_vpc.flow` data source and limits the log groups based on a log group name prefix. ``` SOURCE dataSource(['amazon_vpc.flow']) logGroups(namePrefix: ['namePrefix1']) ``` # pattern Use `pattern` to automatically cluster your log data into patterns. A pattern is a shared text structure that recurs among your log fields. You can use `pattern` to surface emerging trends, monitor known errors, and identify frequently occurring or high-cost log lines. CloudWatch Logs Insights also provides a console experience you can use to find and further analyze patterns in your log events. For more information, see [Pattern analysis](CWL_AnalyzeLogData_Patterns.md). Because the `pattern` command automatically identifies common patterns, you can use it as a starting point to search and analyze yours logs. You can also combine `pattern` with the ` filter`, ` parse`, or ` sort` commands to identify patterns in more fine-tuned queries. **Pattern Command Input** The `pattern` command expects one of the following inputs: the `@message` field, an extracted field created using the ` parse` command, or a string manipulated using one or more [String functions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-operations-functions.html#CWL_QuerySyntax-string-functions). If CloudWatch Logs can't infer the type of data that a dynamic token represents, displays it as , and *number* indicates where in the pattern this token appears, compared to the other dynamic tokens. Common examples of dynamic tokens include error codes, IP addresses, timestamps, and request IDs. **Pattern Command Output** The `pattern` command produces the following output: + `@pattern`: A shared text structure that recurs among your log event fields. Fields that vary within a pattern, such as a request ID or timestamp, are represented by *tokens*. If CloudWatch Logs can determine the type of data that a dynamic token represents, it displays the token as ``. The *string* is a description of the type of data that the token represents. The *number* shows where in the pattern this token appears, compared to the other dynamic tokens. CloudWatch Logs assigns the string part of the name based on analyzing the content of the log events that contain it. If CloudWatch Logs can't infer the type of data that a dynamic token represents, displays it as , and *number* indicates where in the pattern this token appears, compared to the other dynamic tokens. For example, `[INFO] Request time: ms` is a potential output for the log message `[INFO] Request time: 327 ms`. + `@ratio`: The ratio of log events from a selected time period and specified log groups that match an identified pattern. For example, if half of the log events in the selected log groups and time period match the pattern, `@ratio` returns `0.50` + `@sampleCount`: A count of the number of log events from a selected time period and specified log groups that match an identified pattern. + `@severityLabel`: The log severity or level, which indicates the type of information contained in a log. For example, `Error`, `Warning`, `Info`, or `Debug`. **Examples** The following command identifies logs with similar structures in specified log group(s) over the selected time range, grouping them by pattern and count ``` pattern @message ``` The `pattern` command can be used in combination with the ` filter` command ``` filter @message like /ERROR/ | pattern @message ``` The `pattern` command can be use with the ` parse` and ` sort` commands ``` filter @message like /ERROR/ | parse @message 'Failed to do: *' as cause | pattern cause | sort @sampleCount asc ``` # diff Compares the log events found in your requested time period with the log events from a previous time period of equal length. This way, you can look for trends and find whether specific log events are new. Add a modifier to the `diff` command to specify the time period that you want to compare with: + `diff` compares the log events in the currently selected time range to the log events of the immediately preceding time range. + `diff previousDay` compares the log events in the currently selected time range to the log events from the same time the preceding day. + `diff previousWeek` compares the log events in the currently selected time range to the log events from the same time the preceding week. + `diff previousMonth` compares the log events in the currently selected time range to the log events from the same time the preceding month. For more information, see [Compare (diff) with previous time ranges](CWL_AnalyzeLogData_Compare.md). # parse Use `parse` to extract data from a log field and create an extracted field that you can process in your query. **`parse`** supports both glob mode using wildcards, and regular expressions. For information about regular expression syntax, see [Supported regular expressions (regex) syntax](FilterAndPatternSyntax.md#regex-expressions). You can parse nested JSON fields with a regular expression. **Example: Parsing a nested JSON field** The code snippet shows how to parse a JSON log event that's been flattened during ingestion. ``` {'fieldsA': 'logs', 'fieldsB': [{'fA': 'a1'}, {'fA': 'a2'}]} ``` The code snippet shows a query with a regular expression that extracts the values for `fieldsA` and `fieldsB` to create the extracted fields `fld` and `array`. ``` parse @message "'fieldsA': '*', 'fieldsB': ['*']" as fld, array ``` **Named capturing groups** When you use **`parse`** with a regular expression, you can use named capturing groups to capture a pattern into a field. The syntax is `parse @message (?pattern)` The following example uses a capturing group on a VPC flow log to extract the ENI into a field named `NetworkInterface`. ``` parse @message /(?eni-.*?) / | display NetworkInterface, @message ``` **Note** JSON log events are flattened during ingestion. Currently, parsing nested JSON fields with a glob expression isn't supported. You can only parse JSON log events that include no more than 200 log event fields. When you parse nested JSON fields, you must format the regular expression in your query to match the format of your JSON log event. ## Examples of the parse command **Use a glob expression to extract the fields `@user`, `@method`, and `@latency` from the log field `@message` and return the average latency for each unique combination of `@method` and `@user`.** ``` parse @message "user=*, method:*, latency := *" as @user, @method, @latency | stats avg(@latency) by @method, @user ``` **Use a regular expression to extract the fields `@user2`, `@method2`, and `@latency2` from the log field `@message` and return the average latency for each unique combination of `@method2` and `@user2`.** ``` parse @message /user=(?.*?), method:(?.*?), latency := (?.*?)/ | stats avg(latency2) by @method2, @user2 ``` **Extracts the fields `loggingTime`, `loggingType` and `loggingMessage`, filters down to log events that contain `ERROR` or `INFO` strings, and then displays only the `loggingMessage` and `loggingType` fields for events that contain an `ERROR` string.** ``` FIELDS @message | PARSE @message "* [*] *" as loggingTime, loggingType, loggingMessage | FILTER loggingType IN ["ERROR", "INFO"] | DISPLAY loggingMessage, loggingType = "ERROR" as isError ``` # sort Use `sort` to display log events in ascending (`asc`) or descending (`desc`) order by a specified field. You can use this with the `limit` command to create "top N" or "bottom N" queries. The sorting algorithm is an updated version of natural sorting. If you sort in ascending order, the following logic is used. + All non-number values come before all number values. *Number values* are values that include only numbers, not a mix of numbers and other characters. + For non-number values, the algorithm groups consecutive numeric characters and consecutive alphabetic characters into separate chunks for comparison. It orders non-numeric portions by their Unicode values, and it orders numeric portions by their length first and then by their numerical value. For more information about Unicode order, see [List of Unicode characters](https://en.wikipedia.org/wiki/List_of_Unicode_characters). For example, the following is the result of a sort in ascending order. ``` !: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sorted by unicode order # *%04 0# >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alphanumeric starting with numbers 5A 111A >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Starts with more digits than 5A, so it sorted to be later than 5A 2345_ @ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2345 is compared with @ in the unicode order, @_ A >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Values starting with letters A9876fghj a12345hfh 0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Number values 01 1 2 3 ``` If you sort in descending order, the sort results are the reverse. For example, the following query for Amazon VPC flow logs finds the top 15 packet transfers across hosts. ``` stats sum(packets) as packetsTransferred by srcAddr, dstAddr | sort packetsTransferred desc | limit 15 ``` # stats Use `stats` to create visualizations of your log data such as bar charts, line charts, and stacked area charts. This helps you more efficiently identify patterns in your log data. CloudWatch Logs Insights generates visualizations for queries that use the `stats` function and one or more aggregation functions. For example, the following query in a Route 53 log group returns visualizations showing the distribution of Route 53 records per hour, by query type. ``` stats count(*) by queryType, bin(1h) ``` All such queries can produce bar charts. If your query uses the `bin()` function to group the data by one field over time, you can also see line charts and stacked area charts. The following time units and abbreviations are supported with the `bin` function. For all units and abbreviations that include more than one character, adding s to pluralize is supported. So both `hr` and `hrs` work to specify hours. + `millisecond` `ms` `msec` + `second` `s` `sec` + `minute` `m` `min` + `hour` `h` `hr` + `day` `d` + `week` `w` + `month` `mo` `mon` + `quarter` `q` `qtr` + `year` `y` `yr` **Topics** + [ ## Visualize time series data ](#CWL_Insights-Visualizing-TimeSeries) + [ ## Visualize log data grouped by fields ](#CWL_Insights-Visualizing-ByFields) + [ ## Use multiple stats commands in a single query ](#CWL_QuerySyntax-stats-multi) + [ ## Functions to use with stats ](#CWL_QuerySyntax-stats-functions) ## Visualize time series data Time series visualizations work for queries with the following characteristics: + The query contains one or more aggregation functions. For more information, see [Aggregation Functions in the Stats Command](#CWL_Insights_Aggregation_Functions). + The query uses the `bin()` function to group the data by one field. These queries can produce line charts, stacked area charts, bar charts, and pie charts. **Examples** For a complete tutorial, see [Tutorial: Run a query that produces a time series visualization](CWL_AnalyzeLogData_VisualizationQuery.md). Here are more example queries that work for time series visualization. The following query generates a visualization of the average values of the `myfield1` field, with a data point created every five minutes. Each data point is the aggregation of the averages of the `myfield1` values from the logs from the previous five minutes. ``` stats avg(myfield1) by bin(5m) ``` The following query generates a visualization of three values based on different fields, with a data point created every five minutes. The visualization is generated because the query contains aggregate functions and uses `bin()` as the grouping field. ``` stats avg(myfield1), min(myfield2), max(myfield3) by bin(5m) ``` **Line chart and stacked area chart restrictions** Queries that aggregate log entry information but don't use the `bin()` function can generate bar charts. However, the queries cannot generate line charts or stacked area charts. For more information about these types of queries, see [Visualize log data grouped by fields](#CWL_Insights-Visualizing-ByFields). ## Visualize log data grouped by fields You can produce bar charts for queries that use the `stats` function and one or more aggregation functions. For more information, see [Aggregation Functions in the Stats Command](#CWL_Insights_Aggregation_Functions). To see the visualization, run your query. Then choose the **Visualization** tab, select the arrow next to **Line**, and choose **Bar**. Visualizations are limited to up to 100 bars in the bar chart. **Examples** For a complete tutorial, see [Tutorial: Run a query that produces a visualization grouped by log fields](CWL_AnalyzeLogData_VisualizationFieldQuery.md). The following paragraphs include more example queries for visualization by fields. The following VPC flow log query finds the average number of bytes transferred per session for each destination address. ``` stats avg(bytes) by dstAddr ``` You can also produce a chart that includes more than one bar for each resulting value. For example, the following VPC flow log query finds the average and maximum number of bytes transferred per session for each destination address. ``` stats avg(bytes), max(bytes) by dstAddr ``` The following query finds the number of Amazon Route 53 query logs for each query type. ``` stats count(*) by queryType ``` ## Use multiple stats commands in a single query You can use as many as two `stats` commands in a single query. This enables you to perform an additional aggregation on the output of the first aggregation. **Example: Query with two `stats` commands** For example, the following query first find the total traffic volume in 5-minute bins, then calculates the highest, lowest, and average traffic volume among those 5-minute bins. ``` FIELDS strlen(@message) AS message_length | STATS sum(message_length)/1024/1024 as logs_mb BY bin(5m) | STATS max(logs_mb) AS peak_ingest_mb, min(logs_mb) AS min_ingest_mb, avg(logs_mb) AS avg_ingest_mb ``` **Example: Combine multiple stats commands with other functions such as `filter`, `fields`, `bin`** You can combine two `stats` commands with other commands such as `filter` and `fields` in a single query. For example, the following query finds the number of distinct IP addresses in sessions and finds the number of sessions by client platform, filters those IP addresses, and then finally finds the average of session requests per client platform. ``` STATS count_distinct(client_ip) AS session_ips, count(*) AS requests BY session_id, client_platform | FILTER session_ips > 1 | STATS count(*) AS multiple_ip_sessions, sum(requests) / count(*) AS avg_session_requests BY client_platform ``` You can use `bin` and `dateceil` functions in queries with multiple `stats` commands. For example, the following query first combines messages into 5-minute blocks, then aggregates those 5-minute blocks into 10-minute blocks and calculates the highest, lowest, and average traffic volumes within each 10-minute block. ``` FIELDS strlen(@message) AS message_length | STATS sum(message_length) / 1024 / 1024 AS logs_mb BY BIN(5m) as @t | STATS max(logs_mb) AS peak_ingest_mb, min(logs_mb) AS min_ingest_mb, avg(logs_mb) AS avg_ingest_mb BY dateceil(@t, 10m) ``` **Notes and limitations** A query can have a maximum of two `stats` commands. This quota can't be changed. If you use a `sort` or `limit` command, it must appear after the second `stats` command. If it is before the second `stats` command, the query is not valid. When a query has two `stats` commands, the partial results from the query do not begin displaying until the first `stats` aggregation is complete. In the second `stats` command in a single query, you can refer only to fields that are defined in the first `stats` command. For example, the following query is not valid because the `@message` field won't be available after the first `stats` aggregation. ``` FIELDS @message | STATS SUM(Fault) by Operation # You can only reference `SUM(Fault)` or Operation at this point | STATS MAX(strlen(@message)) AS MaxMessageSize # Invalid reference to @message ``` Any fields that you reference after the first `stats` command must be defined in that first `stats` command. ``` STATS sum(x) as sum_x by y, z | STATS max(sum_x) as max_x by z # You can only reference `max(sum_x)`, max_x or z at this point ``` **Important** The `bin` function always implicitly uses the `@timestamp` field. This means that you can't use `bin` in the second `stats` command without using the first `stats` command to propagate the `timestamp` field. For example, the following query is not valid. ``` FIELDS strlen(@message) AS message_length | STATS sum(message_length) AS ingested_bytes BY @logStream | STATS avg(ingested_bytes) BY bin(5m) # Invalid reference to @timestamp field ``` Instead, define the `@timestamp` field in the first `stats` command, and then you can use it with `dateceil` in the second `stats` command as in the following example. ``` FIELDS strlen(@message) AS message_length | STATS sum(message_length) AS ingested_bytes, max(@timestamp) as @t BY @logStream | STATS avg(ingested_bytes) BY dateceil(@t, 5m) ``` ## Functions to use with stats CloudWatch Logs Insights supports both stats aggregation functions and stats non-aggregation functions. Use statsaggregation functions in the `stats` command and as arguments for other functions. | Function | Result type | Description | | --- | --- | --- | | `avg(fieldName: NumericLogField)` | number | The average of the values in the specified field. | | `count()` `count(fieldName: LogField)` | number | Counts the log events. `count()` (or `count(*)`) counts all events returned by the query, while `count(fieldName)` counts all records that include the specified field name. | | `count_distinct(fieldName: LogField)` | number | Returns the number of unique values for the field. If the field has very high cardinality (contains many unique values), the value returned by `count_distinct` is just an approximation. | | `max(fieldName: LogField)` | LogFieldValue | The maximum of the values for this log field in the queried logs. | | `min(fieldName: LogField)` | LogFieldValue | The minimum of the values for this log field in the queried logs. | | `pct(fieldName: LogFieldValue, percent: number)` | LogFieldValue | A percentile indicates the relative standing of a value in a dataset. For example, `pct(@duration, 95)` returns the `@duration` value at which 95 percent of the values of `@duration` are lower than this value, and 5 percent are higher than this value. | | `stddev(fieldName: NumericLogField)` | number | The standard deviation of the values in the specified field. | | `sum(fieldName: NumericLogField)` | number | The sum of the values in the specified field. | **Stats non-aggregation functions** Use non-aggregation functions in the `stats` command and as arguments for other functions. | Function | Result type | Description | | --- | --- | --- | | `earliest(fieldName: LogField)` | LogField | Returns the value of `fieldName` from the log event that has the earliest timestamp in the queried logs. | | `latest(fieldName: LogField)` | LogField | Returns the value of `fieldName` from the log event that has the latest timestamp in the queried logs. | | `sortsFirst(fieldName: LogField)` | LogField | Returns the value of `fieldName` that sorts first in the queried logs. | | `sortsLast(fieldName: LogField)` | LogField | Returns the value of `fieldName` that sorts last in the queried logs. | # limit Use `limit` to specify the number of log events that you want your query to return. If you omit `limit`, the query will return as many as 10,000 log events in the results. For example, the following example returns only the 25 most recent log events ``` fields @timestamp, @message | sort @timestamp desc | limit 25 ``` # dedup Use `dedup` to remove duplicate results based on specific values in fields that you specify. You can use `dedup` with one or more fields. If you specify one field with `dedup`, only one log event is returned for each unique value of that field. If you specify multiple fields, then one log event is returned for each unique combination of values for those fields. Duplicates are discarded based on the sort order, with only the first result in the sort order being kept. We recommend that you sort your results before putting them through the `dedup` command. If the results are not sorted before being run through `dedup`, then the default descending sort order using `@timestamp` is used. Null values are not considered duplicates for evaluation. Log events with null values for any of the specified fields are retained. To eliminate fields with null values, use **`filter`** using the `isPresent(field)` function. The only query command that you can use in a query after the `dedup` command is `limit`. When you use `dedup` in a query, the console displays a message such as **Showing X of Y records**, where X is the number of deduplicated results and Y is the total number of records matched before deduplication. This indicates that duplicate records were removed and does not mean that data is missing. **Example: See only the most recent log event for each unique value of the field named `server`** The following example displays the `timestamp`, `server`, `severity`, and `message` fields for only the most recent event for each unique value of `server`. ``` fields @timestamp, server, severity, message | sort @timestamp desc | dedup server ``` For more samples of CloudWatch Logs Insights queries, see [General queries](CWL_QuerySyntax-examples.md#CWL_QuerySyntax-examples-general). # unmask Use `unmask` to display all the content of a log event that has some content masked because of a data protection policy. To use this command, you must have the `logs:Unmask` permission. For more information about data protection in log groups, see [Help protect sensitive log data with masking](mask-sensitive-log-data.md). # unnest Use `unnest` to flatten a list taken as input to produce multiple records with a single record for each element in the list. Based on the number of items a field contains, this command discards the current record and generates new records. Each record includes the `unnested_field`, which represents an item. All other fields come from the original record. The input for `unnest` is `LIST`, which comes from the `jsonParse` function. For more information, see [Structure types](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-operations-functions.html#CWL_QuerySyntax-structure-types). Any other types, such as `MAP`, `String` and `numbers`, are treated as a list with one item in `unnest`. **Command structure** The following example describes the format of this command. ``` unnest field into unnested_field ``` **Example query** The following example parses a JSON object string and expands a list of field events. ``` fields jsonParse(@message) as json_message | unnest json_message.events into event | display event.name ``` The log event for this example query could be a JSON string as follows: ``` { "events": [ { "name": "exception" }, { "name": "user action" } ] } ``` In this case, the sample query produces two records in the query result, one with `event.name` as `exception` and another with `event.name` as **user action** **Example query** The following example flattens a list and then filters out items. ``` fields jsonParse(@message) as js | unnest js.accounts into account | filter account.type = "internal" ``` **Example query** The following example flattens a list for aggregation. ``` fields jsonParse(trimmedData) as accounts | unnest accounts into account | stats sum(account.droppedSpans) as n by account.accountId | sort n desc | limit 10 ``` # lookup Use `lookup` to enrich your query results with reference data from a lookup table. A lookup table contains CSV data that you upload to Amazon CloudWatch Logs. When a query runs, the `lookup` command matches a field in your log events against a field in the lookup table and appends the specified output fields to the results. Use lookup tables for data enrichment scenarios such as mapping user IDs to user details, product codes to product information, or error codes to error descriptions. ## Creating and managing lookup tables Before you can use the `lookup` command in a query, you must create a lookup table. You can create and manage lookup tables from the CloudWatch console or by using the Amazon CloudWatch Logs API. **To create a lookup table (console)** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Settings**, and then choose the **Logs** tab. 1. Scroll to **Lookup tables** and choose **Manage**. 1. Choose **Create lookup table**. 1. Enter a name for the lookup table. The name can contain only alphanumeric characters, hyphens, and underscores. 1. (Optional) Enter a description. 1. Upload a CSV file. The file must include a header row with column names, use UTF-8 encoding, and not exceed 10 MB. 1. (Optional) Specify a AWS KMS key to encrypt the table data. 1. Choose **Create**. After you create a lookup table, you can view it in the CloudWatch Logs Insights query editor. Choose the **Lookup tables** tab to browse available tables and their fields. To update a lookup table, select the table and choose **Actions**, **Update**. Upload a new CSV file to replace all existing content. To delete a lookup table, choose **Actions**, **Delete**. **Note** You can create up to 100 lookup tables per account per AWS Region. CSV files can be up to 10 MB. You can also manage lookup tables by using the Amazon CloudWatch Logs API. For more information, see [CreateLookupTable](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateLookupTable.html) in the *Amazon CloudWatch Logs API Reference*. **Note** If the lookup table is encrypted with a KMS key, the caller must have the `kms:Decrypt` permission on the key (the KMS key used to encrypt the lookup table) to use the `StartQuery` API with a query that references that lookup table. For more information, see [Encrypt lookup tables in CloudWatch Logs using AWS Key Management Service](encrypt-lookup-tables-kms.md). ## Query syntax for lookup **Command structure** The following shows the format of this command. ``` lookup table lookup-field as log-field [,...] output-mode output-field[,...] ``` The command uses the following arguments: + `table` – The name of the lookup table to use. + `lookup-field` – The field in the lookup table to match against. + `log-field` – The field in your log events to match. The match is exact and case-sensitive. + `output-mode` – Specify `OUTPUT` to add the output fields to the results. If a field with the same name already exists in the log event, it is overwritten. + `output-field` – One or more fields from the lookup table to add to the results. **Example: Enrich log events with user details** Suppose you have a log group with events that contain an `id` field, and a lookup table named `user_data` with columns `id`, `name`, `email`, and `department`. The following query enriches each log event with the user's name, email, and department from the lookup table. ``` fields action, status, name, email, department | lookup user_data id OUTPUT name, email, department ``` **Example: Use lookup with aggregation** You can use lookup output fields with aggregation functions. The following query enriches log events with user details and then counts events grouped by email address. ``` fields user_id, action, username, email, department | lookup user_data user_id OUTPUT username, email, department | stats count(*) by email ``` **Example: Use lookup with filter** You can filter results based on fields returned by the lookup. The following query enriches log events and then filters to show only events from a specific department. ``` fields user_id, action | lookup user_data user_id OUTPUT username, email, department | filter department = "Engineering" ``` # Boolean, comparison, numeric, datetime, and other functions CloudWatch Logs Insights supports many other operations and functions in queries, as explained in the following sections. **Topics** + [ ## Arithmetic operators ](#CWL_QuerySyntax-operations-arithmetic) + [ ## Boolean operators ](#CWL_QuerySyntax-operations-Boolean) + [ ## Comparison operators ](#CWL_QuerySyntax-operations-comparison) + [ ## Numeric operators ](#CWL_QuerySyntax-operations-numeric) + [ ## Structure types ](#CWL_QuerySyntax-structure-types) + [ ## Datetime functions ](#CWL_QuerySyntax-datetime) + [ ## General functions ](#CWL_QuerySyntax-general-functions) + [ ## JSON functions ](#CWL_QuerySyntax-json-functions) + [ ## IP address string functions ](#CWL_QuerySyntax-IPaddress-functions) + [ ## String functions ](#CWL_QuerySyntax-string-functions) ## Arithmetic operators Arithmetic operators accept numeric data types as arguments and return numeric results. Use arithmetic operators in the `filter` and `fields` commands and as arguments for other functions. | Operation | Description | | --- | --- | | `a + b` | Addition | | `a - b` | Subtraction | | `a * b` | Multiplication | | `a / b` | Division | | `a ^ b` | Exponentiation (`2 ^ 3` returns `8`) | | `a % b` | Remainder or modulus (`10 % 3` returns `1`) | ## Boolean operators Use the Boolean operators `and`, `or`, and `not`. **Note** Use Boolean operators only in functions that return a value of **TRUE** or **FALSE**. ## Comparison operators Comparison operators accept all data types as arguments and return a Boolean result. Use comparison operations in the `filter` command and as arguments for other functions. | Operator | Description | | --- | --- | | `=` | Equal | | `!=` | Not equal | | `<` | Less than | | `>` | Greater than | | `<=` | Less than or equal to | | `>=` | Greater than or equal to | ## Numeric operators Numeric operations accept numeric data types as arguments and return numeric results. Use numeric operations in the `filter` and `fields` commands and as arguments for other functions. | Operation | Result type | Description | | --- | --- | --- | | `abs(a: number)` | number | Absolute value | | `ceil(a: number)` | number | Round to ceiling (the smallest integer that is greater than the value of `a`) | | `floor(a: number)` | number | Round to floor (the largest integer that is smaller than the value of `a`) | | `greatest(a: number, ...numbers: number[])` | number | Returns the largest value | | `least(a: number, ...numbers: number[])` | number | Returns the smallest value | | `log(a: number)` | number | Natural log | | `sqrt(a: number)` | number | Square root | ## Structure types A map or list is a structure type in CloudWatch Logs Insights that allows you to access and use attributes for queries. **Example: To get a map or list** Use `jsonParse` to parse a field that's a json string into a map or a list. ``` fields jsonParse(@message) as json_message ``` **Example: To access attributes** Use the dot access operator (map.attribute) to access items in a map.. If an attribute in a map contains special characters, use backticks to enclose the attribute name (map.attributes.`special.char`). ``` fields jsonParse(@message) as json_message | stats count() by json_message.status_code ``` Use the bracket access operator (list[index]) to retrieve an item at a specific position within the list. ``` fields jsonParse(@message) as json_message | filter json_message.users[1].action = "PutData" ``` Wrap special characters in backticks (``) when special characters are present in the key name. ``` fields jsonParse(@message) as json_message | filter json_message.`user.id` = "123" ``` **Example: empty results** Maps and lists are treated as null for string, number, and datetime functions. ``` fields jsonParse(@message) as json_message | display toupper(json_message) ``` Comparing map and list to any other fields result in `false`. **Note** Using map and list in `dedup`,`pattern`, `sort`, and `stats` isn't supported. ## Datetime functions **Datetime functions** Use datetime functions in the `fields` and `filter`commands and as arguments for other functions. Use these functions to create time buckets for queries with aggregate functions. Use time periods that consist of a number and one of the following: + `ms` for milliseconds + `s` for seconds + `m` for minutes + `h` for hours For example, `10m` is 10 minutes, and `1h` is 1 hour. **Note** Use the most appropriate time unit for your datetime function. CloudWatch Logs caps your request according to the time unit that you choose. For example, it caps 60 as the maximum value for any request that uses `s`. So, if you specify `bin(300s)`, CloudWatch Logs actually implements this as 60 seconds, because 60 is the number of seconds in a minute so CloudWatch Logs won't use a number higher than 60 with `s`. To create a 5-minute bucket, use `bin(5m)` instead. The cap for `ms` is 1000, the caps for `s` and `m` are 60, and the cap for `h` is 24. The following table contains a list of the different datetime functions that you can use in query commands. The table lists each function's result type and contains a description of each function. **Tip** When you create a query command, you can use the time interval selector to select a time period that you want to query. For example, you can set a time period between 5 and 30-minute intervals; 1, 3, and 12-hour intervals; or a custom time frame. You also can set time periods between specific dates. | Function | Result type | Description | | --- | --- | --- | | `bin(period: Period)` | Timestamp | Rounds the value of `@timestamp` to the given time period and then truncates. For example, `bin(5m)` rounds the value of `@timestamp` to the nearest 5 minutes. You can use this to group multiple log entries together in a query. The following example returns the count of exceptions per hour: 
filter @message like /Exception/ 
    | stats count(*) as exceptionCount by bin(1h)
    | sort exceptionCount desc
The following time units and abbreviations are supported with the `bin` function. For all units and abbreviations that include more than one character, adding s to pluralize is supported. So both `hr` and `hrs` work to specify hours. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-operations-functions.html) | | `datefloor(timestamp: Timestamp, period: Period)` | Timestamp | Truncates the timestamp to the given period. For example, `datefloor(@timestamp, 1h)` truncates all values of `@timestamp` to the bottom of the hour. | | `dateceil(timestamp: Timestamp, period: Period)` | Timestamp | Rounds up the timestamp to the given period and then truncates. For example, `dateceil(@timestamp, 1h)` truncates all values of `@timestamp` to the top of the hour. | | `fromMillis(fieldName: number)` | Timestamp | Interprets the input field as the number of milliseconds since the Unix epoch and converts it to a timestamp. | | `toMillis(fieldName: Timestamp)` | number | Converts the timestamp found in the named field into a number representing the milliseconds since the Unix epoch. For example, `toMillis(@timestamp)` converts the timestamp `2022-01-14T13:18:031.000-08:00` to `1642195111000`. | | `now()` | number | Returns the time that the query processing was started, in epoch seconds. This function takes no arguments. You can use this to filter your query results according to the current time. For example, the following query returns all 4xx errors from the past two hours: 
parse @message "Status Code: *;" as statusCode\n 
| filter statusCode >= 400 and statusCode <= 499  \n 
| filter toMillis(@timestamp) >= (now() * 1000 - 7200000)
The following example returns all log entries from the past five hours that contain either the word `error` or `failure` 
fields @timestamp, @message 
| filter @message like /(?i)(error|failure)/ 
| filter toMillis(@timestamp) >= (now() * 1000 - 18000000)
| **Note** Currently, CloudWatch Logs Insights doesn't support filtering logs with human readable timestamps. ## General functions **General functions** Use general functions in the `fields` and `filter` commands and as arguments for other functions. | Function | Result type | Description | | --- | --- | --- | | `ispresent(fieldName: LogField)` | Boolean | Returns `true` if the field exists | | `coalesce(fieldName: LogField, ...fieldNames: LogField[])` | LogField | Returns the first non-null value from the list | ## JSON functions **JSON functions** Use JSON functions in the `fields` and `filter` commands and as arguments for other functions. | Function | Result type | Description | | --- | --- | --- | | `jsonParse(fieldName: string)` | Map \$1 List \$1 Empty | Returns a map or list when the input is a string representation of JSON object or a JSON array. Returns an empty value, if the input is not one of the representation. | | `jsonStringify(fieldName: Map \| List)` | String | Returns a JSON string from a map or list data. | ## IP address string functions **IP address string functions** Use IP address string functions in the `filter` and `fields` commands and as arguments for other functions. | Function | Result type | Description | | --- | --- | --- | | `isValidIp(fieldName: string)` | boolean | Returns `true` if the field is a valid IPv4 or IPv6 address. | | `isValidIpV4(fieldName: string)` | boolean | Returns `true` if the field is a valid IPv4 address. | | `isValidIpV6(fieldName: string)` | boolean | Returns `true` if the field is a valid IPv6 address. | | `isIpInSubnet(fieldName: string, subnet: string)` | boolean | Returns `true` if the field is a valid IPv4 or IPv6 address within the specified v4 or v6 subnet. When you specify the subnet, use CIDR notation such as `192.0.2.0/24` or `2001:db8::/32`, where `192.0.2.0` or `2001:db8::` is the start of the CIDR block. | | `isIpv4InSubnet(fieldName: string, subnet: string)` | boolean | Returns `true` if the field is a valid IPv4 address within the specified v4 subnet. When you specify the subnet, use CIDR notation such as `192.0.2.0/24` where `192.0.2.0` is the start of the CIDR block.. | | `isIpv6InSubnet(fieldName: string, subnet: string)` | boolean | Returns `true` if the field is a valid IPv6 address within the specified v6 subnet. When you specify the subnet, use CIDR notation such as `2001:db8::/32` where `2001:db8::` is the start of the CIDR block. | ## String functions **String functions** Use string functions in the `fields` and `filter` commands and as arguments for other functions. | Function | Result type | Description | | --- | --- | --- | | `isempty(fieldName: string)` | Number | Returns `1` if the field is missing or is an empty string. | | `isblank(fieldName: string)` | Number | Returns `1` if the field is missing, an empty string, or contains only white space. | | `concat(str: string, ...strings: string[])` | string | Concatenates the strings. | | `ltrim(str: string)` `ltrim(str: string, trimChars: string)` | string | If the function does not have a second argument, it removes white space from the left of the string. If the function has a second string argument, it does not remove white space. Instead, it removes the characters in `trimChars` from the left of `str`. For example, `ltrim("xyZxyfooxyZ","xyZ")` returns `"fooxyZ"`. | | `rtrim(str: string)` `rtrim(str: string, trimChars: string)` | string | If the function does not have a second argument, it removes white space from the right of the string. If the function has a second string argument, it does not remove white space. Instead, it removes the characters of `trimChars` from the right of `str`. For example, `rtrim("xyZfooxyxyZ","xyZ")` returns `"xyZfoo"`. | | `trim(str: string)` `trim(str: string, trimChars: string)` | string | If the function does not have a second argument, it removes white space from both ends of the string. If the function has a second string argument, it does not remove white space. Instead, it removes the characters of `trimChars` from both sides of `str`. For example, `trim("xyZxyfooxyxyZ","xyZ")` returns `"foo"`. | | `strlen(str: string)` | number | Returns the length of the string in Unicode code points. | | `toupper(str: string)` | string | Converts the string to uppercase. | | `tolower(str: string)` | string | Converts the string to lowercase. | | `substr(str: string, startIndex: number)` `substr(str: string, startIndex: number, length: number)` | string | Returns a substring from the index specified by the number argument to the end of the string. If the function has a second number argument, it contains the length of the substring to be retrieved. For example, `substr("xyZfooxyZ",3, 3)` returns `"foo"`. | | `replace(fieldName: string, searchValue: string, replaceValue: string)` | string | Replaces all instances of `searchValue` in `fieldName: string` with `replaceValue`. For example, the function `replace(logGroup,"smoke_test","Smoke")` searches for log events where the field `logGroup` contains the string value `smoke_test` and replaces the value with the string `Smoke`. | | `strcontains(str: string, searchValue: string)` | number | Returns 1 if `str` contains `searchValue` and 0 otherwise. | # Fields that contain special characters If a field contains non-alphanumeric characters other than the `@` symbol or the period (`.`), you must surround the field with backtick characters (```). For example, the log field `foo-bar` must be enclosed in backticks (``foo-bar``) because it contains a non-alphanumeric character, the hyphen (`-`). # Use aliases and comments in queries Create queries that contain aliases. Use aliases to rename log fields or when extracting values into fields. Use the keyword `as` to give a log field or result an alias. You can use more than one alias in a query. You can use aliases in the following commands: + `fields` + `parse` + `sort` + ` stats ` The following examples show how to create queries that contain aliases. **Example** The query contains an alias in the `fields` command. ``` fields @timestamp, @message, accountId as ID | sort @timestamp desc | limit 20 ``` The query returns the values for the fields `@timestamp`, `@message`, and `accountId`. The results are sorted in descending order and limited to 20. The values for `accountId` are listed under the alias `ID`. **Example** The query contains aliases in the `sort` and `stats` commands. ``` stats count(*) by duration as time | sort time desc ``` The query counts the number of times the field `duration` occurs in the log group and sorts the results in descending order. The values for `duration` are listed under the alias `time`. ## Use comments CloudWatch Logs Insights supports comments in queries. Use the hash character (**\$1**) to set off comments. You can use comments to ignore lines in queries or document queries. **Example: Query** When the following query is run, the second line is ignored. ``` fields @timestamp, @message, accountId # | filter accountId not like "7983124201998" | sort @timestamp desc | limit 20 ``` # Get started with Logs Insights QL: Query tutorials The following sections include sample query tutorials to help you get started with Logs Insights QL. **Topics** + [ # Tutorial: Run and modify a sample query ](CWL_AnalyzeLogData_RunSampleQuery.md) + [ # Tutorial: Run a query with an aggregation function ](CWL_AnalyzeLogData_AggregationQuery.md) + [ # Tutorial: Run a query that produces a visualization grouped by log fields ](CWL_AnalyzeLogData_VisualizationFieldQuery.md) + [ # Tutorial: Run a query that produces a time series visualization ](CWL_AnalyzeLogData_VisualizationQuery.md) # Tutorial: Run and modify a sample query The following tutorial helps you get started with CloudWatch Logs Insights. You run a sample query in Logs Insights QL, and then see how to modify and rerun it. To run a query, you must already have logs stored in CloudWatch Logs. If you are already using CloudWatch Logs and have log groups and log streams set up, you are ready to start. You may also already have logs if you use services such as AWS CloudTrail, Amazon Route 53, or Amazon VPC and you have set up logs from those services to go to CloudWatch Logs. For more information about sending logs to CloudWatch Logs, see [Getting started with CloudWatch Logs](CWL_GettingStarted.md). Queries in CloudWatch Logs Insights return either a set of fields from log events or the result of a mathematical aggregation or other operation performed on log events. This tutorial demonstrates a query that returns a list of log events. ## Run a sample query **To run a CloudWatch Logs Insights sample query** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, and then choose **Logs Insights**. On the **Logs Insights** page, the query editor contains a default query in Logs Insights QL that returns the 20 most recent log events. 1. In the **Select log group(s)** drop down, choose one or more log groups to query. If this is a monitoring account in CloudWatch cross-account observability, you can select log groups in the source accounts as well as the monitoring account. A single query can query logs from different accounts at once. You can filter the log groups by log group name, account ID, or account label. When you select a log group in the Standard log class, CloudWatch Logs Insights automatically detects data fields in the group. To see discovered fields, select the **Fields** menu near the top right of the page. **Note** Discovered fields is supported only for log groups in the Standard log class. For more information about log classes, see [Log classes](CloudWatch_Logs_Log_Classes.md). 1. (Optional) Use the time interval selector to select a time period that you want to query. You can choose between 5 and 30-minute intervals; 1, 3, and 12-hour intervals; or a custom time frame. 1. Choose **Run** to view the results. For this tutorial, the results include the 20 most recently added log events. CloudWatch Logs displays a bar graph of log events in the log group over time. The bar graph shows not only the events in the table, but also the distribution of events in the log group that match the query and timeframe. 1. To see all fields for a returned log event, choose the triangular dropdown icon left of the numbered event. ## Modify the sample query In this tutorial, you modify the sample query to show the 50 most recent log events. If you haven't already run the previous tutorial, do that now. This tutorial starts where that previous tutorial ends. **Note** Some sample queries provided with CloudWatch Logs Insights use `head` or `tail` commands instead of `limit`. These commands are being deprecated and have been replaced with `limit`. Use `limit` instead of `head` or `tail` in all queries that you write. **To modify the CloudWatch Logs Insights sample query** 1. In the query editor, change **20** to **50**, and then choose **Run**. The results of the new query appear. Assuming there is enough data in the log group in the default time range, there are now 50 log events listed. 1. (Optional) You can save queries that you have created. To save this query, choose **Save**. For more information, see [Save and re-run CloudWatch Logs Insights queries](CWL_Insights-Saving-Queries.md). ## Add a filter command to the sample query This tutorial shows how to make a more powerful change to the query in the query editor. In this tutorial, you filter the results of the previous query based on a field in the retrieved log events. If you haven't already run the previous tutorials, do that now. This tutorial starts where that previous tutorial ends. **To add a filter command to the previous query** 1. Decide on a field to filter. To see the most common fields that CloudWatch Logs has detected in the log events contained in the selected log groups in the past 15 minutes, and the percentage of those log events in which each field appears, select **Fields** on the right side of the page. To see the fields contained in a particular log event, choose the icon to the left of that row. The **awsRegion** field might appear in your log event, depending on which events are in your logs. For the rest of this tutorial, we use **awsRegion** as the filter field, but you can use a different field if that field isn't available. 1. In the query editor box, place your cursor after **50** and press Enter. 1. On the new line, first enter \$1 (the pipe character) and a space. Commands in a CloudWatch Logs Insights query must be separated by the pipe character. 1. Enter **filter awsRegion="us-east-1"**. 1. Choose **Run**. The query runs again, and now displays the 50 most recent results that match the new filter. If you filtered on a different field and got an error result, you might need to escape the field name. If the field name includes non-alphanumeric characters, you must put backtick characters (`) before and after the field name (for example, **`error-code`="102"**). You must use the backtick characters for field names that contain non-alphanumeric characters, but not for values. Values are always contained in quotation marks ("). Logs Insights QL includes powerful query abilities, including several commands and support for regular expressions, mathematical, and statistical operations. For more information, see [CloudWatch Logs Insights language query syntax](CWL_QuerySyntax.md). # Tutorial: Run a query with an aggregation function You can use aggregation functions with the `stats` command and as arguments for other functions. In this tutorial, you run a query command that counts the number of log events containing a specified field. The query command returns a total count that's grouped by the specified field's value or values. For more information about aggregation functions, see [Supported operations and functions](https://docs.aws.amazon.com/en_us/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html#CWL_QuerySyntax-operations-functions) in the *Amazon CloudWatch Logs User Guide*. **To run a query with an aggregation function** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, and then choose **Logs Insights**. 1. Confirm that the **Logs Insights QL** tab is selected. 1. In the **Select log group(s)** drop down, choose one or more log groups to query. If this is a monitoring account in CloudWatch cross-account observability, you can select log groups in the source accounts as well as the monitoring account. A single query can query logs from different accounts at once. You can filter the log groups by log group name, account ID, or account label. When you select a log group, CloudWatch Logs Insights automatically detects data fields in the log group if it is a Standard class log group. To see discovered fields, select the **Fields** menu near the top right of the page. 1. Delete the default query in the query editor, and enter the following command: ``` stats count(*) by fieldName ``` 1. Replace *fieldName* with a discovered field from the **Fields** menu. The **Fields** menu is located at the top right of the page and displays all of the discovered fields that CloudWatch Logs Insights detects in your log group. 1. Choose **Run** to view the query results. The query results show the number of records in your log group that match the query command and the total count that's grouped by the specified field's value or values. # Tutorial: Run a query that produces a visualization grouped by log fields When you run a query that uses the `stats` function to group the returned results by the values of one or more fields in the log entries, you can view the results as a bar chart, pie chart, line graph or stacked area graph. This helps you more efficiently visualize trends in your logs. **To run a query for visualization** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, and then choose **Logs Insights**. 1. In the **Select log group(s)** drop down, choose one or more log groups to query. If this is a monitoring account in CloudWatch cross-account observability, you can select log groups in the source accounts as well as the monitoring account. A single query can query logs from different accounts at once. You can filter the log groups by log group name, account ID, or account label. 1. In the query editor, delete the current contents, enter the following `stats` function, and then choose **Run query**. ``` stats count(*) by @logStream | limit 100 ``` The results show the number of log events in the log group for each log stream. The results are limited to only 100 rows. 1. Choose the **Visualization** tab. 1. Select the arrow next to **Line**, and then choose **Bar**. The bar chart appears, showing a bar for each log stream in the log group. # Tutorial: Run a query that produces a time series visualization When you run a query that uses the `bin()` function to group the returned results by a time period, you can view the results as a line graph, stacked area graph, pie chart, or bar chart. This helps you more efficiently visualize trends in log events over time. **To run a query for visualization** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, and then choose **Logs Insights**. 1. Confirm that the **Logs Insights QL** tab is selected. 1. In the **Select log group(s)** drop down, choose one or more log groups to query. If this is a monitoring account in CloudWatch cross-account observability, you can select log groups in the source accounts as well as the monitoring account. A single query can query logs from different accounts at once. You can filter the log groups by log group name, account ID, or account label. 1. In the query editor, delete the current contents, enter the following `stats` function, and then choose **Run query**. ``` stats count(*) by bin(30s) ``` The results show the number of log events in the log group that were received by CloudWatch Logs for each 30-second period. 1. Choose the **Visualization** tab. The results are shown as a line graph. To switch to a bar chart, pie chart, or stacked area chart, choose the arrow next to **Line** at the upper left of the graph. # Sample queries This section contains a list of general and useful query commands that you can run in the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/). For information about how to run a query command, see [Tutorial: Run and modify a sample query](https://docs.aws.amazon.com/en_us/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_RunSampleQuery.html) in the *Amazon CloudWatch Logs User Guide*. For more information about query syntax, see [CloudWatch Logs Insights language query syntax](CWL_QuerySyntax.md). **Topics** + [ ## General queries ](#CWL_QuerySyntax-examples-general) + [ ## Queries for Lambda logs ](#CWL_QuerySyntax-examples-Lambda) + [ ## Queries for Amazon VPC flow logs ](#CWL_QuerySyntax-examples-VPC) + [ ## Queries for Route 53 logs ](#CWL_QuerySyntax-examples-Route53) + [ ## Queries for CloudTrail logs ](#CWL_QuerySyntax-examples-CloudTrail) + [ ## Queries for Amazon API Gateway ](#CWL_QuerySyntax-examples-APIGateway) + [ ## Queries for NAT gateway ](#CWL_QuerySyntax-examples-NATGateway) + [ ## Queries for Apache server logs ](#CWL_QuerySyntax-examples-Apache) + [ ## Queries for Amazon EventBridge ](#CWL_QuerySyntax-examples-EventBridge) + [ ## Examples of the parse command ](#CWL_QuerySyntax-examples-parse) ## General queries **Find the 25 most recently added log events.** ``` fields @timestamp, @message | sort @timestamp desc | limit 25 ``` **Get a list of the number of exceptions per hour.** ``` filter @message like /Exception/ | stats count(*) as exceptionCount by bin(1h) | sort exceptionCount desc ``` **Get a list of log events that aren't exceptions.** ``` fields @message | filter @message not like /Exception/ ``` **Get the most recent log event for each unique value of the `server` field.** ``` fields @timestamp, server, severity, message | sort @timestamp asc | dedup server ``` **Get the most recent log event for each unique value of the `server` field for each `severity` type.** ``` fields @timestamp, server, severity, message | sort @timestamp desc | dedup server, severity ``` ## Queries for Lambda logs **Determine the amount of overprovisioned memory.** ``` filter @type = "REPORT" | stats max(@memorySize / 1000 / 1000) as provisonedMemoryMB, min(@maxMemoryUsed / 1000 / 1000) as smallestMemoryRequestMB, avg(@maxMemoryUsed / 1000 / 1000) as avgMemoryUsedMB, max(@maxMemoryUsed / 1000 / 1000) as maxMemoryUsedMB, provisonedMemoryMB - maxMemoryUsedMB as overProvisionedMB ``` **Create a latency report.** ``` filter @type = "REPORT" | stats avg(@duration), max(@duration), min(@duration) by bin(5m) ``` **Search for slow function invocations, and eliminate duplicate requests that can arise from retries or client-side code. In this query, `@duration` is in milliseconds.** ``` fields @timestamp, @requestId, @message, @logStream | filter @type = "REPORT" and @duration > 1000 | sort @timestamp desc | dedup @requestId | limit 20 ``` ## Queries for Amazon VPC flow logs **Find the top 15 packet transfers across hosts:** ``` stats sum(packets) as packetsTransferred by srcAddr, dstAddr | sort packetsTransferred desc | limit 15 ``` **Find the top 15 byte transfers for hosts on a given subnet.** ``` filter isIpv4InSubnet(srcAddr, "192.0.2.0/24") | stats sum(bytes) as bytesTransferred by dstAddr | sort bytesTransferred desc | limit 15 ``` **Find the IP addresses that use UDP as a data transfer protocol.** ``` filter protocol=17 | stats count(*) by srcAddr ``` **Find the IP addresses where flow records were skipped during the capture window.** ``` filter logStatus="SKIPDATA" | stats count(*) by bin(1h) as t | sort t ``` **Find a single record for each connection, to help troubleshoot network connectivity issues.** ``` fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, protocol, bytes | filter logStream = 'vpc-flow-logs' and interfaceId = 'eni-0123456789abcdef0' | sort @timestamp desc | dedup srcAddr, dstAddr, srcPort, dstPort, protocol | limit 20 ``` ## Queries for Route 53 logs **Find the distribution of records per hour by query type.** ``` stats count(*) by queryType, bin(1h) ``` **Find the 10 DNS resolvers with the highest number of requests.** ``` stats count(*) as numRequests by resolverIp | sort numRequests desc | limit 10 ``` **Find the number of records by domain and subdomain where the server failed to complete the DNS request.** ``` filter responseCode="SERVFAIL" | stats count(*) by queryName ``` ## Queries for CloudTrail logs **Find the number of log entries for each service, event type, and AWS Region.** ``` stats count(*) by eventSource, eventName, awsRegion ``` **Find the Amazon EC2 hosts that were started or stopped in a given AWS Region.** ``` filter (eventName="StartInstances" or eventName="StopInstances") and awsRegion="us-east-2" ``` **Find the AWS Regions, user names, and ARNs of newly created IAM users.** ``` filter eventName="CreateUser" | fields awsRegion, requestParameters.userName, responseElements.user.arn ``` **Find the number of records where an exception occurred while invoking the API `UpdateTrail`.** ``` filter eventName="UpdateTrail" and ispresent(errorCode) | stats count(*) by errorCode, errorMessage ``` **Find log entries where TLS 1.0 or 1.1 was used** ``` filter tlsDetails.tlsVersion in [ "TLSv1", "TLSv1.1" ] | stats count(*) as numOutdatedTlsCalls by userIdentity.accountId, recipientAccountId, eventSource, eventName, awsRegion, tlsDetails.tlsVersion, tlsDetails.cipherSuite, userAgent | sort eventSource, eventName, awsRegion, tlsDetails.tlsVersion ``` **Find the number of calls per service that used TLS versions 1.0 or 1.1** ``` filter tlsDetails.tlsVersion in [ "TLSv1", "TLSv1.1" ] | stats count(*) as numOutdatedTlsCalls by eventSource | sort numOutdatedTlsCalls desc ``` ## Queries for Amazon API Gateway Find the last 10 4XX errors ``` fields @timestamp, status, ip, path, httpMethod | filter status>=400 and status<=499 | sort @timestamp desc | limit 10 ``` Identify the 10 longest-running Amazon API Gateway requests in your Amazon API Gateway access log group ``` fields @timestamp, status, ip, path, httpMethod, responseLatency | sort responseLatency desc | limit 10 ``` Return the list of the most popular API paths in your Amazon API Gateway access log group ``` stats count(*) as requestCount by path | sort requestCount desc | limit 10 ``` Create an integration latency report for your Amazon API Gateway access log group ``` filter status=200 | stats avg(integrationLatency), max(integrationLatency), min(integrationLatency) by bin(1m) ``` ## Queries for NAT gateway If you notice higher than normal costs in your AWS bill, you can use CloudWatch Logs Insights to find the top contributors. For more information about the following query commands, see [How can I find the top contributors to traffic through the NAT gateway in my VPC?](https://aws.amazon.com/premiumsupport/knowledge-center/vpc-find-traffic-sources-nat-gateway/) at the AWS premium support page. **Note** In the following query commands, replace "x.x.x.x" with the private IP of your NAT gateway, and replace "y.y" with the first two octets of your VPC CIDR range. **Find the instances that are sending the most traffic through your NAT gateway.** ``` filter (dstAddr like 'x.x.x.x' and srcAddr like 'y.y.') | stats sum(bytes) as bytesTransferred by srcAddr, dstAddr | sort bytesTransferred desc | limit 10 ``` **Determine the traffic that's going to and from the instances in your NAT gateways.** ``` filter (dstAddr like 'x.x.x.x' and srcAddr like 'y.y.') or (srcAddr like 'xxx.xx.xx.xx' and dstAddr like 'y.y.') | stats sum(bytes) as bytesTransferred by srcAddr, dstAddr | sort bytesTransferred desc | limit 10 ``` **Determine the internet destinations that the instances in your VPC communicate with most often for uploads and downloads.** *****For uploads***** ``` filter (srcAddr like 'x.x.x.x' and dstAddr not like 'y.y.') | stats sum(bytes) as bytesTransferred by srcAddr, dstAddr | sort bytesTransferred desc | limit 10 ``` *****For downloads***** ``` filter (dstAddr like 'x.x.x.x' and srcAddr not like 'y.y.') | stats sum(bytes) as bytesTransferred by srcAddr, dstAddr | sort bytesTransferred desc | limit 10 ``` ## Queries for Apache server logs You can use CloudWatch Logs Insights to query Apache server logs. For more information about the following queries, see [Simplifying Apache server logs with CloudWatch Logs Insights](https://aws.amazon.com/blogs/mt/simplifying-apache-server-logs-with-amazon-cloudwatch-logs-insights/) at the AWS Cloud Operations & Migrations Blog. **Find the most relevant fields, so you can review your access logs and check for traffic in the */admin* path of your application.** ``` fields @timestamp, remoteIP, request, status, filename| sort @timestamp desc | filter filename="/var/www/html/admin" | limit 20 ``` **Find the number unique GET requests that accessed your main page with status code "200" (success).** ``` fields @timestamp, remoteIP, method, status | filter status="200" and referrer= http://34.250.27.141/ and method= "GET" | stats count_distinct(remoteIP) as UniqueVisits | limit 10 ``` **Find the number of times your Apache service restarted.** ``` fields @timestamp, function, process, message | filter message like "resuming normal operations" | sort @timestamp desc | limit 20 ``` ## Queries for Amazon EventBridge Get the number of EventBridge events grouped by event detail type ``` fields @timestamp, @message | stats count(*) as numberOfEvents by `detail-type` | sort numberOfEvents desc ``` ## Examples of the parse command **Use a glob expression to extract the fields `@user`, `@method`, and `@latency` from the log field `@message` and return the average latency for each unique combination of `@method` and `@user`.** ``` parse @message "user=*, method:*, latency := *" as @user, @method, @latency | stats avg(@latency) by @method, @user ``` **Use a regular expression to extract the fields `@user2`, `@method2`, and `@latency2` from the log field `@message` and return the average latency for each unique combination of `@method2` and `@user2`.** ``` parse @message /user=(?.*?), method:(?.*?), latency := (?.*?)/ | stats avg(latency2) by @method2, @user2 ``` **Extracts the fields `loggingTime`, `loggingType` and `loggingMessage`, filters down to log events that contain `ERROR` or `INFO` strings, and then displays only the `loggingMessage` and `loggingType` fields for events that contain an `ERROR` string.** ``` FIELDS @message | PARSE @message "* [*] *" as loggingTime, loggingType, loggingMessage | FILTER loggingType IN ["ERROR", "INFO"] | DISPLAY loggingMessage, loggingType = "ERROR" as isError ``` # Compare (diff) with previous time ranges You can use CloudWatch Logs Insights with the Logs Insights QL to compare changes in your log events over time. You can compare the log events ingested during a recent time range with the logs from the immediately previous time period. Alternatively, you can compare with similar past time periods. This can help you find whether an error in your logs was recently introduced or was already occurring, and can help you find other trends. Comparison queries return only patterns in the results, not raw log events. The patterns returned will help you quickly see the trends and changes in the log events over time. After you run a comparison query and have the pattern results, you can see sample raw log events for the patterns that you're interested in. For more information about log patterns, see [Pattern analysis](CWL_AnalyzeLogData_Patterns.md). When you run a comparison query, your query is analyzed against two different time periods: the original query period that you select, and the comparison period. The comparison period is always of equal length to your original query period. The default time intervals for the comparisons are the following. + **Previous period**— Compares to the time period immediately before your query time period. + **Previous day**— Compares to the time period one day before your query time period. + **Previous week**— Compares to the time period one week before your query time period. + **Previous month**— Compares to the time period one month before your query time period. **Note** Queries using comparisons incur charges similar to running a single CloudWatch Logs Insights query over the combined time range. For more information, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). **To run a comparison query** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, **Logs Insights**. A default query appears in the query box. 1. Confirm that the **Logs Insights QL** tab is selected. 1. Keep the default query or enter a different query. 1. In the **Select log group(s)** drop-down, choose one or more log groups to query. 1. (Optional) Use the time interval selector to select a time period that you want to query. The default query is for the previous hour of log data. 1. By the time range selector, choose **Compare**. Then choose the previous time period that you want to compare the original logs with, and choose **Apply**. 1. Choose **Run query**. To cause the query to fetch the data from the comparison period, the `diff` command is appended to your query. 1. Choose the **Patterns** tab to see the results. The table displays the following information: + Each **Pattern**, with variable parts of the pattern replaced by the dynamic token symbol ``. The *string* is a description of the type of data that the token represents. The *number* shows where in the pattern this token appears, compared to the other dynamic tokens. For more information, see [Pattern analysis](CWL_AnalyzeLogData_Patterns.md). + **Event count** is the number of log events with that pattern in the original, more current time period. + **Difference event count** is the difference between the number of matching log events in the current time period versus the comparison time period. A positive different means there are more such events in the current time period. + **Difference description** briefly summarizes the change in that pattern between the current time period and the comparison period. + **Severity type** is the probable severity of the logs events with this pattern, based on words found in the log events such as `FATAL`, `ERROR`, and `WARN`. 1. To further inspect one of the patterns in the list, choose the icon in the **Inspect** column for one of the patterns. The **Pattern inspect** pane appears and displays the following: + The **Pattern**. Select a token within the pattern to analyze that token's values. + A histogram showing the number of occurrences of the pattern over the queried time range. This can help you to identify interesting trends such as a sudden increase in occurrence of a pattern. + The **Log samples** tab displays a few of the log events that match the selected pattern. + The **Token Values** tab displays the values of the selected dynamic token, if you have selected one. **Note** A maximum of 10 token values is captured for each token. Token counts might not be precise. CloudWatch Logs uses a probabilistic counter to generate the token count, not the absolute value. + The **Related patterns** tab displays other patterns that frequently occurred near the same time as the pattern that you are inspecting. For example, if a pattern for an `ERROR` message was usually accompanied by another log event marked as `INFO` with additional details, that pattern is displayed here. # Visualize log data in graphs You can use visualizations such as bar charts, line charts, and stacked area charts to more efficiently identify patterns in your log data. CloudWatch Logs Insights generates visualizations for queries that use the `stats` function and one or more aggregation functions. For more information, see [stats](CWL_QuerySyntax-Stats.md).