# Transform logs during ingestion With logs transformation and enrichment, you can normalize all your logs in a consistent and context-rich format at the time of ingestion into CloudWatch Logs. You can add structure to your logs by using pre-configured templates for common AWS services such as AWS WAF and Amazon Route 53, or build custom transformers with native parsers such as [Grok](CloudWatch-Logs-Transformation-Configurable.md#CloudWatch-Logs-Transformation-Grok). You can also rename existing attributes and add additional metadata to your logs such as account ID, and Region. Log transformation helps simplify and shorten your log queries across your applications, and helps simplify creating alerts on your logs. This feature provides transformation for common log types with out-of-the-box transformation templates for major AWS log sources like VPC Flow logs, Route 53, and Amazon RDS for PostgreSQL. You can use pre-configured transformation templates or create custom transformers to suit your needs. Log transformation helps you manage logs emitted from various sources that vary widely in format and attribute names. After you create a transformer, ingested log events are converted and stored in a standard format. You can leverage these transformed logs to accelerate your analytics experience with the following features: + [Field indexes](CloudWatchLogs-Field-Indexing.md) + [CloudWatch Logs Insights discovered fields](CWL_AnalyzeLogData-discoverable-fields.md) + Flexibility in alarming using [metric filters](MonitoringLogData.md) + Forwarding via [subscription filters](Subscriptions.md) + Creating metric data from log events with [ Contributor Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContributorInsights.html), where you can choose to have the Contributor Insights rule evaluate log events either before or after they are transformed. Transformations happen only during log ingestion. You can't transform log events that have already been ingested. Transformations are not reversible. Both original and transformed logs are stored in CloudWatch Logs with the same retention policy. Log transformation and enrichment capability is included in the existing Standard log class ingestion price. Log storage costs will be based on log size after transformation, which might exceed the original log volume. **Important** After log events have been transformed, you must use CloudWatch Logs Insights queries to view the transformed versions of the logs. The [ GetLogEvents](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html) and [ FilterLogEvents](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_FilterLogEvents.html) actions return only the original versions of the log events, before they were transformed. **Important** Despite a single log event in PutLogEvents allowing upto 1MB, logs transformation can only handle log event of size less than 512kb. Any log events greather than 512kb will fail in transformation and emit an error. Total size of PutLogEvents can still go over 512kb. In addition to transforming into different formats, you can also enrich your logs with additional context, such as account ID, Region, and keyword. These are extracted from the log group name and from static keywords. Log transformation helps you with logs emitted from various sources that vary widely in format and attribute names. Log transformation and enrichment is supported only for log groups in the Standard log class. You can create transformers for individual log groups, and you can also create account-level transformers that apply to all or many log groups in your account. If a log group has a log group-level transformer, that transformer overrides any account-level transformer that would otherwise apply to that log group. **Topics** + [Create and manage log transformers](CloudWatch-Logs-Transformation-Create.md) + [Configurable parser-type processors](CloudWatch-Logs-Transformation-Configurable.md) + [Built-in processors for AWS vended logs](CloudWatch-Logs-Transformation-BuiltIn.md) + [String mutate processors](CloudWatch-Logs-Transformation-StringMutate.md) + [JSON mutate processors](CloudWatch-Logs-Transformation-JSONMutate.md) + [Datatype converter processors](CloudWatch-Logs-Transformation-Datatype.md) + [Transformation metrics and errors](Transformation-Errors-Metrics.md) # Create and manage log transformers A log transformer includes one or more *processors* that are in a logical pipeline together. Each processor is applied to a log event, one after the other in the order that they are listed in the transformer configuration. Some processors are of the *parser* type. Each transformer must have at least one parser, and the first processor in a transformer must be a parser. Some of the parsers are built-in parsers that are configured for a certain type of AWS vended log. Other processor types are string mutators, JSON mutators, and data processors. You can create transformers for individual log groups, and you can also create account-level transformers that apply to all or many log groups in your account. If a log group has a log group-level transformer, that transformer overrides any account-level transformer that would otherwise apply to that log group. You can have as many as 20 account-level transformers in a Region in your account. You must follow these guidelines when you create a transformer: + If you include a pre-configured parser for a type of AWS vended logs, it must be the first processor listed in the transformer. You can include only one such processor in a transformer. + You can include only one `grok` processor in a transformer. + You must have at least one parser-type processor in a transformer. You can include as many as five parser-type processors. This limit of five includes both built-in parsers and configurable parsers. + You can have as many as 20 processors in a transformer. + You can include only one **addKeys** processor in a transformer. + You can include only one **copyValue** processor in a transformer. + Each transformer can extract up to 200 fields from a log event. + Each log event **MUST** be below 512KB. Total size of log events can still go over 512KB. **Topics** + [Create an account-level transformer policy](CloudWatchLogs-Transformer-CreateAccountLevel.md) + [Edit or delete an account-level transformer policy](CloudWatchLogs-Transformer-EditAccountLevel.md) + [Create a log-group-level log transformer from scratch](CloudWatch-Logs-Transformation-CreateNew.md) + [Create a log-group-level transformer by copying an existing one](CloudWatch-Logs-Transformation-Copy.md) + [Edit a log-group-level transformer](CloudWatch-Logs-Transformation-Edit.md) + [Delete a log-group-level transformer](CloudWatch-Logs-Transformation-Delete.md) # Create an account-level transformer policy Use the steps in this section to create a transformer policy that applies to all log groups in the account, or to multiple log groups that have log group names that start with the same string (prefix). You can have as many as 20 account-level transformer policies in a Region. You can't create two transformer policies in the same Region that use the same prefix or have one prefix contained within another. For example, if you create one transformer policy for the string prefix `/aws/lambda`, you can't create another with the prefix `/aws`. But you could have one transformer for `/aws/lambda` and another for `/aws/waf` **To create an account-level transformer policy** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the left navigation pane, choose **Settings** and then choose the **Logs** tab. 1. In the **Transformer policy for account** section, choose **Create transformer policy**. 1. For **Transformer policy name**, enter a name for your new poiicy. 1. For **Select log groups**, do one of the following: + Choose **All standard log groups** to have the transformer policy apply to all Standard Class log groups in the account. + choose **Log groups by prefix match** to apply the policy to a subset of log groups that all have names that start with the same string. Then, enter the prefix for these log groups in **Selection criteria**. 1. In the **Select parsers** area, use **Parsers** to select a parser to include in your transformer. If it is a pre-configured parser for a type of AWS vended log, you don't have to specify any configuration for it. If it is a different parser, you need to specify its configuration. For more information, see the information for that processor in [Configurable parser-type processors](CloudWatch-Logs-Transformation-Configurable.md). 1. To add another processor, choose **Select processor**. Then select the processor that you want in the **Processor** box, and fill in the configuration parameters. Remember that processors operate on the log events in the order that you add them to the transformer. 1. (Optional) To add additional processors, choose **\$1 Processor** and repeat the previous step. 1. (Optional) At any time, you can test the transformer that you have built so far on a sample log event. To do so, do one of the following in the **Transformer preview** section: + Select as many as five log groups in **Select log groups** and then choose **Load latest log events**. Then choose **Test transformer**. + Copy log events directly into **Sample log events** and then choose **Test transformer**. The transformed version of the log then appears. 1. When you are finished adding processors and satisfied with the tests on sample logs, choose **Save**. 1. When you have finished, choose **Create**. # Edit or delete an account-level transformer policy Use the steps in this section to edit or delete an account-level transformer policy. **To edit or delete an account-level transformer policy** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the left navigation pane, choose **Settings** and then choose the **Logs** tab. 1. In the **Transformer account policy** section, choose **Manage**. 1. Select the button by the transformer policy that you want to manage, and then choose **Edit** or **Delete**. If you're editing the policy, see steps 5-11 in [Configurable parser-type processors](CloudWatch-Logs-Transformation-Configurable.md) to see your options. # Create a log-group-level log transformer from scratch Use these steps to create a log-group-level transformer from scratch. **To use the console to create a log transformer for a log group** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, **Log groups**. 1. Choose the log group that you want to create the transformer for. 1. Choose the **Transformer** tab. You might have to scroll the tab list to the right to see it. 1. Choose **Create transformer**. 1. In the **Choose a parser** box, select a parser to include in your transformer. If it is a pre-configured parser for a type of AWS vended log, you don't have to specify any configuration for it. If it is a different parser, you need to specify its configuration. For more information, see the information for that processor in [Configurable parser-type processors](CloudWatch-Logs-Transformation-Configurable.md). 1. To add another processor, choose **\$1 Add processor**. Then select the processor that you want in the **Choose processors** box, and fill in the configuration parameters. Remember that processors operate on the log events in the order that you add them to the transformer. 1. (Optional) At any time, you can test the transformer that you have built so far on a sample log event. To do so, do the following: 1. In the **Transformation preview** section, either choose **Load sample log** to load a sample log event from the log group that this transformer is for, or paste a log event into the text box. Choose **Test transformer**. The transformed version of the log appears 1. When you are finished adding processors and satisfied with the tests on sample logs, choose **Save**. **To use the AWS CLI to create a log transformer from scratch** + Use the `aws logs put-transformer` command. When using `parseJSON` as the first processor, you must parse the entire log event using `@message` as the source field. After the initial JSON parsing, you can then manipulate specific fields in subsequent processors. The following is an example that creates a transformer that includes the `parseJSON` and `addKeys` processors: ``` aws logs put-transformer \ --transformer-config '[{"parseJSON":{"source":"@message"}},{"addKeys":{"entries":[{"key":"metadata.transformed_in","value":"CloudWatchLogs"},{"key":"feature","value":"Transformation"}]}},{"trimString":{"withKeys":["status"]}}]' \ --log-group-identifier my-log-group-name ``` # Create a log-group-level transformer by copying an existing one You can use the console to copy the JSON configuration of an existing transformer. You can then use that code to create an identical transformer by using the AWS CLI, or you can modify the configuration first. **To create a log transformer by copying an existing one** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, **Log groups**. 1. Choose the log group that has the transformer that you want to copy. 1. Choose the **Transformations** tab. You might have to scroll the tab list to the right to see it. 1. Choose **Manage transformer**. 1. Choose **Copy transformer**. This copies the transformer JSON to your clipboard. 1. Create a file and paste in the transformer configuration. In this example we'll call the file `CopiedTransformer.json` 1. Use the AWS CLI to create a new transformer with that configuration. ``` aws logs put-transformer --log-group-identifier my-log-group-name \ --transformer-config file://CopiedTransformer.json ``` # Edit a log-group-level transformer Use these steps to edit an existing log transformer. **To edit a log transformer** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, **Log groups**. 1. Choose the log group that has the transformer that you want to edit. 1. Choose the **Transformations** tab. You might have to scroll the tab list to the right to see it. 1. Choose **Manage transformer**. 1. In the **Parsers** and **Processors** sections, make your changes. 1. To add another processor, choose **\$1 Add Processor**. Then select the processor that you want in the **Processor** box, and fill in the configuration parameters. Remember that processors operate on the log events in the order that you add them to the transformer. 1. (Optional) At any time, you can test the transformer that you have built so far on a sample log event. To do so, do the following: 1. In the **Transformation Preview** section, either choose **Load Sample Log** to load a sample log event from the log group that this transformer is for, or paste a log event into the text box. Choose **Test Transformation**. The transformed version of the log appears 1. When you are finished adding processors and satisfied with the tests on sample logs, choose **Save**. # Delete a log-group-level transformer Use these steps to delete a log transformer. **To delete a log transformer** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, **Log groups**. 1. Choose the log group that has the transformer that you want to edit. 1. Choose the **Transformations** tab. You might have to scroll the tab list to the right to see it. 1. Choose **Delete**. 1. In the confirmation box, choose **Delete Policy**. # Configurable parser-type processors This section contains information about the configurable data parser processors that you can use in a log event transformer. **Contents** + [parseJSON](#CloudWatch-Logs-Transformation-parseJSON) + [grok](#CloudWatch-Logs-Transformation-Grok) + [Grok examples](#Grok-Examples) + [Example 1: Use grok to extract a field from unstructured logs](#Grok-Example1) + [Example 2: Use grok in combination with parseJSON to extract fields from a JSON log event](#Grok-Example3) + [Example 3: Grok pattern with dotted annotation in FIELD\$1NAME](#Grok-Example4) + [Supported grok patterns](#Grok-Patterns) + [Common log format examples](#Common-Log-Examples) + [Apache log example](#Apache-Log-Example) + [NGINX log example](#NGINX-Log-Example) + [Syslog Protocol (RFC 5424) log example](#syslog5424-Log-Example) + [csv](#CloudWatch-Logs-Transformation-csv) + [parseKeyValue](#CloudWatch-Logs-Transformation-parseKeyValue) ## parseJSON The **parseJSON** processor parses JSON log events and inserts extracted JSON key-value pairs under the destination. If you don't specify a destination, the processor places the key-value pair under the root node. When using `parseJSON` as the first processor, you must parse the entire log event using `@message` as the source field. After the initial JSON parsing, you can then manipulate specific fields in subsequent processors. The original `@message` content is not changed, the new keys are added to the message. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | source | Path to the field in the log event that will be parsed. Use dot notation to access child fields. For example, store.book | No | `@message` | Maximum length: 128 Maximum nested key depth: 3 | | destination | The destination field of the parsed JSON | No | `Parent JSON node` | Maximum length: 128 Maximum nested key depth: 3 | **Example** Suppose an ingested log event looks like this: ``` { "outer_key": { "inner_key": "inner_value" } } ``` Then if we have this **parseJSON** processor: ``` [ { "parseJSON": { "destination": "new_key" } } ] ``` The transformed log event would be the following. ``` { "new_key": { "outer_key": { "inner_key": "inner_value" } } } ``` ## grok Use the grok processor to parse and structure unstructured data using pattern matching. This processor can also extract fields from log messages. | Field | Description | Required? | Default | Limits | Notes | | --- | --- | --- | --- | --- | --- | | source | Path of the field to apply Grok matching on | No | `@message` | Maximum length: 128 Maximum nested key depth: 3 | | match | The grok pattern to match against the log event | Yes | | Maximum length: 512 Maximum grok patterns: 20 Some grok pattern types have individual usage limits. Any combination of the following patterns can be used as many as five times: \$1URI, URIPARAM, URIPATHPARAM, SPACE, DATA, GREEDYDATA, GREEDYDATA\$1MULTILINE\$1 Grok patterns don't support type conversions. For common log format patterns (APACHE\$1ACCESS\$1LOG, NGINX\$1ACCESS\$1LOG, SYSLOG5424), only DATA, GREEDYDATA, or GREEDYDATA\$1MULTILINE patterns are supported to be included after the common log pattern. | [See all supported Grok patterns](#Grok-Patterns) | **Structure of a Grok Pattern** This is the supported grok pattern structure: ``` %{PATTERN_NAME:FIELD_NAME} ``` + **PATTERN\$1NAME**: Refers to a pre-defined regular expression for matching a specific type of data. Only predefined [grok patterns](#Grok-Patterns) are supported. Creating custom patterns is not allowed. + **FIELD\$1NAME**: Assigns a name to the extracted value. `FIELD_NAME` is optional, but if you don't specify this value then the extracted data will be dropped from the transformed log event. If `FIELD_NAME` uses dotted notation (e.g., "parent.child"), it is considered as a JSON path. + **Type Conversion**: Explicit type conversions are not be supported. Use [TypeConverter processor](CloudWatch-Logs-Transformation-Datatype.md#CloudWatch-Logs-Transformation-typeConverter) to convert the datatype of any value extracted by grok. To create more complex matching expressions, you can combine several grok patterns. As many as 20 grok patterns can be combined to match a log event. For example, this combination of patterns `%{NUMBER:timestamp} [%{NUMBER:db} %{IP:client_ip}:%{NUMBER:client_port}] %{GREEDYDATA:data}` can be used to extract fields from a Redis slow log entry like this: `1629860738.123456 [0 127.0.0.1:6379] "SET" "key1" "value1"` ### Grok examples #### Example 1: Use grok to extract a field from unstructured logs Sample log: ``` 293750 server-01.internal-network.local OK "[Thread-000] token generated" ``` Transformer used: ``` [ { "grok": { "match": "%{NUMBER:version} %{HOSTNAME:hostname} %{NOTSPACE:status} %{QUOTEDSTRING:logMsg}" } } ] ``` Output: ``` { "version": "293750", "hostname": "server-01.internal-network.local", "status": "OK", "logMsg": "[Thread-000] token generated" } ``` Sample log: ``` 23/Nov/2024:10:25:15 -0900 172.16.0.1 200 ``` Transformer used: ``` [ { "grok": { "match": "%{HTTPDATE:timestamp} %{IPORHOST:clientip} %{NUMBER:response_status}" } } ] ``` Output: ``` { "timestamp": "23/Nov/2024:10:25:15 -0900", "clientip": "172.16.0.1", "response_status": "200" } ``` #### Example 2: Use grok in combination with parseJSON to extract fields from a JSON log event Sample log: ``` { "timestamp": "2024-11-23T16:03:12Z", "level": "ERROR", "logMsg": "GET /page.html HTTP/1.1" } ``` Transformer used: ``` [ { "parseJSON": {} }, { "grok": { "source": "logMsg", "match": "%{WORD:http_method} %{NOTSPACE:request} HTTP/%{NUMBER:http_version}" } } ] ``` Output: ``` { "timestamp": "2024-11-23T16:03:12Z", "level": "ERROR", "logMsg": "GET /page.html HTTP/1.1", "http_method": "GET", "request": "/page.html", "http_version": "1.1" } ``` #### Example 3: Grok pattern with dotted annotation in FIELD\$1NAME Sample log: ``` 192.168.1.1 GET /index.html?param=value 200 1234 ``` Transformer used: ``` [ { "grok": { "match": "%{IP:client.ip} %{WORD:method} %{URIPATHPARAM:request.uri} %{NUMBER:response.status} %{NUMBER:response.bytes}" } } ] ``` Output: ``` { "client": { "ip": "192.168.1.1" }, "method": "GET", "request": { "uri": "/index.html?param=value" }, "response": { "status": "200", "bytes": "1234" } } ``` ### Supported grok patterns The following tables list the patterns that are supported by the `grok` processor. **General grok patterns** | Grok Pattern | Description | Maximum pattern limit | Example | | --- | --- | --- | --- | | USERNAME or USER | Matches one or more characters that can include lowercase letters (a-z), uppercase letters (A-Z), digits (0-9), dots (.), underscores (\$1), or hyphens (-) | 20 | Input: `user123.name-TEST` Pattern: `%{USERNAME:name}` Output: `{"name": "user123.name-TEST"}` | | INT | Matches an optional plus or minus sign followed by one or more digits. | 20 | Input: `-456` Pattern: `%{INT:num}` Output: `{"num": "-456"}` | | BASE10NUM | Matches an integer or a floating-point number with optional sign and decimal point | 20 | Input: `-0.67` Pattern: `%{BASE10NUM:num}` Output: `{"num": "-0.67"}` | | BASE16NUM | Matches decimal and hexadecimal numbers with an optional sign (\$1 or -) and an optional 0x prefix | 20 | Input: `+0xA1B2` Pattern: `%{BASE16NUM:num}` Output: `{"num": "+0xA1B2"}` | | POSINT | Matches whole positive integers without leading zeros, consisting of one or more digits (1-9 followed by 0-9) | 20 | Input: `123` Pattern: `%{POSINT:num}` Output: `{"num": "123"}` | | NONNEGINT | Matches any whole numbers (consisting of one or more digits 0-9) including zero and numbers with leading zeros. | 20 | Input: `007` Pattern: `%{NONNEGINT:num}` Output: `{"num": "007"}` | | WORD | Matches whole words composed of one or more word characters (\$1w), including letters, digits, and underscores | 20 | Input: `user_123` Pattern: `%{WORD:user}` Output: `{"user": "user_123"}` | | NOTSPACE | Matches one or more non-whitespace characters. | 5 | Input: `hello_world123` Pattern: `%{NOTSPACE:msg}` Output: `{"msg": "hello_world123"}` | | SPACE | Matches zero or more whitespace characters. | 5 | Input: `" "` Pattern: `%{SPACE:extra}` Output: `{"extra": " "}` | | DATA | Matches any character (except newline) zero or more times, non-greedy. | 5 | Input: `abc def ghi` Pattern: `%{DATA:x} %{DATA:y}` Output: `{"x": "abc", "y": "def ghi"}` | | GREEDYDATA | Matches any character (except newline) zero or more times, greedy. | 5 | Input: `abc def ghi` Pattern: `%{GREEDYDATA:x} %{GREEDYDATA:y}` Output: `{"x": "abc def", "y": "ghi"}` | | GREEDYDATA\$1MULTILINE | Matches any character (including newline) zero or more times, greedy. | 1 | Input: `abc` `def` `ghi` Pattern: `%{GREEDYDATA_MULTILINE:data}` Output: `{"data": "abc\ndef\nghi"}` | | QUOTEDSTRING | Matches quoted strings (single or double quotes) with escaped characters. | 20 | Input: `"Hello, world!"` Pattern: `%{QUOTEDSTRING:msg}` Output: `{"msg": "Hello, world!"}` | | UUID | Matches a standard UUID format: 8 hexadecimal characters, followed by three groups of 4 hexadecimal characters, and ending with 12 hexadecimal characters, all separated by hyphens. | 20 | Input: `550e8400-e29b-41d4-a716-446655440000` Pattern: `%{UUID:id}` Output: `{"id": "550e8400-e29b-41d4-a716-446655440000"}` | | URN | Matches URN (Uniform Resource Name) syntax. | 20 | Input: `urn:isbn:0451450523` Pattern: `%{URN:urn}` Output: `{"urn": "urn:isbn:0451450523"}` | **AWS grok patterns** | Pattern | Description | Maximum pattern limit | Example | | --- | --- | --- | --- | | ARN | Matches AWS Amazon Resource Names (ARNs), capturing the partition (`aws`, `aws-cn`, or `aws-us-gov`), service, Region, account ID, and up to 5 hierarchical resource identifiers separated by slashes. It will not match ARNs that are missing information between colons. | 5 | Input: `arn:aws:iam:us-east-1:123456789012:user/johndoe` Pattern: `%{ARN:arn}` Output: `{"arn": "arn:aws:iam:us-east-1:123456789012:user/johndoe"}` | **Networking grok patterns** | Grok Pattern | Description | Maximum pattern limit | Example | | --- | --- | --- | --- | | CISCOMAC | Matches a MAC address in 4-4-4 hexadecimal format. | 20 | Input: `0123.4567.89AB` Pattern: `%{CISCOMAC:MacAddress}` Output: `{"MacAddress": "0123.4567.89AB"}` | | WINDOWSMAC | Matches a MAC address in hexadecimal format with hyphens | 20 | Input: `01-23-45-67-89-AB` Pattern: `%{WINDOWSMAC:MacAddress}` Output: `{"MacAddress": "01-23-45-67-89-AB"}` | | COMMONMAC | Matches a MAC address in hexadecimal format with colons. | 20 | Input: `01:23:45:67:89:AB` Pattern: `%{COMMONMAC:MacAddress}` Output: `{"MacAddress": "01:23:45:67:89:AB"}` | | MAC | Matches one of CISCOMAC, WINDOWSMAC or COMMONMAC grok patterns | 20 | Input: `01:23:45:67:89:AB` Pattern: `%{MAC:m1}` Output: `{"m1":"01:23:45:67:89:AB"}` | | IPV6 | Matches IPv6 addresses, including compressed forms and IPv4-mapped IPv6 addresses. | 5 | Input: `2001:db8:3333:4444:5555:6666:7777:8888` Pattern: `%{IPV6:ip}` Output: `{"ip": "2001:db8:3333:4444:5555:6666:7777:8888"}` | | IPV4 | Matches an IPv4 address. | 20 | Input: `192.168.0.1` Pattern: `%{IPV4:ip}` Output: `{"ip": "192.168.0.1"}` | | IP | Matches either IPv6 addresses as supported by %\$1IPv6\$1 or IPv4 addresses as supported by %\$1IPv4\$1 | 5 | Input: `192.168.0.1` Pattern: `%{IP:ip}` Output: `{"ip": "192.168.0.1"}` | | HOSTNAME or HOST | Matches domain names, including subdomains | 5 | Input: `server-01.internal-network.local` Pattern: `%{HOST:host}` Output: `{"host": "server-01.internal-network.local"}` | | IPORHOST | Matches either a hostname or an IP address | 5 | Input: `2001:db8:3333:4444:5555:6666:7777:8888` Pattern: `%{IPORHOST:ip}` Output: `{"ip": "2001:db8:3333:4444:5555:6666:7777:8888"}` | | HOSTPORT | Matches an IP address or hostname as supported by %\$1IPORHOST\$1 pattern followed by a colon and a port number, capturing the port as "PORT" in the output. | 5 | Input: `192.168.1.1:8080` Pattern: `%{HOSTPORT:ip}` Output: `{"ip":"192.168.1.1:8080","PORT":"8080"}` | | URIHOST | Matches an IP address or hostname as supported by %\$1IPORHOST\$1 pattern, optionally followed by a colon and a port number, capturing the port as "port" if present. | 5 | Input: `example.com:443 10.0.0.1` Pattern: `%{URIHOST:host} %{URIHOST:ip}` Output: `{"host":"example.com:443","port":"443","ip":"10.0.0.1"}` | **Path grok patterns** | Grok Pattern | Description | Maximum pattern limit | Example | | --- | --- | --- | --- | | UNIXPATH | Matches URL paths, potentially including query parameters. | 20 | Input: `/search?q=regex` Pattern: `%{UNIXPATH:path}` Output: `{"path":"/search?q=regex"}` | | WINPATH | Matches Windows file paths. | 5 | Input: `C:\Users\John\Documents\file.txt` Pattern: `%{WINPATH:path}` Output: `{"path": "C:\\Users\\John\\Documents\\file.txt"}` | | PATH | Matches either URL or Windows file paths | 5 | Input: `/search?q=regex` Pattern: `%{PATH:path}` Output: `{"path":"/search?q=regex"}` | | TTY | Matches Unix device paths for terminals and pseudo-terminals. | 20 | Input: `/dev/tty1` Pattern: `%{TTY:path}` Output: `{"path":"/dev/tty1"}` | | URIPROTO | Matches letters, optionally followed by a plus (\$1) character and additional letters or plus (\$1) characters | 20 | Input: `web+transformer` Pattern: `%{URIPROTO:protocol}` Output: `{"protocol":"web+transformer"}` | | URIPATH | Matches the path component of a URI | 20 | Input: `/category/sub-category/product_name` Pattern: `%{URIPATH:path}` Output: `{"path":"/category/sub-category/product_name"}` | | URIPARAM | Matches URL query parameters | 5 | Input: `?param1=value1¶m2=value2` Pattern: `%{URIPARAM:url}` Output: `{"url":"?param1=value1¶m2=value2"}` | | URIPATHPARAM | Matches a URI path optionally followed by query parameters | 5 | Input: `/category/sub-category/product?id=12345&color=red` Pattern: `%{URIPATHPARAM:path}` Output: `{"path":"/category/sub-category/product?id=12345&color=red"}` | | URI | Matches a complete URI | 5 | Input: `https://user:password@example.com/path/to/resource?param1=value1¶m2=value2` Pattern: `%{URI:uri}` Output: `{"path":"https://user:password@example.com/path/to/resource?param1=value1¶m2=value2"}` | **Date and time grok patterns** | Grok Pattern | Description | Maximum pattern limit | Example | | --- | --- | --- | --- | | MONTH | Matches full or abbreviated english month names as whole words | 20 | Input: `Jan` Pattern: `%{MONTH:month}` Output: `{"month":"Jan"}` Input: `January` Pattern: `%{MONTH:month}` Output: `{"month":"January"}` | | MONTHNUM | Matches month numbers from 1 to 12, with optional leading zero for single-digit months. | 20 | Input: `5` Pattern: `%{MONTHNUM:month}` Output: `{"month":"5"}` Input: `05` Pattern: `%{MONTHNUM:month}` Output: `{"month":"05"}` | | MONTHNUM2 | Matches two-digit month numbers from 01 to 12. | 20 | Input: `05` Pattern: `%{MONTHNUM2:month}` Output: `{"month":"05"}` | | MONTHDAY | Matches day of the month from 1 to 31, with optional leading zero. | 20 | Input: `31` Pattern: `%{MONTHDAY:monthDay}` Output: `{"monthDay":"31"}` | | YEAR | Matches year in two or four digits | 20 | Input: `2024` Pattern: `%{YEAR:year}` Output: `{"year":"2024"}` Input: `24` Pattern: `%{YEAR:year}` Output: `{"year":"24"}` | | DAY | Matches full or abbreviated day names. | 20 | Input: `Tuesday` Pattern: `%{DAY:day}` Output: `{"day":"Tuesday"}` | | HOUR | Matches hour in 24-hour format with an optional leading zero (0)0-23. | 20 | Input: `22` Pattern: `%{HOUR:hour}` Output: `{"hour":"22"}` | | MINUTE | Matches minutes (00-59). | 20 | Input: `59` Pattern: `%{MINUTE:min}` Output: `{"min":"59"}` | | SECOND | Matches a number representing seconds (0)0-60, optionally followed by a decimal point or colon and one or more digits for fractional minutes | 20 | Input: `3` Pattern: `%{SECOND:second}` Output: `{"second":"3"}` Input: `30.5` Pattern: `%{SECOND:minSec}` Output: `{"minSec":"30.5"}` Input: `30:5` Pattern: `%{SECOND:minSec}` Output: `{"minSec":"30:5"}` | | TIME | Matches a time format with hours, minutes, and seconds in the format (H)H:mm:(s)s. Seconds include leap second (0)0-60. | 20 | Input: `09:45:32` Pattern: `%{TIME:time}` Output: `{"time":"09:45:32"}` | | DATE\$1US | Matches a date in the format of (M)M/(d)d/(yy)yy or (M)M-(d)d-(yy)yy. | 20 | Input: `11/23/2024` Pattern: `%{DATE_US:date}` Output: `{"date":"11/23/2024"}` Input: `1-01-24` Pattern: `%{DATE_US:date}` Output: `{"date":"1-01-24"}` | | DATE\$1EU | Matches date in format of (d)d/(M)M/(yy)yy, (d)d-(M)M-(yy)yy, or (d)d.(M)M.(yy)yy. | 20 | Input: `23/11/2024` Pattern: `%{DATE_EU:date}` Output: `{"date":"23/11/2024"}` Input: `1.01.24` Pattern: `%{DATE_EU:date}` Output: `{"date":"1.01.24"}` | | ISO8601\$1TIMEZONE | Matches UTC offset 'Z' or time zone offset with optional colon in format [\$1-](H)H(:)mm. | 20 | Input: `+05:30` Pattern: `%{ISO8601_TIMEZONE:tz}` Output: `{"tz":"+05:30"}` Input: `-530` Pattern: `%{ISO8601_TIMEZONE:tz}` Output: `{"tz":"-530"}` Input: `Z` Pattern: `%{ISO8601_TIMEZONE:tz}` Output: `{"tz":"Z"}` | | ISO8601\$1SECOND | Matches a number representing seconds (0)0-60, optionally followed by a decimal point or colon and one or more digits for fractional seconds | 20 | Input: `60` Pattern: `%{ISO8601_SECOND:second}` Output: `{"second":"60"}` | | TIMESTAMP\$1ISO8601 | Matches ISO8601 datetime format (yy)yy-(M)M-(d)dT(H)H:mm:((s)s)(Z\$1[\$1-](H)H:mm) with optional seconds and timezone. | 20 | Input: `2023-05-15T14:30:00+05:30` Pattern: `%{TIMESTAMP_ISO8601:timestamp}` Output: `{"timestamp":"2023-05-15T14:30:00+05:30"}` Input: `23-5-1T1:25+5:30` Pattern: `%{TIMESTAMP_ISO8601:timestamp}` Output: `{"timestamp":"23-5-1T1:25+5:30"}` Input: `23-5-1T1:25Z` Pattern: `%{TIMESTAMP_ISO8601:timestamp}` Output: `{"timestamp":"23-5-1T1:25Z"}` | | DATE | Matches either a date in the US format using %\$1DATE\$1US\$1 or in the EU format using %\$1DATE\$1EU\$1 | 20 | Input: `11/29/2024` Pattern: `%{DATE:date}` Output: `{"date":"11/29/2024"}` Input: `29.11.2024` Pattern: `%{DATE:date}` Output: `{"date":"29.11.2024"}` | | DATESTAMP | Matches %\$1DATE\$1 followed by %\$1TIME\$1 pattern, separated by space or hyphen. | 20 | Input: `29-11-2024 14:30:00` Pattern: `%{DATESTAMP:dateTime}` Output: `{"dateTime":"29-11-2024 14:30:00"}` | | TZ | Matches common time zone abbreviations (PST, PDT, MST, MDT, CST CDT, EST, EDT, UTC). | 20 | Input: `PDT` Pattern: `%{TZ:tz}` Output: `{"tz":"PDT"}` | | DATESTAMP\$1RFC822 | Matches date and time in format: Day MonthName (D)D (YY)YY (H)H:mm:(s)s Timezone | 20 | Input: `Monday Jan 5 23 1:30:00 CDT` Pattern: `%{DATESTAMP_RFC822:dateTime}` Output: `{"dateTime":"Monday Jan 5 23 1:30:00 CDT"}` Input: `Mon January 15 2023 14:30:00 PST` Pattern: `%{DATESTAMP_RFC822:dateTime}` Output: `{"dateTime":"Mon January 15 2023 14:30:00 PST"}` | | DATESTAMP\$1RFC2822 | Matches RFC2822 date-time format: Day, (d)d MonthName (yy)yy (H)H:mm:(s)s Z\$1[\$1-](H)H:mm | 20 | Input: `Mon, 15 May 2023 14:30:00 +0530` Pattern: `%{DATESTAMP_RFC2822:dateTime}` Output: `{"dateTime":"Mon, 15 May 2023 14:30:00 +0530"}` Input: `Monday, 15 Jan 23 14:30:00 Z` Pattern: `%{DATESTAMP_RFC2822:dateTime}` Output: `{"dateTime":"Monday, 15 Jan 23 14:30:00 Z"}` | | DATESTAMP\$1OTHER | Matches date and time in format: Day MonthName (d)d (H)H:mm:(s)s Timezone (yy)yy | 20 | Input: `Mon May 15 14:30:00 PST 2023` Pattern: `%{DATESTAMP_OTHER:dateTime}` Output: `{"dateTime":"Mon May 15 14:30:00 PST 2023"}` | | DATESTAMP\$1EVENTLOG | Matches compact datetime format without separators: (yy)yyMM(d)d(H)Hmm(s)s | 20 | Input: `20230515143000` Pattern: `%{DATESTAMP_EVENTLOG:dateTime}` Output: `{"dateTime":"20230515143000"}` | **Log grok patterns** | Grok Pattern | Description | Maximum pattern limit | Example | | --- | --- | --- | --- | | LOGLEVEL | Matches standard log levels in different capitalizations and abbreviations, including the following: Alert/ALERT, Trace/TRACE, Debug/DEBUG, Notice/NOTICE, Info/INFO, Warn/Warning/WARN/WARNING, Err/Error/ERR/ERROR, Crit/Critical/CRIT/CRITICAL, Fatal/FATAL, Severe/SEVERE, Emerg/Emergency/EMERG/EMERGENCY | 20 | Input: `INFO` Pattern: `%{LOGLEVEL:logLevel}` Output: `{"logLevel":"INFO"}` | | HTTPDATE | Matches date and time format often used in log files. Format: (d)d/MonthName/(yy)yy:(H)H:mm:(s)s Timezone MonthName: Matches full or abbreviated english month names (Example: "Jan" or "January") Timezone: Matches %\$1INT\$1 grok pattern | 20 | Input: `23/Nov/2024:14:30:00 +0640` Pattern: `%{HTTPDATE:date}` Output: `{"date":"23/Nov/2024:14:30:00 +0640"}` | | SYSLOGTIMESTAMP | Matches date format with MonthName (d)d (H)H:mm:(s)s MonthName: Matches full or abbreviated english month names (Example: "Jan" or "January") | 20 | Input: `Nov 29 14:30:00` Pattern: `%{SYSLOGTIMESTAMP:dateTime}` Output: `{"dateTime":"Nov 29 14:30:00"}` | | PROG | Matches a program name consisting of string of letters, digits, dot, underscore, forward slash, percent sign, and hyphen characters. | 20 | Input: `user.profile/settings-page` Pattern: `%{PROG:program}` Output: `{"program":"user.profile/settings-page"}` | | SYSLOGPROG | Matches PROG grok pattern optionally followed by a process ID in square brackets. | 20 | Input: `user.profile/settings-page[1234]` Pattern: `%{SYSLOGPROG:programWithId}` Output: `{"programWithId":"user.profile/settings-page[1234]","program":"user.profile/settings-page","pid":"1234"}` | | SYSLOGHOST | Matches either a %\$1HOST\$1 or %\$1IP\$1 pattern | 5 | Input: `2001:db8:3333:4444:5555:6666:7777:8888` Pattern: `%{SYSLOGHOST:ip}` Output: `{"ip": "2001:db8:3333:4444:5555:6666:7777:8888"}` | | SYSLOGFACILITY | Matches syslog priority in decimal format. The value should be enclosed in angular brackets (<>). | 20 | Input: `<13.6>` Pattern: `%{SYSLOGFACILITY:syslog}` Output: `{"syslog":"<13.6>","facility":"13","priority":"6"}` | **Common log grok patterns** You can use pre-defined custom grok patterns to match Apache, NGINX and Syslog Protocol (RFC 5424) log formats. When you use these specific patterns, they must be the first patterns in your matching configuration, and no other patterns can precede them. Also, you can follow them only with exactly one **DATA**. **GREEDYDATA** or **GREEDYDATA\$1MULTILINE** pattern. | Grok pattern | Description | Maximum pattern limit | | --- | --- | --- | | APACHE\$1ACCESS\$1LOG | Matches Apache access logs | 1 | | NGINX\$1ACCESS\$1LOG | Matches NGINX access logs | 1 | | SYSLOG5424 | Matches Syslog Protocol (RFC 5424) logs | 1 | The following shows valid and invalid examples for using these common log format patterns. ``` "%{NGINX_ACCESS_LOG} %{DATA}" // Valid "%{SYSLOG5424}%{DATA:logMsg}" // Valid "%{APACHE_ACCESS_LOG} %{GREEDYDATA:logMsg}" // Valid "%{APACHE_ACCESS_LOG} %{SYSLOG5424}" // Invalid (multiple common log patterns used) "%{NGINX_ACCESS_LOG} %{NUMBER:num}" // Invalid (Only GREEDYDATA and DATA patterns are supported with common log patterns) "%{GREEDYDATA:logMsg} %{SYSLOG5424}" // Invalid (GREEDYDATA and DATA patterns are supported only after common log patterns) ``` #### Common log format examples ##### Apache log example Sample log: ``` 127.0.0.1 - - [03/Aug/2023:12:34:56 +0000] "GET /page.html HTTP/1.1" 200 1234 ``` Transformer: ``` [ { "grok": { "match": "%{APACHE_ACCESS_LOG}" } } ] ``` Output: ``` { "request": "/page.html", "http_method": "GET", "status_code": 200, "http_version": "1.1", "response_size": 1234, "remote_host": "127.0.0.1", "timestamp": "2023-08-03T12:34:56Z" } ``` ##### NGINX log example Sample log: ``` 192.168.1.100 - Foo [03/Aug/2023:12:34:56 +0000] "GET /account/login.html HTTP/1.1" 200 42 "https://www.amazon.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36" ``` Transformer: ``` [ { "grok": { "match": "%{NGINX_ACCESS_LOG}" } } ] ``` Output: ``` { "request": "/account/login.html", "referrer": "https://www.amazon.com/", "agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36", "http_method": "GET", "status_code": 200, "auth_user": "Foo", "http_version": "1.1", "response_size": 42, "remote_host": "192.168.1.100", "timestamp": "2023-08-03T12:34:56Z" } ``` ##### Syslog Protocol (RFC 5424) log example Sample log: ``` <165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"][examplePriority@32473 class="high"] ``` Transformer: ``` [ { "grok": { "match": "%{SYSLOG5424}" } } ] ``` Output: ``` { "pri": 165, "version": 1, "timestamp": "2003-10-11T22:14:15.003Z", "hostname": "mymachine.example.com", "app": "evntslog", "msg_id": "ID47", "structured_data": "exampleSDID@32473 iut=\"3\" eventSource= \"Application\" eventID=\"1011\"", "message": "[examplePriority@32473 class=\"high\"]" } ``` ## csv The **csv** processor parses comma-separated values (CSV) from the log events into columns. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | source | Path to the field in the log event that will be parsed | No | `@message` | Maximum length: 128 Maximum nested key depth: 3 | | delimiter | The character used to separate each column in the original comma-separated value log event | No | `,` | Maximum length: 1 unless the value is `\t` or `\s` | | quoteCharacter | Character used as a text qualifier for a single column of data | No | `"` | Maximum length: 1 | | columns | List of names to use for the columns in the transformed log event. | No | `[column_1, column_2 ...]` | Maximum CSV columns: 100 Maximum length: 128 Maximum nested key depth: 3 | | destination | The parent field to put transformed key value pairs under | No | `Root node` | Maximum length: 128 Maximum nested key depth: 3 | Setting `delimiter` to `\t` will separate each column on a tab character, and `\t` will separate each column on a single space character. **Example** Suppose part of an ingested log event looks like this: ``` 'Akua Mansa':28:'New York: USA' ``` Suppose we use only the **csv** processor: ``` [ { "csv": { "delimiter": ":", "quoteCharacter": "'" } } ] ``` The transformed log event would be the following. ``` { "column_1": "Akua Mansa", "column_2": "28", "column_3": "New York: USA" } ``` **Example 2** Suppose an ingested log event looks like this: ``` { "timestamp": "2024-11-23T16:03:12Z", "type": "user_data", "logMsg": "'Akua Mansa':28:'New York: USA'" } ``` Suppose we parse the event as JSON, they parse a JSON field with the **csv** processor, specifying column names and destination: ``` [ { "parseJSON": {} }, { "csv": { "source": "logMsg", "delimiter": ":", "quoteCharacter": "'", "columns":["name","age","location"], "destination": "msg" } } ] ``` The transformed log event would be the following. ``` { "timestamp": "2024-11-23T16:03:12Z", "logMsg": "'Akua Mansa':28:'New York: USA'", "type": "user_data", "msg": { "name": "Akua Mansa", "age": "28", "location": "New York: USA" } } ``` ## parseKeyValue Use the **parseKeyValue** processor to parse a specified field into key-value pairs. You can customize the processor to parse field information with the following options. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | source | Path to the field in the log event that will be parsed | No | `@message` | Maximum length: 128 Maximum nested key depth: 3 | | destination | The destination field to put the extracted key-value pairs into | No | | Maximum length: 128 | | fieldDelimiter | The field delimiter string that is used between key-value pairs in the original log events | No | `&` | Maximum length: 128 | | keyValueDelimiter | The delimiter string to use between the key and value in each pair in the transformed log event | No | `=` | Maximum length: 128 | | nonMatchValue | A value to insert into the value field in the result, when a key-value pair is not successfully split. | No | | Maximum length: 128 | | keyPrefix | If you want to add a prefix toall transformed keys, specify it here. | No | | Maximum length: 128 | | overwriteIfExists | Whether to overwrite the value if the destination key already exists | No | `false` | | **Example** Take the following example log event: ``` key1:value1!key2:value2!key3:value3!key4 ``` Suppose we use the following processor configuration: ``` [ { "parseKeyValue": { "destination": "new_key", "fieldDelimiter": "!", "keyValueDelimiter": ":", "nonMatchValue": "defaultValue", "keyPrefix": "parsed_" } } ] ``` The transformed log event would be the following. ``` { "new_key": { "parsed_key1": "value1", "parsed_key2": "value2", "parsed_key3": "value3", "parsed_key4": "defaultValue" } } ``` # Built-in processors for AWS vended logs This section contains information about the built-in processors that you can use with AWS services that vend logs. **Contents** + [parseWAF](#CloudWatch-Logs-Transformation-parseWAF) + [parsePostgres](#CloudWatch-Logs-Transformation-parsePostGres) + [parseCloudfront](#CloudWatch-Logs-Transformation-parseCloudFront) + [parseRoute53](#CloudWatch-Logs-Transformation-parseRoute53) + [parseVPC](#CloudWatch-Logs-Transformation-parseVPC) + [parseToOCSF](CloudWatch-Logs-Transformation-parseToOCSF.md) ## parseWAF Use this processor to parse AWS WAF vended logs, It takes the contents of `httpRequest.headers` and creates JSON keys from each header name, with the corresponding value. It also does the same for `labels`. These transformations can make querying AWS WAF logs much easier. For more information about AWS WAF log format, see [ Log examples for web ACL traffic](https://docs.aws.amazon.com/waf/latest/developerguide/logging-examples.html). This processor accepts only `@message` as the input. **Important** If you use this processor, it must be the first processor in your transformer. **Example** Take the following example log event: ``` { "timestamp": 1576280412771, "formatVersion": 1, "webaclId": "arn:aws:wafv2:ap-southeast-2:111122223333:regional/webacl/STMTest/1EXAMPLE-2ARN-3ARN-4ARN-123456EXAMPLE", "terminatingRuleId": "STMTest_SQLi_XSS", "terminatingRuleType": "REGULAR", "action": "BLOCK", "terminatingRuleMatchDetails": [ { "conditionType": "SQL_INJECTION", "sensitivityLevel": "HIGH", "location": "HEADER", "matchedData": ["10", "AND", "1"] } ], "httpSourceName": "-", "httpSourceId": "-", "ruleGroupList": [], "rateBasedRuleList": [], "nonTerminatingMatchingRules": [], "httpRequest": { "clientIp": "1.1.1.1", "country": "AU", "headers": [ { "name": "Host", "value": "localhost:1989" }, { "name": "User-Agent", "value": "curl/7.61.1" }, { "name": "Accept", "value": "*/*" }, { "name": "x-stm-test", "value": "10 AND 1=1" } ], "uri": "/myUri", "args": "", "httpVersion": "HTTP/1.1", "httpMethod": "GET", "requestId": "rid" }, "labels": [{ "name": "value" }] } ``` The processor configuration is this: ``` [ { "parseWAF": {} } ] ``` The transformed log event would be the following. ``` { "httpRequest": { "headers": { "Host": "localhost:1989", "User-Agent": "curl/7.61.1", "Accept": "*/*", "x-stm-test": "10 AND 1=1" }, "clientIp": "1.1.1.1", "country": "AU", "uri": "/myUri", "args": "", "httpVersion": "HTTP/1.1", "httpMethod": "GET", "requestId": "rid" }, "labels": { "name": "value" }, "timestamp": 1576280412771, "formatVersion": 1, "webaclId": "arn:aws:wafv2:ap-southeast-2:111122223333:regional/webacl/STMTest/1EXAMPLE-2ARN-3ARN-4ARN-123456EXAMPLE", "terminatingRuleId": "STMTest_SQLi_XSS", "terminatingRuleType": "REGULAR", "action": "BLOCK", "terminatingRuleMatchDetails": [ { "conditionType": "SQL_INJECTION", "sensitivityLevel": "HIGH", "location": "HEADER", "matchedData": ["10", "AND", "1"] } ], "httpSourceName": "-", "httpSourceId": "-", "ruleGroupList": [], "rateBasedRuleList": [], "nonTerminatingMatchingRules": [] } ``` ## parsePostgres Use this processor to parse Amazon RDS for PostgreSQL vended logs, extract fields, and convert them to JSON format. For more information about RDS for PostgreSQL log format, see [ RDS for PostgreSQL database log files](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.PostgreSQL.html#USER_LogAccess.Concepts.PostgreSQL.Log_Format.log-line-prefix). This processor accepts only `@message` as the input. **Important** If you use this processor, it must be the first processor in your transformer. **Example** Take the following example log event: ``` 2019-03-10 03:54:59 UTC:10.0.0.123(52834):postgres@logtestdb:[20175]:ERROR: column "wrong_column_name" does not exist at character 8 ``` The processor configuration is this: ``` [ { "parsePostgres": {} } ] ``` The transformed log event would be the following. ``` { "logTime": "2019-03-10 03:54:59 UTC", "srcIp": "10.0.0.123(52834)", "userName": "postgres", "dbName": "logtestdb", "processId": "20175", "logLevel": "ERROR" } ``` ## parseCloudfront Use this processor to parse Amazon CloudFront vended logs, extract fields, and convert them into JSON format. Encoded field values are decoded. Values that are integers and doubles are treated as such. For more information about Amazon CloudFront log format, see [ Configure and use standard logs (access logs)](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html). This processor accepts only `@message` as the input. **Important** If you use this processor, it must be the first processor in your transformer. **Example** Take the following example log event: ``` 2019-12-04 21:02:31 LAX1 392 192.0.2.24 GET d111111abcdef8.cloudfront.net /index.html 200 - Mozilla/5.0%20(Windows%20NT%2010.0;%20Win64;%20x64)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/78.0.3904.108%20Safari/537.36 - - Hit SOX4xwn4XV6Q4rgb7XiVGOHms_BGlTAC4KyHmureZmBNrjGdRLiNIQ== d111111abcdef8.cloudfront.net https 23 0.001 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Hit HTTP/2.0 - - 11040 0.001 Hit text/html 78 - - ``` The processor configuration is this: ``` [ { "parseCloudfront": {} } ] ``` The transformed log event would be the following. ``` { "date": "2019-12-04", "time": "21:02:31", "x-edge-location": "LAX1", "sc-bytes": 392, "c-ip": "192.0.2.24", "cs-method": "GET", "cs(Host)": "d111111abcdef8.cloudfront.net", "cs-uri-stem": "/index.html", "sc-status": 200, "cs(Referer)": "-", "cs(User-Agent)": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36", "cs-uri-query": "-", "cs(Cookie)": "-", "x-edge-result-type": "Hit", "x-edge-request-id": "SOX4xwn4XV6Q4rgb7XiVGOHms_BGlTAC4KyHmureZmBNrjGdRLiNIQ==", "x-host-header": "d111111abcdef8.cloudfront.net", "cs-protocol": "https", "cs-bytes": 23, "time-taken": 0.001, "x-forwarded-for": "-", "ssl-protocol": "TLSv1.2", "ssl-cipher": "ECDHE-RSA-AES128-GCM-SHA256", "x-edge-response-result-type": "Hit", "cs-protocol-version": "HTTP/2.0", "fle-status": "-", "fle-encrypted-fields": "-", "c-port": 11040, "time-to-first-byte": 0.001, "x-edge-detailed-result-type": "Hit", "sc-content-type": "text/html", "sc-content-len": 78, "sc-range-start": "-", "sc-range-end": "-" } ``` ## parseRoute53 Use this processor to parse Amazon Route 53 Public Data Plane vended logs, extract fields, and convert them into JSON format. Encoded field values are decoded. This processor does not support Amazon Route 53 Resolver logs. This processor accepts only `@message` as the input. **Important** If you use this processor, it must be the first processor in your transformer. **Example** Take the following example log event: ``` 1.0 2017-12-13T08:15:50.235Z Z123412341234 example.com AAAA NOERROR TCP IAD12 192.0.2.0 198.51.100.0/24 ``` The processor configuration is this: ``` [ { "parseRoute53": {} } ] ``` The transformed log event would be the following. ``` { "version": 1.0, "queryTimestamp": "2017-12-13T08:15:50.235Z", "hostZoneId": "Z123412341234", "queryName": "example.com", "queryType": "AAAA", "responseCode": "NOERROR", "protocol": "TCP", "edgeLocation": "IAD12", "resolverIp": "192.0.2.0", "ednsClientSubnet": "198.51.100.0/24" } ``` ## parseVPC Use this processor to parse Amazon VPC vended logs, extract fields, and convert them into JSON format. Encoded field values are decoded. This processor accepts only `@message` as the input. **Important** If you use this processor, it must be the first processor in your transformer. **Example** Take the following example log event: ``` 2 123456789010 eni-abc123de 192.0.2.0 192.0.2.24 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK ``` The processor configuration is this: ``` [ { "parseVPC": {} } ] ``` The transformed log event would be the following. ``` { "version": 2, "accountId": "123456789010", "interfaceId": "eni-abc123de", "srcAddr": "192.0.2.0", "dstAddr": "192.0.2.24", "srcPort": 20641, "dstPort": 22, "protocol": 6, "packets": 20, "bytes": 4249, "start": 1418530010, "end": 1418530070, "action": "ACCEPT", "logStatus": "OK" } ``` # parseToOCSF The `parseToOCSF` processor converts logs into Open Cybersecurity Schema Framework (OCSF) events. OCSF is an open standard that provides a common schema for security data, enabling better interoperability and analysis across different security tools and platforms. This processor is particularly useful for security analytics workflows where you need to standardize log formats from various AWS services into a consistent schema for downstream analysis. **Parameters** `eventSource` (required) Specifies the AWS service or process that produces the log events to be converted. Valid values are: + `CloudTrail` - CloudTrail logs + `Route53Resolver` - Route 53 Resolver logs + `VPCFlow` - Amazon VPC Flow Logs + `EKSAudit` - Amazon EKS audit logs + `AWSWAF` - AWS WAF logs `ocsfVersion` (required) Specifies which version of the OCSF schema to use for the transformed log events. Currently supported versions: `V1.1, V1.5` `mappingVersion` (optional) Specifies the OCSF transformation mapping version. Controls which transformation logic is applied when converting logs to OCSF format. If not specified, uses the latest available version at policy creation time. Existing policies do not automatically upgrade when new mapping versions are released. Current latest version: `v1.5.0`. **Note:** Not supported when `ocsfVersion` is `V1.1`. `source` (optional) The path to the field in the log event that you want to parse. If omitted, the entire log message is parsed. **Example** The following example shows how to use `parseToOCSF` to convert VPC Flow Logs to OCSF format: ``` { "parseToOCSF": { "eventSource": "VPCFlow", "ocsfVersion": "V1.1" } } ``` The following example shows how to specify a particular mapping version for consistent transformation behavior: ``` { "parseToOCSF": { "eventSource": "CloudTrail", "ocsfVersion": "V1.5", "mappingVersion": "v1.5.0" } } ``` # String mutate processors This section contains information about the string mutate processors that you can use with a log event transformer. **Contents** + [lowerCaseString](#CloudWatch-Logs-Transformation-lowerCaseString) + [upperCaseString](#CloudWatch-Logs-Transformation-upperCaseString) + [splitString](#CloudWatch-Logs-Transformation-splitString) + [substituteString](#CloudWatch-Logs-Transformation-substituteString) + [trimString](#CloudWatch-Logs-Transformation-trimString) ## lowerCaseString The `lowerCaseString` processor converts a string to its lowercase version. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | withKeys | A list of keys to convert to lowercase | Yes | | Maximum entries: 10 | **Example** Take the following example log event: ``` { "outer_key": { "inner_key": "INNER_VALUE" } } ``` The transformer configuration is this, using `lowerCaseString` with `parseJSON`: ``` [ { "parseJSON": {} }, { "lowerCaseString": { "withKeys":["outer_key.inner_key"] } } ] ``` The transformed log event would be the following. ``` { "outer_key": { "inner_key": "inner_value" } } ``` ## upperCaseString The `upperCaseString` processor converts a string to its uppercase version. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | withKeys | A list of keys to convert to uppercase | Yes | | Maximum entries: 10 | **Example** Take the following example log event: ``` { "outer_key": { "inner_key": "inner_value" } } ``` The transformer configuration is this, using `upperCaseString` with `parseJSON`: ``` [ { "parseJSON": {} }, { "upperCaseString": { "withKeys":["outer_key.inner_key"] } } ] ``` The transformed log event would be the following. ``` { "outer_key": { "inner_key": "INNER_VALUE" } } ``` ## splitString The `splitString` processor is a type of string mutate processor which splits a field into an array using a delimiting character. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | entries | Array of entries. Each item in the array must contain source and delimiter fields. | Yes | | Maximum entries: 10 | | source | The key of the field value to split | Yes | | Maximum length: 128 | | delimiter | The delimiter string to split the field value on | Yes | | Maximum length: 128 | **Example 1** Take the following example log event: ``` { "outer_key": { "inner_key": "inner_value" } } ``` The transformer configuration is this, using `splitString` with `parseJSON`: ``` [ { "parseJSON": {} }, { "splitString": { "entries": [ { "source": "outer_key.inner_key", "delimiter": "_" } ] } } ] ``` The transformed log event would be the following. ``` { "outer_key": { "inner_key": [ "inner", "value" ] } } ``` **Example 2** The delimiter to split the string on can be multiple characters long. Take the following example log event: ``` { "outer_key": { "inner_key": "item1, item2, item3" } } ``` The transformer configuration is as follows: ``` [ { "parseJSON": {} }, { "splitString": { "entries": [ { "source": "outer_key.inner_key", "delimiter": ", " } ] } } ] ``` The transformed log event would be the following. ``` { "outer_key": { "inner_key": [ "item1", "item2", "item3" ] } } ``` ## substituteString The `substituteString` processor is a type of string mutate processor which matches a key’s value against a regular expression and replaces all matches with a replacement string. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | entries | Array of entries. Each item in the array must contain source, from, and to fields. | Yes | | Maximum entries: 10 | | source | The key of the field to modify | Yes | | Maximum length: 128 Maximum nested key depth: 3 | | from | The regular expression string to be replaced. Special regex characters such as [ and ] must be escaped using \$1\$1 when using double quotes and with \$1 when using single quotes or when configured from the AWS Management Console. For more information, see [ Class Pattern](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/regex/Pattern.html) on the Oracle web site. You can wrap a pattern in `(...)` to create a numbered capturing group and create `(?P...)` named capturing groups that can be referenced in the `to` field. | Yes | | Maximum length: 128 | | to | The string to be substituted for each match of from Backreferences to capturing groups can be used. Use the form \$1n for numbered groups such as \$11 and use \$1\$1group\$1name\$1 for named groups such as \$1\$1my\$1group\$1.> | Yes | | Maximum length: 128 Maximum number of backreferences: 10 Maximum number of duplicate backreferences: 2 | **Example 1** Take the following example log event: ``` { "outer_key": { "inner_key1": "[]", "inner_key2": "123-345-567", "inner_key3": "A cat takes a catnap." } } ``` The transformer configuration is this, using `substituteString` with `parseJSON`: ``` [ { "parseJSON": {} }, { "substituteString": { "entries": [ { "source": "outer_key.inner_key1", "from": "\\[\\]", "to": "value1" }, { "source": "outer_key.inner_key2", "from": "[0-9]{3}-[0-9]{3}-[0-9]{3}", "to": "xxx-xxx-xxx" }, { "source": "outer_key.inner_key3", "from": "cat", "to": "dog" } ] } } ] ``` The transformed log event would be the following. ``` { "outer_key": { "inner_key1": "value1", "inner_key2": "xxx-xxx-xxx", "inner_key3": "A dog takes a dognap." } } ``` **Example 2** Take the following example log event: ``` { "outer_key": { "inner_key1": "Tom, Dick, and Harry", "inner_key2": "arn:aws:sts::123456789012:assumed-role/MyImportantRole/MySession" } } ``` The transformer configuration is this, using `substituteString` with `parseJSON`: ``` [ { "parseJSON": {} }, { "substituteString": { "entries": [ { "source": "outer_key.inner_key1", "from": "(\w+), (\w+), and (\w+)", "to": "$1 and $3" }, { "source": "outer_key.inner_key2", "from": "^arn:aws:sts::(?P\\d{12}):assumed-role/(?P[\\w+=,.@-]+)/(?P[\\w+=,.@-]+)$", "to": "${account_id}:${role_name}:${role_session_name}" } ] } } ] ``` The transformed log event would be the following. ``` { "outer_key": { "inner_key1": "Tom and Harry", "inner_key2": "123456789012:MyImportantRole:MySession" } } ``` ## trimString The `trimString` processor removes whitespace from the beginning and end of a key. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | withKeys | A list of keys to trim | Yes | | Maximum entries: 10 | **Example** Take the following example log event: ``` { "outer_key": { "inner_key": " inner_value " } } ``` The transformer configuration is this, using `trimString` with `parseJSON`: ``` [ { "parseJSON": {} }, { "trimString": { "withKeys":["outer_key.inner_key"] } } ] ``` The transformed log event would be the following. ``` { "outer_key": { "inner_key": "inner_value" } } ``` # JSON mutate processors This section contains information about the JSON mutate processors that you can use with a log event transformer. **Contents** + [addKeys](#CloudWatch-Logs-Transformation-addKeys) + [deleteKeys](#CloudWatch-Logs-Transformation-deleteKeys) + [moveKeys](#CloudWatch-Logs-Transformation-moveKeys) + [renameKeys](#CloudWatch-Logs-Transformation-renameKeys) + [copyValue](#CloudWatch-Logs-Transformation-copyValue) + [listToMap](#CloudWatch-Logs-Transformation-listToMap) ## addKeys Use the `addKeys` processor to add new key-value pairs to the log event. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | entries | Array of entries. Each item in the array can contain key, value, and overwriteIfExists fields. | Yes | | Maximum entries: 5 | | key | The key of the new entry to be added | Yes | | Maximum length: 128 Maximum nested key depth: 3 | | value | The value of the new entry to be added | Yes | | Maximum length: 256 | | overwriteIfExists | If you set this to true, the existing value is overwritten if key already exists in the event. The default value is false. | No | false | No limit | **Example** Take the following example log event: ``` { "outer_key": { "inner_key": "inner_value" } } ``` The transformer configuration is this, using `addKeys` with `parseJSON`: ``` [ { "parseJSON": {} }, { "addKeys": { "entries": [ { "key": "outer_key.new_key", "value": "new_value" } ] } } ] ``` The transformed log event would be the following. ``` { "outer_key": { "inner_key": "inner_value", "new_key": "new_value" } } ``` ## deleteKeys Use the `deleteKeys` processor to delete fields from a log event. These fields can include key-value pairs. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | withKeys | The list of keys to delete. | Yes | No limit | Maximum entries: 5 | **Example** Take the following example log event: ``` { "outer_key": { "inner_key": "inner_value" } } ``` The transformer configuration is this, using `deleteKeys` with `parseJSON`: ``` [ { "parseJSON": {} }, { "deleteKeys": { "withKeys":["outer_key.inner_key"] } } ] ``` The transformed log event would be the following. ``` { "outer_key": {} } ``` ## moveKeys Use the `moveKeys` processor to move a key from one field to another. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | entries | Array of entries. Each item in the array can contain source, target, and overwriteIfExists fields. | Yes | | Maximum entries: 5 | | source | The key to move | Yes | | Maximum length: 128 Maximum nested key depth: 3 | | target | The key to move to | Yes | | Maximum length: 128 Maximum nested key depth: 3 | | overwriteIfExists | If you set this to true, the existing value is overwritten if key already exists in the event. The default value is false. | No | false | No limit | **Example** Take the following example log event: ``` { "outer_key1": { "inner_key1": "inner_value1" }, "outer_key2": { "inner_key2": "inner_value2" } } ``` The transformer configuration is this, using `moveKeys` with `parseJSON`: ``` [ { "parseJSON": {} }, { "moveKeys": { "entries": [ { "source": "outer_key1.inner_key1", "target": "outer_key2" } ] } } ] ``` The transformed log event would be the following. ``` { "outer_key1": {}, "outer_key2": { "inner_key2": "inner_value2", "inner_key1": "inner_value1" } } ``` ## renameKeys Use the `renameKeys` processor to rename keys in a log event. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | entries | Array of entries. Each item in the array can contain key, target, and overwriteIfExists fields. | Yes | No limit | Maximum entries: 5 | | key | The key to rename | Yes | No limit | Maximum length: 128 | | target | The new key name | Yes | No limit | Maximum length: 128 Maximum nested key depth: 3 | | overwriteIfExists | If you set this to true, the existing value is overwritten if key already exists in the event. The default value is false. | No | false | No limit | **Example** Take the following example log event: ``` { "outer_key": { "inner_key": "inner_value" } } ``` The transformer configuration is this, using `renameKeys` with `parseJSON`: ``` [ { "parseJSON": {} }, { "renameKeys": { "entries": [ { "key": "outer_key", "target": "new_key" } ] } } ] ``` The transformed log event would be the following. ``` { "new_key": { "inner_key": "inner_value" } } ``` ## copyValue Use the `copyValue` processor to copy values within a log event. You can also use this processor to add metadata to log events, by copying the values of the following metadata keys into the log events: `@logGroupName`, `@logGroupStream`, `@accountId`, `@regionName`. This is illustrated in the following example. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | entries | Array of entries. Each item in the array can contain source, target, and overwriteIfExists fields. | Yes | | Maximum entries: 5 | | source | The key to copy | Yes | | Maximum length: 128 Maximum nested key depth: 3 | | target | The key to copy the value to | Yes | No limit | Maximum length: 128 Maximum nested key depth: 3 | | overwriteIfExists | If you set this to true, the existing value is overwritten if key already exists in the event. The default value is false. | No | false | No limit | **Example** Take the following example log event: ``` { "outer_key": { "inner_key": "inner_value" } } ``` The transformer configuration is this, using `copyValue` with `parseJSON`: ``` [ { "parseJSON": {} }, { "copyValue": { "entries": [ { "key": "outer_key.new_key", "target": "new_key" }, { "source": "@logGroupName", "target": "log_group_name" }, { "source": "@logGroupStream", "target": "log_group_stream" }, { "source": "@accountId", "target": "account_id" }, { "source": "@regionName", "target": "region_name" } ] } } ] ``` The transformed log event would be the following. ``` { "outer_key": { "inner_key": "inner_value" }, "new_key": "inner_value", "log_group_name": "myLogGroupName", "log_group_stream": "myLogStreamName", "account_id": "012345678912", "region_name": "us-east-1" } ``` ## listToMap The `listToMap` processor takes a list of objects that contain key fields, and converts them into a map of target keys. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | source | The key in the ProcessingEvent with a list of objects that will be converted to a map | Yes | | Maximum length: 128 Maximum nested key depth: 3 | | key | The key of the fields to be extracted as keys in the generated map | Yes | | Maximum length: 128 | | valueKey | If this is specified, the values that you specify in this parameter will be extracted from the source objects and put into the values of the generated map. Otherwise, original objects in the source list will be put into the values of the generated map. | No | | Maximum length: 128 | | target | The key of the field that will hold the generated map | No | Root node | Maximum length: 128 Maximum nested key depth: 3 | | flatten | A Boolean value to indicate whether the list will be flattened into single items or if the values in the generated map will be lists. By default the values for the matching keys will be represented in an array. Set `flatten` to `true` to convert the array to a single value based on the value of `flattenedElement`. | No | false | | | flattenedElement | If you set flatten to true, use flattenedElement to specify which element, first or last, to keep. | Required when `flatten` is set to `true` | | Value can only be first or last | **Example** Take the following example log event: ``` { "outer_key": [ { "inner_key": "a", "inner_value": "val-a" }, { "inner_key": "b", "inner_value": "val-b1" }, { "inner_key": "b", "inner_value": "val-b2" }, { "inner_key": "c", "inner_value": "val-c" } ] } ``` **Transformer for use case 1:** `flatten` is `false` ``` [ { "parseJSON": {} }, { "listToMap": { "source": "outer_key" "key": "inner_key", "valueKey": "inner_value", "flatten": false } } ] ``` The transformed log event would be the following. ``` { "outer_key": [ { "inner_key": "a", "inner_value": "val-a" }, { "inner_key": "b", "inner_value": "val-b1" }, { "inner_key": "b", "inner_value": "val-b2" }, { "inner_key": "c", "inner_value": "val-c" } ], "a": [ "val-a" ], "b": [ "val-b1", "val-b2" ], "c": [ "val-c" ] } ``` **Transformer for use case 2:** `flatten` is `true` and `flattenedElement` is `first` ``` [ { "parseJSON": {} }, { "listToMap": { "source": "outer_key" "key": "inner_key", "valueKey": "inner_value", "flatten": true, "flattenedElement": "first" } } ] ``` The transformed log event would be the following. ``` { "outer_key": [ { "inner_key": "a", "inner_value": "val-a" }, { "inner_key": "b", "inner_value": "val-b1" }, { "inner_key": "b", "inner_value": "val-b2" }, { "inner_key": "c", "inner_value": "val-c" } ], "a": "val-a", "b": "val-b1", "c": "val-c" } ``` **Transformer for use case 3:** `flatten` is `true` and `flattenedElement` is `last` ``` [ { "parseJSON": {} }, { "listToMap": { "source": "outer_key" "key": "inner_key", "valueKey": "inner_value", "flatten": true, "flattenedElement": "last" } } ] ``` The transformed log event would be the following. ``` { "outer_key": [ { "inner_key": "a", "inner_value": "val-a" }, { "inner_key": "b", "inner_value": "val-b1" }, { "inner_key": "b", "inner_value": "val-b2" }, { "inner_key": "c", "inner_value": "val-c" } ], "a": "val-a", "b": "val-b2", "c": "val-c" } ``` # Datatype converter processors This section contains information about the datatype converter processors that you can use with a log event transformer. **Contents** + [typeConverter](#CloudWatch-Logs-Transformation-typeConverter) + [datetimeConverter](#CloudWatch-Logs-Transformation-datetimeConverter) ## typeConverter Use the `typeConverter` processor to convert a value type associated with the specified key to the specified type. It's a casting processor that changes the types of the specified fields. Values can be converted into one of the following datatypes: `integer`, `double`, `string` and `boolean`. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | entries | Array of entries. Each item in the array must contain key and type fields. | Yes | | Maximum entries: 10 | | key | The key with the value that is to be converted to a different type | Yes | | Maximum length: 128 Maximum nested key depth: 3 | | type | The type to convert to. Valid values are integer, double, string and boolean. | Yes | | | **Example** Take the following example log event: ``` { "name": "value", "status": "200" } ``` The transformer configuration is this, using `typeConverter` with `parseJSON`: ``` [ { "parseJSON": {} }, { "typeConverter": { "entries": [ { "key": "status", "type": "integer" } ] } } ] ``` The transformed log event would be the following. ``` { "name": "value", "status": 200 } ``` ## datetimeConverter Use the `datetimeConverter` processor to convert a datetime string into a format that you specify. | Field | Description | Required? | Default | Limits | | --- | --- | --- | --- | --- | | source | The key to apply the date conversion to. | Yes | | Maximum entries: 10 | | matchPatterns | A list of patterns to match against the source field | Yes | | Maximum entries: 5 | | target | The JSON field to store the result in. | Yes | | Maximum length: 128 Maximum nested key depth: 3 | | targetFormat | The datetime format to use for the converted data in the target field. | No | ` yyyy-MM-dd'T'HH:mm:ss.SSS'Z` | Maximum length:64 | | sourceTimezone | The time zone of the source field. For a list of possible values, see [ Java Supported Zone Ids and Offsets](https://howtodoinjava.com/java/date-time/supported-zone-ids-offsets/#3-java-supported-zone-ids-and-offsets). | No | UTC | Minimum length:1 | | targetTimezone | The time zone of the target field. For a list of possible values, see [ Java Supported Zone Ids and Offsets](https://howtodoinjava.com/java/date-time/supported-zone-ids-offsets/#3-java-supported-zone-ids-and-offsets). | No | UTC | Minimum length:1 | | locale | The locale of the source field. For a list of possible values, see [ Locale getAvailableLocales() Method in Java with Examples](https://www.geeksforgeeks.org/locale-getavailablelocales-method-in-java-with-examples/). | Yes | | Minimum length:1 | **Example** Take the following example log event: ``` {"german_datetime": "Samstag 05. Dezember 1998 11:00:00"} ``` The transformer configuration is this, using `dateTimeConverter` with `parseJSON`: ``` [ { "parseJSON": {} }, { "dateTimeConverter": { "source": "german_datetime", "target": "target_1", "locale": "de", "matchPatterns": ["EEEE dd. MMMM yyyy HH:mm:ss"], "sourceTimezone": "Europe/Berlin", "targetTimezone": "America/New_York", "targetFormat": "yyyy-MM-dd'T'HH:mm:ss z" } } ] ``` The transformed log event would be the following. ``` { "german_datetime": "Samstag 05. Dezember 1998 11:00:00", "target_1": "1998-12-05T17:00:00 MEZ" } ``` # Transformation metrics and errors CloudWatch Logs publishes transformation metrics to CloudWatch. These metrics include `TransformedLogEvents`, `TransformedBytes`, and `TransformationErrors`. For more information, see [Log transformer metrics and dimensions](CloudWatch-Logs-Monitoring-CloudWatch-Metrics.md#CloudWatchLogs-Transformer-Metrics). Whenever CloudWatch Logs tries and fails to transform a log event, it adds a `@transformationError` system field to that log event. When you run a CloudWatch Logs Insights query, you will see this field in all log events that had a transformation failure. You can query for this field with a query such as `filter ispresent(@transformationError)` to find all the failed transformation events.