# AWS service logs
Centralized Logging with OpenSearch supports ingesting AWS service logs into Amazon OpenSearch Service through log analytics pipelines, which you can build using the **Centralized Logging with OpenSearch web console** or via a **standalone CloudFormation template**.
Centralized Logging with OpenSearch reads the data source, parse, cleanup/enrich, and ingest logs into Amazon OpenSearch Service domains for analysis. Moreover, the solution provides templated dashboards to facilitate log visualization.
Amazon OpenSearch Service is suitable for real-time log analytics and frequent queries and has full-text search capability.
As of release 2.1.0, the solution starts to support log ingestion into Light Engine, which is suitable for non-real-time log analytics and infrequent queries and has SQL-like search capability. You will see an option to choose the desired log analytics engine when creating the log analytics pipeline
**Important**
Supported AWS services must be in the same Region as Centralized Logging with OpenSearch. To ingest logs from different AWS Regions, we recommend using [S3 Cross-Region Replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html#crr-scenario).
# Supported AWS services
Most of the supported AWS services output logs to Amazon CloudWatch Logs, Amazon S3, Amazon Kinesis Data Streams, or Amazon Kinesis DataFirehose. The log outputs must be in the same AWS Region as the Centralized Logging with OpenSearch solution.
The following table lists the supported AWS services and the supported log analytics engines.
| AWS Service | Log Type | OpenSearch Engine Support | Light Engine Support |
| --- | --- | --- | --- |
| AWS CloudTrail | N/A | Yes | Yes |
| Amazon S3 | [Access logs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html) | Yes | No |
| Amazon RDS/Aurora | [MySQL Logs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.MySQL.LogFileSize.html) | Yes | Yes |
| Amazon CloudFront | [Standard access logs](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html) | Yes | Yes |
| Application Load Balancer | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) | Yes | Yes |
| AWS WAF | [Web ACL logs](https://docs.aws.amazon.com/waf/latest/developerguide/logging.html) | Yes | Yes |
| AWS Lambda | N/A | Yes | No |
| Amazon VPC | [Flow logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) | Yes | Yes |
| AWS Config | N/A | Yes | No |
The solution supports detects the log location of the resource automatically, reads the logs, and then ingests them into the log analytics engines. The solution also provides dashboard templates for all supported AWS service. It automatically ingests logs into the log analytics engine. You can go to the OpenSearch Dashboards or Grafana to view the dashboards after the pipeline being provisioned.
In this chapter, you will learn how to create log ingestion and dashboards for the following AWS services:
+ [AWS CloudTrail](aws-cloudtrail-logs.md)
+ [Amazon S3](amazon-s3-logs.md)
+ [Amazon RDS/Aurora](amazon-rdsaurora-logs.md)
+ [Amazon CloudFront](amazon-cloudfront-logs.md)
+ [AWS Lambda](aws-lambda-logs.md)
+ [Application Load Balancer](application-load-balancer-application-load-balancer-logs.md)
+ [AWS WAF](aws-waf-logs.md)
+ [Amazon VPC](vpc-flow-logs.md)
+ [AWS Config](aws-config-logs.md)
# Cross-Region log ingestion
When you deploy Centralized Logging with OpenSearch in one Region, the solution allows you to ingest service logs from another Region.
**Note**
For Amazon RDS/Aurora and AWS Lambda service logs, this feature is not supported.
The Region where the service resides is referred to as "Source Region", while the Region where the Centralized Logging with OpenSearch console is deployed as "Logging Region".
For AWS CloudTrail, you can create a new trail that sends logs into a S3 bucket in the Logging Region, and you can find the CloudTrail in the list. To learn how to create a new trail, refer to [Creating a trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html).
For other services with logs located in S3 buckets, you can manually transfer logs (for example, using the S3 Cross-Region Replication feature) to the Logging Region S3 bucket.
You can complete the following steps to implement cross-Region log ingestion:
1. Set the service log location in another Region to be the Logging Region (such as AWS WAF), or automatically copy logs from the Source Region to the Logging Region using [Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-how-setup.html).
1. In the solution console, choose **AWS Service Log** in the left navigation pane, and choose **Create a pipeline**.
1. In the **Select an AWS Service** area, choose a service in the list, and choose **Next**.
1. In **Creation Method**, choose **Manual**, then enter the resource name and Amazon S3 log location parameter, and choose **Next**.
1. Change log analytics engines and log lifecycle settings, and choose **Next**.
1. Add tags if you need, and choose **Next** to create the pipeline.
Then you can use the OpenSearch dashboard or Grafana to discover logs and view dashboards.
# AWS CloudTrail logs
AWS CloudTrail monitors and records account activity across your AWS infrastructure. It outputs all the data to the specified S3 bucket or a CloudWatch Log Group.
You can create a log analytics pipeline either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.
**Important**
The CloudTrail logging bucket must be in the same Region as the Centralized Logging with OpenSearch solution.
The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.
## Create log ingestion (OpenSearch Engine)
### Using the Centralized Logging with OpenSearch console
1. Sign in to the Centralized Logging with OpenSearch console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose **Create a log ingestion**.
1. In the AWS Services section, choose AWS CloudTrail.
1. Choose **Next**.
1. Under Specify settings, choose Automatic or Manual.
+ For **Automatic** mode, choose a CloudTrail in the dropdown list.
+ For **Manual** mode, enter the CloudTrail name.
+ (Optional) If you are ingesting CloudTrail logs from another account, select a [linked account](cross-account-ingestion.md) from the **Account** dropdown list first.
1. Under **Log Source**, Select **Amazon S3** or **CloudWatch** as the log source.
1. Choose **Next**.
1. In **the Specify OpenSearch domain** section, select an imported domain for the Amazon OpenSearch Service domain.
1. Choose **Yes** for **Sample dashboard** if you want to ingest an associated built-in Amazon OpenSearch Service dashboard.
1. You can change the **Index Prefix** of the target Amazon OpenSearch Service index if needed. The default prefix is your trail name.
1. In the **Log Lifecycle** section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. In the **Select log processor** section, choose the log processor.
1. When selecting Lambda as a log processor, you can configure the Lambda concurrency if needed.
1. (Optional) OSI as log processor is now supported in these [Regions](https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-opensearch-service-ingestion/). When OSI is selected, type in the minimum and maximum number of OCU. See more information [here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html#ingestion-scaling).
1. Choose **Next**.
1. Add tags if needed.
1. Choose **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - CloudTrail Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudTrailLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudTrailLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudTrailLog.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudTrailLog.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudTrailLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudTrailLog.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-cloudtrail-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates IAM resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Global Control | awsRegion | Provides users with the ability to drill down data by Region. |
| Event History | log event | Presents a bar chart that displays the distribution of events over time. |
| Event by Account ID | userIdentity.accountId | Breaks down events based on the AWS account ID, enabling you to analyze activity patterns across different accounts within your organization. |
| Top Event Names | eventName | Shows the most frequently occurring event names, helping you identify common activities or potential anomalies. |
| Top Event Sources | eventSource | Highlights the top sources generating events, providing insights into the services or resources that are most active or experiencing the highest event volume. |
| Event Category | eventCategory | Categorizes events into different types or classifications, facilitating analysis and understanding of event distribution across categories. |
| Top Users | \$1 userIdentity.sessionContext.sessionIssuer.userName \$1 userIdentity.sessionContext.sessionIssuer.arn \$1 userIdentity.accountId \$1 userIdentity.sessionContext.sessionIssuer.type | Identifies the users or IAM roles associated with the highest number of events, aiding in user activity monitoring and access management. |
| Top Source IPs | sourceIPAddress | Lists the source IP addresses associated with events, enabling you to identify and investigate potentially suspicious or unauthorized activities. |
| Amazon S3 Access Denied | \$1 eventSource: s3\$1 \$1 errorCode: AccessDenied | Displays events where access to Amazon S3 resources was denied, helping you identify and troubleshoot permission issues or potential security breaches. |
| S3 Buckets | requestParameters.bucketName | Provides a summary of S3 bucket activity, including create, delete, and modify operations, allowing you to monitor changes and access patterns. |
| Top Amazon S3 Change Events | \$1 eventName \$1 requestParameters.bucketName | Presents the most common types of changes made to Amazon S3 resources, such as object uploads, deletions, or modifications, aiding in change tracking and auditing. |
| EC2 Change Event Count | \$1 eventSource: ec2\$1 \$1 eventName: (RunInstances or TerminateInstances or RunInstances or StopInstances) | Shows the total count of EC2-related change events, giving an overview of the volume and frequency of changes made to EC2 instances and resources. |
| EC2 Changed By | userIdentity.sessionContext.sessionIssuer.userName | Identifies the users or IAM roles responsible for changes to EC2 resources, assisting in accountability and tracking of modifications. |
| Top EC2 Change Events | eventName | Highlights the most common types of changes made to EC2 instances or related resources, allowing you to focus on the most significant or frequent changes. |
| Error Events | \$1 awsRegion \$1 errorCode \$1 errorMessage \$1 eventName \$1 eventSource \$1 sourceIPAddress \$1 userAgent \$1 userIdentity.accountId \$1 userIdentity.sessionContext.sessionIssuer.accountId \$1 userIdentity.sessionContext.sessionIssuer.arn \$1 userIdentity.sessionContext.sessionIssuer.type \$1 userIdentity.sessionContext.sessionIssuer.userName | Displays events that resulted in errors or failures, helping you identify and troubleshoot issues related to API calls or resource operations. |
You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the [Access Dashboard](getting-started.md#step-4-access-the-dashboard).
You can choose the following image to view the high-resolution sample dashboard.
**CloudTrail logs sample dashboard.**
![\[image32\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image32.png)
## Create log ingestion (Light Engine)
### Using the Centralized Logging with OpenSearch console
1. Sign in to the Centralized Logging with OpenSearch console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose **Create a log ingestion**.
1. In the **AWS Services** section, choose AWS CloudTrail.
1. Choose **Next**.
1. Under Specify settings, choose Automatic or Manual.
+ For **Automatic** mode, choose a CloudTrail in the dropdown list.
+ For **Manual** mode, enter the CloudTrail name.
+ (Optional) If you are ingesting CloudTrail logs from another account, select a [linked account](cross-account-ingestion.md) from the **Account** dropdown list first.
1. Choose **Next**.
1. In the **Specify Light Engine Configuration** section, if you want to ingest associated templated Grafana dashboards, select **Yes** for the sample dashboard.
1. You can choose an existing Grafana, or if you must import a new one, you can go to Grafana for configuration.
1. Select an S3 bucket to store partitioned logs and define a name for the log table. We have provided a predefined table name, but you can modify it according to your business needs.
1. If needed, change the log processing frequency, which is set to **5** minutes by default, with a minimum processing frequency of **1** minute.
1. In the **Log Lifecycle** section, enter the log merge time and log archive time. We have provided default values, but you can adjust them based on your business requirements.
1. Select **Next**.
1. If desired, add tags.
1. Select **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - CloudTrail Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudTrailPipeline.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudTrailPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudTrailPipeline.template) |
| AWS China Regions | ![\[Launch solution\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image17.png) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudTrailPipeline.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
1. Parameters for **Pipeline settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-cloudtrail-logs.html)
1. Parameters for **Destination settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-cloudtrail-logs.html)
1. Parameters for **Scheduler settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-cloudtrail-logs.html)
1. Parameters for **Notification settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-cloudtrail-logs.html)
1. Parameters for **Dashboard settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-cloudtrail-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
**CloudTrail log sample dashboard**
![\[image33\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image33.png)
# Amazon S3 logs
[Amazon S3 server access logging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html) provides detailed records for the requests made to the bucket. S3 Access Logs can be enabled and saved in another S3 bucket.
You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.
**Important**
The S3 Bucket Region must be the same as the Centralized Logging with OpenSearch solution Region.
The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.
## Create log ingestion (OpenSearch Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the **AWS Services** section, choose **Amazon S3**.
1. Choose **Next**.
1. Under **Specify settings**, choose **Automatic** or **Manual** for **Amazon S3 Access Log enabling**. The automatic mode will enable the Amazon S3 Access Log and save the logs to a centralized S3 bucket if logging is not enabled yet.
+ For **Automatic mode**, choose the S3 bucket from the dropdown list.
+ For Manual mode, enter the Bucket Name and Amazon S3 Access Log location.
+ (Optional) If you are ingesting Amazon S3 logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown list first.
1. Choose **Next**.
1. In the Specify OpenSearch domain section, select an imported domain for the Amazon OpenSearch Service domain.
1. Choose **Yes** for **Sample dashboard** if you want to ingest an associated built-in Amazon OpenSearch Service dashboard.
1. You can change the **Index Prefix** of the target Amazon OpenSearch Service index if needed. The default prefix is your bucket name.
1. In the **Log Lifecycle** section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. Choose **Next**.
1. Add tags if needed.
1. Choose **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - Amazon S3 Access Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/S3AccessLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/S3AccessLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/S3AccessLog.template) |
| AWS China Regions | ![\[Launch solution\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image17.png) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/S3AccessLog.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-s3-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Total Requests | \$1 log event | A visualization showing the total number of requests made to the Amazon S3 bucket, including all types of operations (for example, GET, PUT, DELETE). |
| Unique Visitors | \$1 log event | This visualization displays the count of unique visitors accessing the Amazon S3 bucket, identified by their IP addresses. |
| Access History | \$1 log event | Provides a chronological log of all access events made to the Amazon S3 bucket, including details about the operations and their outcomes. |
| Request By Operation | \$1 operation | This visualization categorizes and shows the distribution of requests based on different operations (for example, GET, PUT, DELETE). |
| Status Code | \$1 http\$1status | Displays the count of requests made to the Amazon S3 bucket, grouped by HTTP status codes returned by the server (for example, 200, 404, 403). |
| Status Code History | \$1 http\$1status | Shows the historical trend of HTTP status codes returned by the Amazon S3 server over a specific period of time. |
| Status Code Pie | \$1 http\$1status | Represents the distribution of requests based on different HTTP status codes using a pie chart. |
| Average Time | \$1 total\$1time | This visualization calculates and presents the average time taken for various operations in the Amazon S3 bucket (for example, average time for GET, PUT requests). |
| Average Turn Around Time | \$1 turn\$1around\$1time | Shows the average turnaround time for different operations, which is the time between receiving a request and sending the response back to the client. |
| Data Transfer | \$1 bytes\$1sent \$1 object\$1size \$1 operation | Provides insights into data transfer activities, including the total bytes transferred, object sizes, and different operations involved. |
| Top Client IPs | \$1 remote\$1ip | Displays the top client IP addresses with the highest number of requests made to the Amazon S3 bucket. |
| Top Request Keys | \$1 key \$1 object\$1size | Shows the top requested keys in the Amazon S3 bucket along with the corresponding object sizes. |
| Delete Events | \$1 operation \$1 key \$1 version\$1id \$1 object\$1size \$1 remote\$1ip \$1 http\$1status \$1 error\$1code | Focuses on delete events, including the operation, key, version ID, object size, client IP, HTTP status, and error code associated with the delete requests. |
| Access Failures | \$1 operation \$1 key \$1 version\$1id \$1 object\$1size \$1 remote\$1ip \$1 http\$1status \$1 error\$1code | Highlights access failures, showing the details of the failed requests, including operation, key, version ID, object size, client IP, HTTP status, and error code. |
You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the [Access Dashboard](getting-started.md#step-4-access-the-dashboard).
**Amazon S3 logs sample dashboard.**
![\[image34\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image34.png)
# Amazon RDS/Aurora logs
You can [publish database instance logs to Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Procedural.UploadtoCloudWatch.html). Then, you can perform real-time analysis of the log data, store the data in highly durable storage, and manage the data with the CloudWatch Logs Agent.
Prerequisites
Make sure your database logs are enabled. Some databases logs are not enabled by default, and you must update your database parameters to enable the logs.
Refer to [How do I enable and monitor logs for an Amazon RDS MySQL DB instance?](https://aws.amazon.com/premiumsupport/knowledge-center/rds-mysql-logs/) to learn how to output logs to CloudWatch Logs.
The following table lists the requirements for Amazon RDS/Aurora MySQL parameters.
| Parameter | Requirement |
| --- | --- |
| Audit Log | The database instance must use a custom option group with the MARIADB\$1AUDIT\$1PLUGIN option. |
| General log | The database instance must use a custom parameter group with the parameter setting general\$1log = 1 to enable the general log. |
| Slow query log | The database instance must use a custom parameter group with the parameter setting slow\$1query\$1log = 1 to enable the slow query log. |
| Log output | The database instance must use a custom parameter group with the parameter setting log\$1output = FILE to write logs to the file system and publish them to CloudWatch Logs. |
You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.
**Important**
The Amazon RDS and CloudWatch Region must be the same as the Centralized Logging with OpenSearch solution Region.
The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.
## Create log ingestion (OpenSearch Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the **AWS Services** section, choose **Amazon RDS**.
1. Choose **Next**.
1. Under **Specify settings**, choose **Automatic** or **Manual** for **RDS log enabling**. The automatic mode will detect your Amazon RDS log configurations and ingest logs from CloudWatch.
+ For **Automatic mode**, choose the Amazon RDS cluster from the dropdown list.
+ For **Manual mode**, enter the **DB identifier**, select the **Database type** and input the CloudWatch log location in **Log type and location**.
+ (Optional) If you are ingesting Amazon RDS/Aurora logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown first.
1. Choose **Next**.
1. In the Specify OpenSearch domain section, select an imported domain for the Amazon OpenSearch Service domain.
1. Choose **Yes** for **Sample dashboard** if you want to ingest an associated templated Amazon OpenSearch Service dashboard.
1. You can change the **Index Prefix** of the target Amazon OpenSearch Service index if needed. The default prefix is the Database identifier.
1. In the **Log Lifecycle** section, input the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. In the **Log processor settings** section, choose **Log processor type**, configure the Lambda concurrency if needed, and then choose **Next**.
1. Add tags if needed.
1. Choose **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - RDS Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/RDSLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/RDSLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/RDSLog.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/RDSLog.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/RDSLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/RDSLog.template) |
1. Log in to the AWS Management Console and select the button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the Centralized Logging with OpenSearch in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-rdsaurora-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 15 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Controller | \$1 db-identifier \$1 sq-table-name | This visualization allows users to filter data based on the db-identifier and sq-table-name fields. |
| Total Log Events Overview | \$1 db-identifier \$1 log event | This visualization presents an overview of the total log events for the specified database ('db-identifier'). It helps monitor the frequency of various log events. |
| Slow Query History | \$1 log event | This visualization shows the historical data of slow query log events. It allows you to track the occurrences of slow queries and identify potential performance issues. |
| Average Slow Query Time History | \$1 Average sq-duration | This visualization depicts the historical trend of the average duration of slow queries ('sq-duration'). It helps in understanding the database’s performance over time and identifying trends related to slow query durations. |
| Total Slow Queries | \$1 log event | This visualization provides the total count of slow queries in the log events. It gives an immediate view of how many slow queries have occurred during a specific time period, which is useful for assessing the database’s performance and potential bottlenecks. |
| Average Slow Query Duration | \$1 Average sq-duration | This visualization shows the average duration of slow queries ('sq-duration') over time. It is valuable for understanding the typical performance of slow queries in the database. |
| Top Slow Query IP | \$1 sq-ip \$1 sq-duration | This visualization highlights the IP addresses ('sq-ip') associated with the slowest queries and their respective durations ('sq-duration'). It helps identify sources of slow queries and potential areas for optimization. |
| Slow Query Scatter Plot | \$1 sq-duration \$1 sq-ip \$1 sq-query | This scatter plot visualization represents the relationship between the duration of slow queries ('sq-duration'), the IP addresses ('sq-ip') from which they originated, and the actual query content ('sq-query'). It helps in understanding query performance patterns and identifying potential issues related to specific queries and their sources. |
| Slow Query Pie | \$1 sq-query | This pie chart visualization shows the distribution of slow queries based on their content ('sq-query'). It provides an overview of the types of queries causing performance issues, allowing you to focus on optimizing specific query patterns. |
| Slow Query Table Name Pie | \$1 sq-table-name | This pie chart visualization displays the distribution of slow queries based on the table names ('sq-table-name') they access. It helps identify which tables are affected by slow queries, enabling targeted optimization efforts for specific tables. |
| Top Slow Query | \$1 sq-query | This visualization presents the slowest individual queries based on their content ('sq-query'). It is helpful in pinpointing specific queries that have the most significant impact on performance, allowing developers and administrators to focus on optimizing these critical queries. |
| Slow Query Logs | \$1 db-identifier \$1 sq-db-name \$1 sq-table-name \$1 sq-query \$1 sq-ip \$1 sq-host-name \$1 sq-rows-examined \$1 sq-rows-sent \$1 sq-id \$1 sq-duration \$1 sq-lock-wait | This visualization provides detailed logs of slow queries, including database ('sq-db-name'), table ('sq-table-name'), query content ('sq-query'), IP address ('sq-ip'), hostname ('sq-host-name'), rows examined ('sq-rows-examined'), rows sent ('sq-rows-sent'), query ID ('sq-id'), query duration ('sq-duration'), and lock wait time ('sq-lock-wait'). It is beneficial for in-depth analysis and troubleshooting of slow query performance. |
| Total Deadlock Queries | \$1 log event | This visualization shows the total number of deadlock occurrences based on the log events. Deadlocks are critical issues that can cause database transactions to fail, and monitoring their frequency is essential for database stability. |
| Deadlock History | \$1 log event | This visualization displays the historical data of deadlock occurrences based on the log events. Understanding the pattern of deadlocks over time can help identify recurring issues and take preventive measures to reduce their impact on the database. |
| Deadlock Query Logs | \$1 db-identifier \$1 log-detail \$1 deadlock-ip-1 \$1 deadlock-action-1 \$1 deadlock-os-thread-handle-1 \$1 deadlock-query-1 \$1 deadlock-query-id-1 \$1 deadlock-thread-id-1 \$1 deadlock-user-1 \$1 deadlock-action-2 \$1 deadlock-ip-2 \$1 deadlock-os-thread-handle-2 \$1 deadlock-query-2 \$1 deadlock-query-id-2 \$1 deadlock-thread-id-2 \$1 deadlock-user-2 | This visualization provides detailed logs of deadlock occurrences |
| Total Error Logs | \$1 log event | This visualization presents the total count of error log events. Monitoring error logs helps identify database issues and potential errors that need attention and resolution. |
| Error History | \$1 log event | This visualization shows the historical data of error log events. Understanding the error patterns over time can aid in identifying recurring issues and taking corrective actions to improve the database’s overall health and stability. |
| Error Logs | \$1 db-identifier \$1 err-label \$1 err-code \$1 err-detail \$1 err-sub-system \$1 err-thread | This visualization displays the error logs generated by the Amazon RDS instance. It provides valuable insights into any errors, warnings, or issues encountered within the database system, helping to identify and troubleshoot problems effectively. Monitoring error logs is essential for maintaining the health and reliability of the database. |
| Audit History | \$1 log event | This visualization presents the audit history of the Amazon RDS instance. It tracks the various log events and activities related to database access, modifications, and security-related events. Monitoring the audit logs is crucial for compliance, detecting unauthorized access, and tracking changes made to the database. |
| Audit Logs | \$1 db-identifier \$1 audit-operation \$1 audit-ip \$1 audit-query \$1 audit-retcode \$1 audit-connection-id \$1 audit-host-name \$1 audit-query-id \$1 audit-user | This visualization provides an overview of the audit logs generated by the Amazon RDS instance. It shows the operations performed on the database, including queries executed, connection details, IP addresses, and associated users. Monitoring audit logs enhances the security and governance of the database, helping to detect suspicious activities and track user actions. |
You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the [Access Dashboard](getting-started.md#step-4-access-the-dashboard).
**Amazon RDS/Aurora logs sample dashboard.**
![\[image35\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image35.png)
## Create log ingestion (Light Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the **AWS Services** section, choose **Amazon RDS**.
1. Choose **Light Engine**, choose **Next**.
1. Under **Specify settings**, choose **Automatic** or **Manual** for **RDS log enabling**. The automatic mode will detect your Amazon RDS log configurations and ingest logs from CloudWatch.
+ For **Automatic mode**, choose the Amazon RDS cluster from the dropdown list.
+ For **Manual mode**, enter the **DB identifier**, select the **Database type** and input the CloudWatch log location in **Log type and location**.
+ (Optional) If you are ingesting Amazon RDS/Aurora logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown first.
1. Choose **Next**.
1. In the **Specify Light Engine Configuration** section, if you want to ingest associated templated Grafana dashboards, select **Yes** for the sample dashboard.
1. You can choose an existing Grafana, or if you must import a new one, you can go to Grafana for configuration.
1. Select an S3 bucket to store partitioned logs and define a name for the log table. We have provided a predefined table name, but you can modify it according to your business needs.
1. If needed, change the log processing frequency, which is set to **5** minutes by default, with a minimum processing frequency of **1** minute.
1. In the **Log Lifecycle** section, enter the log merge time and log archive time. We have provided default values, but you can adjust them based on your business requirements.
1. Select **Next**.
1. If desired, add tags.
1. Select **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - RDS Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesRDSPipeline.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesRDSPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesRDSPipeline.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesRDSPipeline.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesRDSPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesRDSPipeline.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
1. Parameters for **Pipeline settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-rdsaurora-logs.html)
1. Parameters for **Destination settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-rdsaurora-logs.html)
1. Parameters for **Scheduler settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-rdsaurora-logs.html)
1. Parameters for **Notification settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-rdsaurora-logs.html)
1. Parameters for **Dashboard settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-rdsaurora-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates IAM resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
**Amazon RDS/Aurora logs sample dashboard.**
![\[image36\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image36.jpeg)
# Amazon CloudFront logs
[CloudFront standard logs](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html) provide detailed records about every request made to a distribution.
You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.
**Important**
The CloudFront logging bucket must be the same Region as the Centralized Logging with OpenSearch solution.
The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.
## Create log ingestion (OpenSearch Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the AWS Services section, choose Amazon CloudFront.
1. Choose **Next**.
1. Under **Specify settings**, choose **Automatic** or **Manual** for **CloudFront logs enabling**. The automatic mode will detect the CloudFront log location automatically.
+ For **Automatic mode**, choose the CloudFront distribution and Log Type from the dropdown list.
+ For Standard Log, the solution will automatically detect the log location if logging is enabled.
+ For Real-time log, the solution will prompt you for confirmation to create or replace the CloudFront real-time log configuration.
+ For **Manual mode**, enter the **CloudFront Distribution ID** and **CloudFront Standard Log location**. (Note that CloudFront real-time log is not supported in Manual mode)
+ (Optional) If you are ingesting CloudFront logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown list first.
1. Choose **Next**.
1. In the Specify OpenSearch domain section, select an imported domain for the Amazon OpenSearch Service domain.
1. Choose **Yes** for **Sample dashboard** if you want to ingest an associated templated Amazon OpenSearch Service dashboard.
1. You can change the **Index Prefix** of the target Amazon OpenSearch Service index if needed. The default prefix is the CloudFront distribution ID.
1. In the **Log Lifecycle** section, input the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. In the **Log processor settings** section, choose **Log processor type**, configure the Lambda concurrency if needed, and then choose **Next**.
1. Add tags if needed.
1. Choose **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - CloudFront Standard Log Ingestion* template in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudFrontLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudFrontLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudFrontLog.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudFrontLog.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudFrontLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/CloudFrontLog.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-cloudfront-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Total Requests | \$1 log event | Displays the total number of viewer requests received by the Amazon CloudFront, for all HTTP methods and for both HTTP and HTTPS requests. |
| Edge Locations | \$1 x-edge-location | Shows a pie chart representing the proportion of the locations of CloudFront edge servers. |
| Request History | \$1 log event | Presents a bar chart that displays the distribution of events over time. |
| Unique Visitors | \$1 c-ip | Displays unique visitors identified by client IP address. |
| Cache Hit Rate | \$1 sc-bytes | Shows the proportion of your viewer requests that are served directly from the CloudFront cache instead of going to your origin servers for content. |
| Result Type | \$1 x-edge-response-result-type | Shows the percentage of hits, misses, and errors to the total viewer requests for the selected CloudFront distribution: \$1 Hit - A viewer request for which the object is served from a CloudFront edge cache. In access logs, these are requests for which the value of x-edge-response-result-type is Hit \$1 Miss - A viewer request for which the object isn’t currently in an edge cache, so CloudFront must get the object from your origin. In access logs, these are requests for which the value of x-edge-response-result-type is Miss. \$1 Error - A viewer request that resulted in an error, so CloudFront didn’t serve the object. In access logs, these are requests for which the value of x-edge-response-result-type is Error, LimitExceeded, or CapacityExceeded. The chart does not include refresh hits—requests for objects that are in the edge cache but that have expired. In access logs, refresh hits are requests for which the value of x-edge-response-result-type is RefreshHit. |
| Top Miss URI | \$1 cs-uri-stem \$1 cs-method | Shows top 10 of the requested objects that are not in the cache. |
| Bandwidth | \$1 cs-bytes \$1 sc-bytes | Provides insights into data transfer activities from the locations of CloudFront edge. |
| Bandwidth History | \$1 cs-bytes \$1 sc-bytes | Shows the historical trend of the data transfer activities from the locations of CloudFront edge. |
| Top Client IPs | \$1 c-ip | Provides the top 10 IP address accessing your Amazon CloudFront. |
| Status Code Count | \$1 sc-status | Displays the count of requests made to the Amazon CloudFront, grouped by HTTP status codes(e.g., 200, 404, 403, etc.). |
| Status History | \$1 @timestamp \$1 sc-status | Shows the historical trend of HTTP status codes returned by the Amazon CloudFront over a specific period of time. |
| Status Code | \$1 sc-status | Identifies the users or IAM roles responsible for changes to EC2 resources, assisting in accountability and tracking of modifications. |
| Average Time Taken | \$1 time-taken | This visualization calculates and presents the average time taken for various operations in the Amazon CloudFront (e.g., average time for GET, PUT requests, etc.). |
| Average Time History | \$1 time-taken \$1 time-to-first-byte \$1 @timestamp | Shows the historical trend of the average time taken for various operations in the Amazon CloudFront. |
| Http Method | \$1 cs-method | Displays the count of requests made to the Amazon CloudFront using a pie chart, grouped by HTTP request method names (for example, POST, GET, HEAD). |
| Average Time To First Byte | \$1 time-to-first-byte | Provides the average time taken in seconds by the origin server to respond back with the first byte of the response. |
| Top Request URIs | \$1 cs-uri-stem \$1 cs-method | Provides the top 10 request URIs accessing your CloudFront. |
| Top User Agents | \$1 cs-user-agent | Provides the top 10 user agents accessing your CloudFront. |
| Edge Location Heatmap | \$1 x-edge-location \$1 x-edge-result-type | Shows a heatmap representing the result type of each edge location. |
| Top Referrers | \$1 cs-referer | Top 10 referrers with the Amazon CloudFront access. |
| Top Countries or Regions | \$1 c\$1country | Top 10 countries with the Amazon CloudFront access. |
You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the [Access Dashboard](getting-started.md#step-4-access-the-dashboard).
**CloudFront logs sample dashboard.**
![\[image37\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image37.png)
## Create log ingestion (Light Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the AWS Services section, choose Amazon CloudFront.
1. Choose **Next**.
1. Under **Specify settings**, choose **Automatic** or **Manual** for **CloudFront logs enabling**. The automatic mode will detect the CloudFront log location automatically.
+ For **Automatic mode**, choose the CloudFront distribution and Log Type from the dropdown list.
+ For Standard Log, the solution will automatically detect the log location if logging is enabled.
+ For Real-time log, the solution will prompt you for confirmation to create or replace the CloudFront real-time log configuration.
+ For **Manual mode**, enter the **CloudFront Distribution ID** and **CloudFront Standard Log location**. (Note that CloudFront real-time log is not supported in Manual mode)
+ (Optional) If you are ingesting CloudFront logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown list first.
1. Choose **Next**.
1. Choose **Log Processing Enriched fields** if needed. The available plugins are **location** and **OS/User Agent**. Enabling rich fields increases data processing latency and processing costs. By default, it is not selected.
1. In the **Specify Light Engine Configuration** section, if you want to ingest associated templated Grafana dashboards, select **Yes** for the sample dashboard.
1. You can choose an existing Grafana, or if you must import a new one, you can go to Grafana for configuration.
1. Select an S3 bucket to store partitioned logs and define a name for the log table. We have provided a predefined table name, but you can modify it according to your business needs.
1. If needed, change the log processing frequency, which is set to **5** minutes by default, with a minimum processing frequency of **1** minute.
1. In the **Log Lifecycle** section, enter the log merge time and log archive time. We have provided default values, but you can adjust them based on your business requirements.
1. Select **Next**.
1. If desired, add tags.
1. Select **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - CloudFront Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudFrontPipeline.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudFrontPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudFrontPipeline.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudFrontPipeline.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudFrontPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesCloudFrontPipeline.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
1. Parameters for **Pipeline settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-cloudfront-logs.html)
1. Parameters for **Destination settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-cloudfront-logs.html)
1. Parameters for **Scheduler settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-cloudfront-logs.html)
1. Parameters for **Notification settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-cloudfront-logs.html)
1. Parameters for **Dashboard settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/amazon-cloudfront-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Filters | Filters | The following data can be filtered by query filter conditions. |
| Total Requests | log event | Displays the total number of viewer requests received by the Amazon CloudFront, for all HTTP methods and for both HTTP and HTTPS requests. |
| Unique Visitors | c-ip | Displays unique visitors identified by client IP address. |
| Requests History | log event | Presents a bar chart that displays the distribution of events over time. |
| Request By Edge Location | x-edge-location | Shows a pie chart representing the proportion of the locations of CloudFront edge servers. |
| HTTP Status Code | sc-status | Displays the count of requests made to the Amazon CloudFront, grouped by HTTP status codes (e.g., 200, 404, 403, etc.). |
| Status Code History | sc-status | Shows the historical trend of HTTP status codes returned by the Amazon CloudFront over a specific period of time. |
| Status Code Pie | sc-status | Represents the distribution of requests based on different HTTP status codes using a pie chart. |
| Average Processing Time | time-taken time-to-first-byte | This visualization calculates and presents the average time taken for various operations in the Amazon CloudFront (for example, average time for GET, PUT requests). |
| Avg. Processing Time History | time-taken time-to-first-byte | Shows the historical trend of the average time taken for various operations in the Amazon CloudFront. |
| Avg. Processing Time History | time-taken time-to-first-byte | Shows the historical trend of the average time taken for various operations in the Amazon CloudFront. |
| HTTP Method | cs-method | Displays the count of requests made to the Amazon CloudFront using a pie chart, grouped by HTTP request method names (for example, POST, GET, HEAD). |
| Total Bytes | cs-bytes sc-bytes | Provides insights into data transfer activities, including the total bytes transferred. |
| Response Bytes History | cs-bytes sc-bytes | Displays the historical trend of the received bytes, send bytes. |
| Edge Response Type | x-edge-response-result-type | Shows the percentage of hits, misses, and errors to the total viewer requests for the selected CloudFront distribution: - Hit - A viewer request for which the object is served from a CloudFront edge cache. In access logs, these are requests for which the value of x-edge-response-result-type is Hit. - Miss - A viewer request for which the object isn’t currently in an edge cache, so CloudFront must get the object from your origin. In access logs, these are requests for which the value of x-edge-response-result-type is Miss. - Error - A viewer request that resulted in an error, so CloudFront didn’t serve the object. In access logs, these are requests for which the value of x-edge-response-result-type is Error, LimitExceeded, or CapacityExceeded. The chart does not include refresh hits—requests for objects that are in the edge cache but that have expired. In access logs, refresh hits are requests for which the value of x-edge-response-result-type is RefreshHit. |
| Requests / Origin Requests | log event | Displays the number of requests made to CloudFront and the number of requests back to the origin. |
| Requests / Origin Requests Latency | log event time-taken | Displays the request latency from the client to CloudFront and the request latency back to the origin. |
| Top 20 URLs with most requests | log event | Top 20 URLs based on the number of requests. |
| Requests 3xx / 4xx / 5xx error rate | log event sc-status | Displays the ratio of 3xx/4xx/5xx status codes from the client to CloudFront. |
| Origin Requests 3xx / 4xx / 5xx error rate | log event sc-status x-edge-detailed-result-type | Display the proportion of 3xx/4xx/5xx status codes returned to the origin. |
| Requests 3xx / 4xx / 5xx error latency | log event sc-status time-taken | Displays the latency from the client to CloudFront for 3xx/4xx/5xx status codes. |
| Origin Requests 3xx / 4xx / 5xx error latency | log event sc-status x-edge-detailed-result-type time-taken | Displays the delay in returning to the source 3xx/4xx/5xx status code. |
| Response Latency (>= 1sec) rate | log event time-taken | Display the proportion of delay above 1s. |
| Bandwidth | sc-bytes | Displays the bandwidth from the client to CloudFront and the bandwidth back to the origin. |
| Data transfer | sc-bytes | Display the response traffic. |
| Top 20 URLs with most traffic | cs-uri-stem sc-bytes | Top 20 URLs calculated by traffic. |
| Cache hit rate (calculated using requests) | log event x-edge-result-type | Displays the cache hit ratio calculated by the number of requests. |
| Cache hit rate (calculated using bandwidth) | log event sc-bytes x-edge-result-type | Displays the cache hit ratio calculated by bandwidth. |
| Cache Result | log event x-edge-result-type | Displays the number of requests of various x-edge-result-types, such as the number of requests that hit the cache and the number of requests that missed the cache. |
| Cache Result Latency | log event sc-bytes x-edge-result-type | Displays the request latency of various x-edge-result-types, such as the request latency that hits the cache and the request latency that misses the cache. |
| Requests by OS | ua\$1os | Displays the count of requests made to the Application Load Balancer, grouped by user agent OS. |
| Requests by Device | ua\$1device | Displays the count of requests made to the Application Load Balancer, grouped by user agent device. |
| Requests by Browser | ua\$1browser | Displays the count of requests made to the Application Load Balancer, grouped by user agent browser. |
| Requests by Category | ua\$1category | Displays the count of categories made to the Application Load Balancer, grouped by user agent category (for example, PC, Mobile, Tablet). |
| Requests by Countries or Regions | geo\$1iso\$1code | Displays the count of requests made to the Application Load Balancer (grouped by the corresponding country or Region resolved by the client IP). |
| Top Countries or Regions | geo\$1country | Top 10 countries with the Application Load Balancer Access. |
| Top Cities | geo\$1city | Top 10 cities with Application Load Balancer Access. |
**CloudFront logs sample dashboard.**
![\[image38\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image38.png)
# AWS Lambda logs
AWS Lambda automatically monitors Lambda functions on your behalf and sends function metrics to Amazon CloudWatch.
You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.
**Important**
The Lambda Region must be the same as the Centralized Logging with OpenSearch solution.
The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.
## Create log ingestion (OpenSearch Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the **AWS Services** section, choose **AWS Lambda**.
1. Choose **Next**.
1. Under **Specify settings**, choose the Lambda function from the dropdown list. (Optional) If you are ingesting logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown first.
1. Choose **Next**.
1. In the **Specify OpenSearch domain** section, select an imported domain for the Amazon OpenSearch Service domain.
1. Choose **Yes** for **Sample dashboard** if you want to ingest an associated templated Amazon OpenSearch Service dashboard.
1. You can change the **Index Prefix** of the target Amazon OpenSearch Service index if needed. The default prefix is the Lambda function name.
1. In the **Log Lifecycle** section, input the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. In the **Log processor settings** section, choose **Log processor type**, configure the Lambda concurrency if needed, and then choose **Next**.
1. Add tags if needed.
1. Choose **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - Lambda Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/LambdaLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/LambdaLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/LambdaLog.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/LambdaLog.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/LambdaLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/LambdaLog.template) |
1. Log in to the AWS Management Console and select the button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the Centralized Logging with OpenSearch in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-lambda-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 15 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Lambda Events | \$1 log event | Presents a chart that displays the distribution of events over time. |
| Log Accounts | \$1 owner | Shows a pie chart representing the proportion of log events from different AWS accounts (owners). |
| Log Groups | \$1 log\$1group | Displays a pie chart depicting the distribution of log events among various log groups in the Lambda environment. |
| Log-List | \$1 time \$1 log\$1group \$1 log\$1stream \$1 log\$1detail | Provides a detailed list of log events, including timestamps, log groups, log streams, and log details. |
You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the [Access Dashboard](getting-started.md#step-4-access-the-dashboard).
**Lambda logs sample dashboard.**
![\[image39\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image39.png)
# Application Load Balancer logs
[Application Load Balancer access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) capture detailed information about requests sent to your load balancer. Application Load Balancer publishes a log file for each load balancer node every 5 minutes.
You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.
**Important**
The Elastic Load Balancing logging bucket must be the same as the Centralized Logging with OpenSearch solution.
The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.
## Create log ingestion (OpenSearch Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the AWS Services section, choose **Elastic Load Balancer**.
1. Choose **Next**.
1. Under Specify settings, choose Automatic or Manual.
+ For **Automatic** mode, choose an Application Load Balancer in the dropdown list. (If the selected Application Load Balancer access log is not enabled, choose **Enable** to enable the Application Load Balancer access log.)
+ For Manual mode, enter the Application Load Balancer identifier and Log location.
+ (Optional) If you are ingesting logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown first.
1. Choose **Next**.
1. In the Specify OpenSearch domain section, select an imported domain for the Amazon OpenSearch Service domain.
1. Choose **Yes** for **Sample dashboard** if you want to ingest an associated templated Amazon OpenSearch Service dashboard.
1. You can change the **Index Prefix** of the target Amazon OpenSearch Service index if needed. The default prefix is the Load Balancer Name.
1. In the **Log Lifecycle** section, input the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. In the **Log Lifecycle** section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. In the **Select log processor** section, choose the log processor.
1. When selecting Lambda as a log processor, you can configure the Lambda concurrency if needed.
1. (Optional) OSI as log processor is now supported in these [Regions](https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-opensearch-service-ingestion/). When OSI is selected, type in the minimum and maximum number of OCU. See more information [here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html#ingestion-scaling).
1. Choose **Next**.
1. Add tags if needed.
1. Choose **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - ELB Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ELBLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ELBLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ELBLog.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ELBLog.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ELBLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ELBLog.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/application-load-balancer-application-load-balancer-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Total Requests | \$1 log event | Displays aggregated events based on a specified time interval. |
| Request History | \$1 log event | Presents a bar chart that displays the distribution of events over time. |
| Request By Target | \$1 log event \$1 target\$1ip | Presents a bar chart that displays the distribution of events over time and IP. |
| Unique Visitors | \$1 client\$1ip | Displays unique visitors identified by client IP address. |
| Status Code | \$1 elb\$1status\$1code | Displays the count of requests made to the Application Load Balancer, grouped by HTTP status codes (for example, 200, 404, 403). |
| Status History | \$1 elb\$1status\$1code | Shows the historical trend of HTTP status codes returned by the Application Load Balancer over a specific period of time. |
| Status Code Pipe | \$1 elb\$1status\$1code | Represents the distribution of requests based on different HTTP status codes using a pie chart. |
| Average Processing Time | \$1 request\$1processing\$1time \$1 response\$1processing\$1time \$1 target\$1processing\$1time | This visualization calculates and presents the average time taken for various operations in the Application Load Balancer. |
| Avg. Processing Time History | \$1 request\$1processing\$1time \$1 response\$1processing\$1time \$1 target\$1processing\$1time | Displays the historical trend of the average time-consuming of each operation returned by the Application Load Balancer within a specific period of time. |
| Request Verb | \$1 request\$1verb | Displays the count of requests made to the Application Load Balancer using a pie chart, grouped by HTTP request method names (for example, POST, GET, HEAD). |
| Total Bytes | \$1 received\$1bytes \$1 sent\$1bytes | Provides insights into data transfer activities, including the total bytes transferred. |
| Sent and Received Bytes History | \$1 received\$1bytes \$1 sent\$1bytes | Displays the historical trend of the received bytes, send bytes |
| SSL Protocol | \$1 ssl\$1protocol | Displays the count of requests made to the Application Load Balancer, grouped by SSL Protocol |
| Top Request URLs | \$1 request\$1url | The web requests view enables you to analyze the top web requests. |
| Top Client IPs | \$1 client\$1ip | Provides the top 10 IP address accessing your Application Load Balancer. |
| Top User Agents | \$1 user\$1agent | Provides the top 10 user agents accessing your Application Load Balancer. |
| Target Status | \$1 target\$1ip \$1 target\$1status\$1code | Displays the HTTP status code request count for targets in the Application Load Balancer target group. |
| Abnormal Requests | \$1 @timestamp \$1 client\$1ip \$1 target\$1ip \$1 elb\$1status\$1code \$1 error\$1reason \$1 request\$1verb \$1 target\$1status\$1code \$1 target\$1status\$1code\$1list \$1 request\$1url \$1 request\$1proto \$1 trace\$1id | Provides a detailed list of log events, including timestamps, client ip, and target ip. |
| Requests by OS | \$1 ua\$1os | Displays the count of requests made to the Application Load Balancer, grouped by user agent OS |
| Request by Device | \$1 ua\$1device | Displays the count of requests made to the Application Load Balancer, grouped by user agent device. |
| Request by Browser | \$1 ua\$1browser | Displays the count of requests made to the Application Load Balancer, grouped by user agent browser. |
| Request by Category | \$1 ua\$1category | Displays the count of categories made to the Application Load Balancer, grouped by user agent category (for example, PC, Mobile, Tablet). |
| Requests by Countries or Regions | \$1 geo\$1iso\$1code | Displays the count of requests made to the Application Load Balancer (grouped by the corresponding country or Region resolved by the client IP). |
| Top Countries or Regions | \$1 geo\$1country | Top 10 countries with the Application Load Balancer Access. |
| Top Cities | \$1 geo\$1city | Top 10 cities with Application Load Balancer Access |
You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the [Access Dashboard](getting-started.md#step-4-access-the-dashboard).
**Application Load Balancer logs sample dashboard.**
![\[image40\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image40.png)
## Create log ingestion (Light Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the AWS Services section, choose **Elastic Load Balancer**.
1. Choose **Next**.
1. Under **Specify settings**, choose **Automatic** or **Manual** for **CloudFront logs enabling**. The automatic mode will detect the CloudFront log location automatically.
+ For **Automatic** mode, choose an Application Load Balancer in the dropdown list. (If the selected Application Load Balancer access log is not enabled, choose **Enable** to enable the Application Load Balancer access log.)
+ For Manual mode, enter the Application Load Balancer identifier and Log location.
+ (Optional) If you are ingesting CloudFront logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown list first.
1. Choose **Next**.
1. Choose **Log Processing Enriched fields** if needed. The available plugins are **location** and **OS/User Agent**. Enabling rich fields increases data processing latency and processing costs. By default, it is not selected.
1. In the **Specify Light Engine Configuration** section, if you want to ingest associated templated Grafana dashboards, select **Yes** for the sample dashboard.
1. You can choose an existing Grafana, or if you must import a new one, you can go to Grafana for configuration.
1. Select an S3 bucket to store partitioned logs and define a name for the log table. We have provided a predefined table name, but you can modify it according to your business needs.
1. If needed, change the log processing frequency, which is set to **5** minutes by default, with a minimum processing frequency of **1** minute.
1. In the **Log Lifecycle** section, enter the log merge time and log archive time. We have provided default values, but you can adjust them based on your business requirements.
1. Select **Next**.
1. If desired, add tags.
1. Select **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - ELB Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesAlbPipeline.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesAlbPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesAlbPipeline.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesAlbPipeline.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesAlbPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesAlbPipeline.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
1. Parameters for **Pipeline settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/application-load-balancer-application-load-balancer-logs.html)
1. Parameters for **Destination settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/application-load-balancer-application-load-balancer-logs.html)
1. Parameters for **Scheduler settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/application-load-balancer-application-load-balancer-logs.html)
1. Parameters for **Notification settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/application-load-balancer-application-load-balancer-logs.html)
1. Parameters for **Dashboard settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/application-load-balancer-application-load-balancer-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Filters | Filters | The following data can be filtered by query filter conditions. |
| Total Requests | log event | Displays aggregated events based on a specified time interval. |
| Unique Visitors | client\$1ip | Displays unique visitors identified by client IP address. |
| Requests History | log event | Presents a bar chart that displays the distribution of events over time. |
| Request By Target | log event target\$1ip | Presents a bar chart that displays the distribution of events over time and IP. |
| HTTP Status Code | elb\$1status\$1code | Displays the count of requests made to the Application Load Balancer, grouped by HTTP status codes (for example, 200, 404, 403). |
| Status Code History | elb\$1status\$1code | Shows the historical trend of HTTP status codes returned by the Application Load Balancer over a specific period of time. |
| Status Code Pie | elb\$1status\$1code | Represents the distribution of requests based on different HTTP status codes using a pie chart. |
| Average Processing Time | request\$1processing\$1time response\$1processing\$1time target\$1processing\$1time | This visualization calculates and presents the average time taken for various operations in the Application Load Balancer. |
| Avg. Processing Time History | request\$1processing\$1time response\$1processing\$1time target\$1processing\$1time | Displays the historical trend of the average time-consuming of each operation returned by the Application Load Balancer within a specific period of time. |
| HTTP Method | request\$1verb | Displays the count of requests made to the Application Load Balancer using a pie chart, grouped by HTTP request method names (for example, POST, GET, HEAD). |
| Total Bytes | received\$1bytes sent\$1bytes | Provides insights into data transfer activities, including the total bytes transferred. |
| Sent and Received Bytes History | received\$1bytes sent\$1bytes | Displays the historical trend of the received bytes, send bytes. |
| SSL Protocol | ssl\$1protocol | Displays the count of requests made to the Application Load Balancer, grouped by SSL Protocol. |
| Top Request URLs | request\$1url | The web requests view enables you to analyze the top web requests. |
| Top Client IPs | client\$1ip | Provides the top 10 IP addresses accessing your Application Load Balancer. |
| Bad Requests | type client\$1ip target\$1group\$1arn target\$1ip elb\$1status\$1code request\$1verb request\$1url ssl\$1protocol received\$1bytes sent\$1bytes | Provides a detailed list of log events, including timestamps, client IP, and target IP. |
| Requests by OS | ua\$1os | Displays the count of requests made to the Application Load Balancer, grouped by user agent OS. |
| Requests by Device | ua\$1device | Displays the count of requests made to the Application Load Balancer, grouped by user agent device. |
| Requests by Browser | ua\$1browser | Displays the count of requests made to the Application Load Balancer, grouped by user agent browser. |
| Requests by Category | ua\$1category | Displays the count of categories made to the Application Load Balancer, grouped by user agent category (for example, PC, Mobile, Tablet). |
| Requests by Countries or Regions | geo\$1iso\$1code | Displays the count of requests made to the Application Load Balancer (grouped by the corresponding country or Region resolved by the client IP). |
| Top Countries or Regions | geo\$1country | Top 10 countries with the Application Load Balancer Access. |
**Application Load Balancer logs sample dashboard.**
![\[image41\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image41.png)
# AWS WAF logs
[AWS WAF Access Logs](https://docs.aws.amazon.com/waf/latest/developerguide/logging.html) provide detailed information about traffic that is analyzed by your web ACL. Logged information includes the time that AWS WAF received a web request from your AWS resource, detailed information about the request, and details about the rules that the request matched.
You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.
**Important**
Deploy Centralized Logging with OpenSearch solution in the same Region as your Web ACLs, or you will not be able to create a AWS WAF pipeline. For example:
If your Web ACL is associated with Global CloudFront, you must deploy the solution in us-east-1.
If your Web ACL is associated with other resources in Regions like Ohio, your Centralized Logging with OpenSearch stack must also be deployed in that Region.
The AWS WAF logging bucket must be the same as the Centralized Logging with OpenSearch solution.
[AWS WAF Classic](https://docs.aws.amazon.com/waf/latest/developerguide/classic-waf-chapter.html) logs are not supported in Centralized Logging with OpenSearch. Learn more about [migrating rules from AWS WAF Classic to the new AWS WAF](https://aws.amazon.com/blogs/security/migrating-rules-from-aws-waf-classic-to-new-aws-waf/).
The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.
## Create log ingestion (OpenSearch Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the **AWS Services** section, choose **AWS WAF**.
1. Choose **Next**.
1. Under Specify settings, choose Automatic or Manual.
+ For **Automatic** mode, choose a Web ACL in the dropdown list.
+ For **Manual** mode, enter the **Web ACL name**.
+ (Optional) If you are ingesting AWS WAF logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown list first.
1. Specify an Ingest Options. Choose between Sampled Request or Full Request.
+ For **Sampled Request**, enter how often you want to ingest sample requests in minutes.
+ For **Full Request**, if the Web ACL log is not enabled, choose **Enable** to enable the access log, or enter **Log location** in Manual mode. Note that Centralized Logging with OpenSearch will automatically enable logging with a Firehose stream as destination for your AWS WAF.
1. Choose **Next**.
1. In the Specify OpenSearch domain section, select an imported domain for the Amazon OpenSearch Service domain.
1. Choose **Yes** for **Sample dashboard** if you want to ingest an associated templated Amazon OpenSearch Service dashboard.
1. You can change the **Index Prefix** of the target Amazon OpenSearch Service index if needed. The default prefix is the Web ACL Name.
1. In the **Log Lifecycle** section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. In the **Select log processor** section, choose the log processor.
1. When selecting Lambda as a log processor, you can configure the Lambda concurrency if needed.
1. (Optional) OSI as log processor is now supported in these [Regions](https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-opensearch-service-ingestion/). When OSI is selected, type in the minimum and maximum number of OCU. See more information [here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html#ingestion-scaling).
1. Choose **Next**.
1. Add tags if needed.
1. Choose **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - AWS WAF Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions (Full Request) | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFLog.template) |
| AWS China Regions (Full Request) | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFLog.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFLog.template) |
| AWS Regions (Sampled Request) | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFSampledLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFSampledLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFSampledLog.template) |
| AWS China Regions (Sampled Request) | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFSampledLog.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFSampledLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/WAFSampledLog.template) |
1. Log in to the AWS Management Console and select the button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
+ Parameters for **Full Request** only
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-waf-logs.html)
+ Parameters for **Sampled Request** only
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-waf-logs.html)
+ Common parameters
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-waf-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Filters | \$1 Filters | The following data can be filtered by query filter conditions. |
| Web ACLs | \$1 log event \$1 webaclName | Displays the count of requests made to the AWS WAF, grouped by Web ACL Names. |
| Total Requests | \$1 log event | Displays the total number of web requests. |
| Request Timeline | \$1 log event | Presents a bar chart that displays the distribution of events over time. |
| AWS WAF Rules | \$1 terminatingRuleId | Presents a pie chart that displays the distribution of events over the AWS WAF rules in the Web ACL. |
| Total Blocked Requests | \$1 log event | Displays the total number of blocked web requests. |
| Unique Client IPs | \$1 Request.ClientIP | Displays unique visitors identified by client IP. |
| Country or Region By Request | \$1 Request.Country | Displays the count of requests made to the Web ACL (grouped by the corresponding country or Region resolved by the client IP). |
| Http Methods | \$1 Request.HTTPMethod | Displays the count of requests made to the Web ACL using a pie chart, grouped by HTTP request method names (for example, POST, GET, HEAD). |
| Http Versions | \$1 Request.HTTPVersion | Displays the count of requests made to the Web ACL using a pie chart, grouped by HTTP protocol version (for example, HTTP/2.0, HTTP/1.1). |
| Top WebACLs | \$1 webaclName \$1 webaclId.keyword | The web requests view enables you to analyze the top web requests. |
| Top Hosts | \$1 host | Lists the source IP addresses associated with events, enabling you to identify and investigate potentially suspicious or unauthorized activities. |
| Top Request URIs | \$1 Request.URI | Top 10 request URIs. |
| Top Countries or Regions | \$1 Request.country | Top 10 countries with the Web ACL Access. |
| Top Rules | \$1 terminatingRuleId | Top 10 rules in the web ACL that matched the request. |
| Top Client IPs | \$1 Request.ClientIP | Provides the top 10 IP address. |
| Top User Agents | \$1 userAgent | Provides the top 10 user agents |
| Block Allow Host Uri | \$1 host \$1 Request.URI \$1 action | Provides blocked or allowed web requests. |
| Top Labels with Host, Uri | \$1 labels.name \$1 host \$1 Request.URI | Top 10 detailed logs by labels with host, URI |
| View by Matching Rule | \$1 sc-status | This visualization provides detailed logs by DQL "terminatingRuleId:\$1". |
| View by httpRequest args,uri,path | \$1 sc-status | This visualization provides detailed logs by DQL. |
You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the [Access Dashboard](getting-started.md#step-4-access-the-dashboard).
**AWS WAF logs sample dashboard.**
![\[image42\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image42.png)
## Create log ingestion (Light Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the **AWS Services** section, choose **AWS WAF**.
1. Choose **Light Engine**, choose **Next**.
1. Under Specify settings, choose Automatic or Manual.
+ For **Automatic** mode, choose a Web ACL from the dropdown list.
+ For **Manual** mode, enter the Web ACL name.
+ (Optional) If you are ingesting CloudFront logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown list first.
1. Choose **Next**.
1. Choose **Log Processing Enriched fields** if needed. The available plugins are **location** and **OS/User Agent**. Enabling rich fields increases data processing latency and processing costs. By default, it is not selected.
1. In the **Specify Light Engine Configuration** section, if you want to ingest associated templated Grafana dashboards, select **Yes** for the sample dashboard.
1. You can choose an existing Grafana, or if you must import a new one, you can go to Grafana for configuration.
1. Select an S3 bucket to store partitioned logs and define a name for the log table. We have provided a predefined table name, but you can modify it according to your business needs.
1. If needed, change the log processing frequency, which is set to **5** minutes by default, with a minimum processing frequency of **1** minute.
1. In the **Log Lifecycle** section, enter the log merge time and log archive time. We have provided default values, but you can adjust them based on your business requirements.
1. Select **Next**.
1. If desired, add tags.
1. Select **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - AWS WAF Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesWafPipeline.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesWafPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesWafPipeline.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesWafPipeline.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesWafPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesWafPipeline.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
1. Parameters for **Pipeline settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-waf-logs.html)
1. Parameters for **Destination settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-waf-logs.html)
1. Parameters for **Scheduler settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-waf-logs.html)
1. Parameters for **Notification settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-waf-logs.html)
1. Parameters for **Dashboard settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-waf-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates IAM resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Filters | Filters | The following data can be filtered by query filter conditions. |
| Total Requests | log event | Displays the total number of web requests. |
| Total Blocked Requests | log event | Displays the total number of blocked web requests. |
| Requests History | log event | Presents a bar chart that displays the distribution of events over time. |
| AWS WAF ACLs | log event webaclName | Displays the count of requests made to the AWS WAF, grouped by Web ACL Names. |
| AWS WAF Rules | terminatingRuleId | Presents a pie chart that displays the distribution of events over the AWS WAF rules in the Web ACL. |
| Sources | httpSourceId | Presents a pie chart that displays the distribution of events over the id of the associated resource. |
| HTTP Methods | httpRequest.HTTPMethod | Displays the count of requests made to the Web ACL using a pie chart, grouped by HTTP request method names (for example, POST, GET, HEAD). |
| Country or Region By Blocked Requests | HTTPRequest.Country | Displays the count of blocked web requests made to the Web ACL (grouped by the corresponding country or Region resolved by the client IP). |
| Top WebACLs | webaclName | The web requests view enables you to analyze the top web requests. |
| Top Sources | httpSourceId | Top 10 id of the associated resource. |
| Top Requests URIs | httpRequest.URI | Top 10 request URIs. |
| Top Countries or Regions | httpRequest.country | Top 10 countries with the Web ACL Access. |
| Top Rules | terminatingRuleId | Top 10 rules in the web ACL that matched the request. |
| Top Client IPs | httpRequest.ClientIP | Provides the top 10 IP addresses. |
| Top Blocked / Allowed Hosts URI | host httpRequest.URI action | Provides blocked or allowed web requests. |
| Top Labels with Host, URI | labels host httpRequest.URI | Top 10 detailed logs by labels with host, URI. |
| Metrics | webaclId webaclName terminatingRuleId terminatingRuleType httpSourceId httpRequest.HTTPMethod httpRequest.country httpRequest.ClientIP labels httpRequest.URI action | Provides a detailed list of log events, including timestamps, web ACL, and client IP. |
**AWS WAF logs sample dashboard.**
![\[image43\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image43.jpeg)
# VPC Flow Logs
[VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) enable you to capture information about the IP traffic going to and from network interfaces in your VPC.
You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.
**Important**
Centralized Logging with OpenSearch supports VPCs who publish the flow log data to an Amazon S3 bucket or a CloudWatch log group. When publishing to Amazon S3, the S3 bucket Region must be the same as the Centralized Logging with OpenSearch solution Region.
The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.
## Create log ingestion (OpenSearch Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the **AWS Services** section, choose **VPC Flow Logs**.
1. Choose **Next**.
1. Under **Specify settings**, choose **Automatic** or **Manual** for **VPC Flow Log enabling**. The automatic mode will enable the VPC Flow Log and save the logs to a centralized S3 bucket if logging is not enabled yet.
+ For **Automatic mode**, choose the VPC from the dropdown list.
+ For Manual mode, enter the VPC Name and VPC Flow Logs location.
+ (Optional) If you are ingesting VPC Flow Logs from another account, select a [linked account](cross-account-ingestion.md) from the **Account** dropdown list first.
1. Under **Log Source**, select **S3** or **CloudWatch** as the source.
1. Choose **Next**.
1. In the Specify OpenSearch domain section, select an imported domain for the Amazon OpenSearch Service domain.
1. Choose **Yes** for **Sample dashboard** if you want to ingest an associated built-in Amazon OpenSearch Service dashboard.
1. You can change the **Index Prefix** of the target Amazon OpenSearch Service index if needed. The default prefix is your VPC name.
1. In the **Log Lifecycle** section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. In the **Select log processor** section, choose the log processor.
1. When selecting Lambda as a log processor, you can configure the Lambda concurrency if needed.
1. (Optional) OSI as log processor is now supported in these [Regions](https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-opensearch-service-ingestion/). When OSI is selected, please type in the minimum and maximum number of OCU. See more information [here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html#ingestion-scaling).
1. Choose **Next**.
1. Add tags if needed.
1. Choose **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - VPC Flow Logs Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Standard Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/VPCFlowLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/VPCFlowLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/VPCFlowLog.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/VPCFlowLog.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/VPCFlowLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/VPCFlowLog.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/vpc-flow-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Global Filters | \$1 account-id \$1 Region \$1 vpc-id \$1 subnet-id \$1 action \$1 flow-direction \$1 log-status \$1 protocol-code \$1 type | The charts are filtered according to Account ID, Region, VPC ID, and other conditions. |
| Total Requests | \$1 log event | Shows the total number of network requests logged by VPC Flow Logs during a selected time period. |
| Request History | \$1 log event | Presents a bar chart that displays the distribution of events over time. |
| Requests By VPC ID | \$1 vpc-id | Displays the proportional breakdown of network requests by source VPC using a pie chart. |
| Total Requests By Action | \$1 action | Displays the total volume of requests segmented by action over time. |
| Total Bytes | \$1 bytes | Provides visibility into overall bandwidth usage and traffic patterns across the monitored VPCs, subnets, network interfaces, and security groups. |
| Total Packets | \$1 packets | Displays total logged packets over time to visualize trends, surges, and dips. |
| Bytes Metric | \$1 bytes \$1 flow-direction | Shows the distribution of incoming (Ingress) and outgoing (Egress) network traffic volumes in bytes across the range of flows logged by VPC Flow Logs over a time period. |
| Requests By Direction | \$1 flow-direction | Provides visibility into the proportional composition of incoming versus outgoing requests. |
| Requests By Direction | \$1 flow-direction | Displays the total number of network flows logged by VPC Flow Logs segmented by traffic direction - Ingress vs Egress. |
| Requests By Type | \$1 type | Shows the volume of flows for each type. This provides visibility into the protocol composition of network requests traversing the environment. |
| Top Source Bytes | \$1 srcaddr \$1 bytes | Displays the source IP addresses transmitting the highest outbound volume of data during the selected time period. |
| Top Destination Bytes | \$1 dstaddr \$1 bytes | Enables you to monitor and analyze outbound traffic from your VPC to external destinations. |
| Top Source Requests | \$1 srcaddr | Allows you to see which resources inside your VPC are initiating external requests. |
| Top Destination Requests | \$1 dstaddr | Allows you to see which external hosts are being contacted most by your VPC resources. |
| Requests by Protocol | \$1 protocol-code | Displays network flows logged by VPC Flow Logs segmented by traffic type - TCP, UDP, ICMP. |
| Requests by Status | \$1 log-status | Provides a breakdown of network flows by their traffic status - Accepted, Rejected, or Other. |
| Top Sources AWS Services | \$1 pkt-src-aws-service | Show the proportional distribution of flows originating from top AWS sources like Amazon S3, CloudFront, Lambda, etc. during the selected time period. |
| Top Destination AWS Services | \$1 pkt-dst-aws-service | Provide visibility into IP traffic going to and from AWS services located outside your VPC. By enabling flow logs on VPC subnets/interfaces and filtering on traffic with an ACCEPT action, you can view outbound flows from your VPC to various AWS services. |
| Network Flow | \$1 srcaddr \$1 dstaddr | Allows you to view information about the IP traffic going to and from network interfaces in your VPC. |
| Heat Map | \$1 srcaddr \$1 dstaddr | Offers a visual summary of connections between source and destination IPs in your flow log data. |
| Egress Traffic Path | \$1 traffic-path | Allows you to enable flow logging on VPC network interfaces to capture information about all IP traffic going to and from that interface. |
| Search | \$1 @timestamp \$1 account-id \$1 vpc-id \$1 flow-direction \$1 action \$1 protocol-code \$1 srcaddr \$1 scaport \$1 dstaddr \$1 dstport \$1 bytes \$1 packets \$1 log-status | Searching through the detailed flow log data allows pinpoint analysis of traffic around security events, network issues, changes in usage patterns, and more. |
You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the [Access Dashboard](getting-started.md#step-4-access-the-dashboard).
**VPC Flow Logs sample dashboard.**
![\[image44\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image44.png)
## Create log ingestion (Light Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the **AWS Services** section, choose **AWS VPC Flow**.
1. Choose **Light Engine**, choose **Next**.
1. Under **Specify settings**, choose **Automatic** or **Manual** for **CloudFront logs enabling**. The automatic mode will detect the CloudFront log location automatically.
+ For **Automatic** mode, choose the VPC Flow from the dropdown list.
+ For Standard Log, the solution will automatically detect the log location if logging is enabled.
+ For Manual mode, enter the VPC Flow ID and VPC Flow Log location.
+ (Optional) If you are ingesting CloudFront logs from another account, select a [linked account](cross-account-ingestion.md#add-a-member-account) from the **Account** dropdown list first.
1. Choose **Next**.
1. In the **Specify Light Engine Configuration** section, if you want to ingest associated templated Grafana dashboards, select **Yes** for the sample dashboard.
1. You can choose an existing Grafana, or if you must import a new one, you can go to Grafana for configuration.
1. Select an S3 bucket to store partitioned logs and define a name for the log table. We have provided a predefined table name, but you can modify it according to your business needs.
1. If needed, change the log processing frequency, which is set to **5** minutes by default, with a minimum processing frequency of **1** minute.
1. In the **Log Lifecycle** section, enter the log merge time and log archive time. We have provided default values, but you can adjust them based on your business requirements.
1. Select **Next**.
1. If desired, add tags.
1. Select **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - VpcFlow Standard Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesVpcFlowPipeline.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesVpcFlowPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesVpcFlowPipeline.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesVpcFlowPipeline.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesVpcFlowPipeline.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/MicroBatchAwsServicesVpcFlowPipeline.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
1. Parameters for **Pipeline settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/vpc-flow-logs.html)
1. Parameters for **Destination settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/vpc-flow-logs.html)
1. Parameters for **Scheduler settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/vpc-flow-logs.html)
1. Parameters for **Notification settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/vpc-flow-logs.html)
1. Parameters for **Dashboard settings**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/vpc-flow-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
**VPC Flow Logs sample dashboard.**
![\[image45\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image45.png)
# AWS Config logs
By default, AWS Config delivers configuration history and snapshot files to your Amazon S3 bucket.
You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.
**Important**
AWS Config must be enabled in the same Region as the Centralized Logging with OpenSearch solution.
The Amazon S3 bucket Region must be the same as the Centralized Logging with OpenSearch solution.
The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.
## Create log ingestion (OpenSearch Engine)
### Using the Centralized Logging with OpenSearch Console
1. Sign in to the Centralized Logging with OpenSearch Console.
1. In the navigation pane, under **Log Analytics Pipelines**, choose **Service Log**.
1. Choose the Create a log ingestion button.
1. In the AWS Services section, choose AWS Config Logs.
1. Choose **Next**.
1. Under Specify settings, choose Automatic or Manual for Log creation.
+ For **Automatic mode**, make sure that the S3 bucket location is correct, and enter the **AWS Config Name**.
+ For Manual mode, enter the AWS Config Name and Log location.
+ (Optional) If you are ingesting VPC Flow Logs from another account, select a [linked account](cross-account-ingestion.md) from the **Account** dropdown list first.
1. Choose **Next**.
1. In the Specify OpenSearch domain section, select an imported domain for the Amazon OpenSearch Service domain.
1. Choose **Yes** for **Sample dashboard** if you want to ingest an associated built-in Amazon OpenSearch Service dashboard.
1. You can change the **Index Prefix** of the target Amazon OpenSearch Service index if needed. The default prefix is your VPC name.
1. In the **Log Lifecycle** section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated [Index State Management (ISM)](https://opensearch.org/docs/latest/im-plugin/ism/index/) policy automatically for this pipeline.
1. Choose **Next**.
1. Add tags if needed.
1. Choose **Create**.
### Using the CloudFormation Stack
This automated AWS CloudFormation template deploys the *Centralized Logging with OpenSearch - AWS Config Log Ingestion* solution in the AWS Cloud.
| | Launch in AWS Management Console | Download Template |
| --- | --- | --- |
| AWS Standard Regions | [https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ConfigLog.template](https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ConfigLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ConfigLog.template) |
| AWS China Regions | [https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ConfigLog.template](https://console.amazonaws.cn/cloudformation/home#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ConfigLog.template) | [Template](https://solutions-reference.s3.amazonaws.com/centralized-logging-with-opensearch/latest/ConfigLog.template) |
1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.
1. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.
1. On the **Create stack** page, verify that the correct template URL shows in the **Amazon S3 URL** text box and choose **Next**.
1. On the **Specify stack details** page, assign a name to your solution stack.
1. Under **Parameters**, review the parameters for the template and modify them as necessary. This solution uses the following parameters.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-config-logs.html)
1. Choose **Next**.
1. On the **Configure stack options** page, choose **Next**.
1. On the **Review and create** page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.
1. Choose **Submit** to deploy the stack.
You can view the status of the stack in the AWS CloudFormation console in the **Status** column. You should receive a **CREATE\$1COMPLETE** status in approximately 10 minutes.
### View dashboard
The dashboard includes the following visualizations.
| Visualization Name | Source Field | Description |
| --- | --- | --- |
| Global Filters | awsAccountId awsRegion resourceType resourceId resourceName | The charts are filtered according to Account ID, Region, Resource Type, and other conditions. |
| Total Change Events | log event | Shows the number of configuration changes detected across all AWS resources during a selected time period. |
| Top Resource Types | resourceType | Displays the breakdown of configuration changes by the most frequently modified AWS resource types during a selected time period. |
| Config History | log event | Presents a bar chart that displays the distribution of events over time. |
| Total Delete Events | log event | Shows the number of AWS resource deletion events detected by AWS Config during a selected time period. |
| Config Status | configurationItemStatus | Displays the operational state of the AWS Config service across monitored Regions and accounts. |
| Top S3 Changes | resourceName | Displays the Amazon S3 buckets undergoing the highest number of configuration changes during a selected time period. |
| Top Changed Resources | resourceName resourceId resourceType | Displays the individual AWS resources undergoing the highest number of configuration changes during a selected time period. |
| Top VPC Changes | resourceId | Presents a bar chart that Displays the Amazon VPCs undergoing the highest number of configuration changes during a selected time period. |
| Top Subnet Changes | resourceId | Delivers targeted visibility into the subnets undergoing the most transformation for governance, security, and stability. |
| Top Network Interface Changes | resourceId | Spotlights the Amazon VPC network interfaces seeing the most configuration changes during a selected period. |
| Top Security Group Changes | resourceId | Top 10 changed groups rank by total modification count. |
| EC2 Config | @timestamp awsAccountId awsRegion resourceId configurationItemStatus | Allows reconstructing the incremental changes applied to EC2 configurations over time for auditing. |
| RDS Config | @timestamp awsAccountId awsRegion resourceId resourceName configurationItemStatus | Shows the configuration history and changes detected by AWS Config for RDS database resources |
| Latest Config Changes | @timestamp awsAccountId awsRegion resourceType resourceId resourceName relationships configurationItemStatus | Offers an at-a-glance overview of infrastructure modifications. |
You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the [Access Dashboard](getting-started.md#step-4-access-the-dashboard).
**AWS Config logs sample dashboard.**
![\[image46\]](http://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/images/image46.png)