# Create a custom log parser for Amazon ECS using a Firelens log router Create a custom log parser for Amazon ECS using Firelens *Varun Sharma, Amazon Web Services* ## Summary Firelens is a log router for Amazon Elastic Container Service (Amazon ECS) and AWS Fargate. You can use Firelens to route container logs from Amazon ECS to Amazon CloudWatch and other destinations (for example, [Splunk](https://www.splunk.com/) or [Sumo Logic](https://www.sumologic.com/)). Firelens works with [Fluentd](https://www.fluentd.org/) or [Fluent Bit](https://fluentbit.io/) as the logging agent, which means that you can use [Amazon ECS task definition parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html) to route logs. By choosing to parse logs at the source level, you can analyze your logging data and perform queries to more efficiently and effectively respond to operational issues. Because different applications have different logging patterns, you need to use a custom parser that structures the logs and makes searching easier at your end destination. This pattern uses a Firelens log router with a custom parser to push logs to CloudWatch from a sample Spring Boot application running on Amazon ECS. You can then use Amazon CloudWatch Logs Insights to filter the logs based on custom fields that are generated by the custom parser. ## Prerequisites and limitations **Prerequisites ** + An active Amazon Web Services (AWS) account. + AWS Command Line Interface (AWS CLI), installed and configured on your local machine. + Docker, installed and configured on your local machine. + An existing Spring Boot-based containerized application on Amazon Elastic Container Registry (Amazon ECR).  ## Architecture ![\[Using a Firelens log router to push logs to CloudWatch from an application running on Amazon ECS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e82b4992-c4e0-4af5-b87e-cb0b1c1ed8c9/images/ef60e087-965a-40e9-9f80-35edbda2befe.png) **Technology stack  ** + CloudWatch + Amazon ECR + Amazon ECS + Fargate + Docker + Fluent Bit ## Tools + [Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) – Amazon Elastic Container Registry (Amazon ECR) is an AWS managed container image registry service that is secure, scalable, and reliable. + [Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) – Amazon Elastic Container Service (Amazon ECS) is a highly scalable, fast container management service that makes it easy to run, stop, and manage containers on a cluster. + [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) – IAM is a web service for securely controlling access to AWS services. + [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) – AWS Command Line Interface (AWS CLI) is an open-source tool that enables you to interact with AWS services using commands in your command-line shell. + [Docker](https://www.docker.com/) – Docker is an open platform for developing, shipping, and running applications. **Code** The following files are attached to this pattern: + `customFluentBit.zip` – Contains the files to add the custom parsing and configurations. + `firelens_policy.json` – Contains the policy document to create an IAM policy. + `Task.json` – Contains a sample task definition for Amazon ECS. ## Epics ### Create a custom Fluent Bit image | Task | Description | Skills required | | --- | --- | --- | | Create an Amazon ECR repository. | Sign in to the AWS Management Console, open the Amazon ECR console, and create a repository called `fluentbit_custom`.For more information about this, see [Creating a repository](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html) in the Amazon ECR documentation. | Systems administrator, Developer | | Unzip the customFluentBit.zip package. |  [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-a-custom-log-parser-for-amazon-ecs-using-a-firelens-log-router.html) | | | Create the custom Docker image. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-a-custom-log-parser-for-amazon-ecs-using-a-firelens-log-router.html)For more information about this, see [Pushing a Docker image](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html) in the Amazon ECR documentation.  | Systems administrator, Developer | ### Set up the Amazon ECS cluster | Task | Description | Skills required | | --- | --- | --- | | Create an Amazon ECS cluster. | Create an Amazon ECS cluster by following the instructions from the *Networking only template* section of [Creating a cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create_cluster.html) in the Amazon ECS documentation.Make sure that you choose **Create VPC** to create a new virtual private cloud (VPC) for your Amazon ECS cluster. | Systems administrator, Developer | ### Set up the Amazon ECS task | Task | Description | Skills required | | --- | --- | --- | | Set up the Amazon ECS task execution IAM role. | Create an Amazon ECS task execution IAM role by using the `AmazonECSTaskExecutionRolePolicy` managed policy. For more information about this, see [Amazon ECS task execution IAM role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html) in the Amazon ECS documentation.Make sure that you record the IAM role’s Amazon Resource Name (ARN). | Systems administrator, Developer | | Attach the IAM policy to the Amazon ECS task execution IAM role. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-a-custom-log-parser-for-amazon-ecs-using-a-firelens-log-router.html) | Systems administrator, Developer | | Set up the Amazon ECS task definition. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-a-custom-log-parser-for-amazon-ecs-using-a-firelens-log-router.html)For more information about this, see [Creating a task definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html) in the Amazon ECS documentation. | Systems administrator, Developer | ### Run the Amazon ECS task | Task | Description | Skills required | | --- | --- | --- | | Run the Amazon ECS task. | On the Amazon ECS console, choose **Clusters**, choose the cluster that you created earlier, and then run the standalone task.For more information about this, see [Run a standalone task](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_run_task.html) in the Amazon ECS documentation. | Systems administrator, Developer | ### Verify the CloudWatch logs | Task | Description | Skills required | | --- | --- | --- | | Verify the logs. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-a-custom-log-parser-for-amazon-ecs-using-a-firelens-log-router.html) | Systems administrator, Developer | ## Related resources + [Docker basics for Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html)  + [Amazon ECS on AWS Fargate](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html)  + [Configuring basic service parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/basic-service-params.html)  ## Attachments To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/e82b4992-c4e0-4af5-b87e-cb0b1c1ed8c9/attachments/attachment.zip)