# Observability
<a name="observability"></a>

 Since microservices architectures are inherently made up of many distributed components, observability across all those components becomes critical. Amazon CloudWatch enables this, collecting and tracking metrics, monitoring log files, and reacting to changes in your AWS environment. It can monitor AWS resources and custom metrics generated by your applications and services. 

**Topics**
+ [

# Monitoring
](monitoring.md)
+ [

# Centralizing logs
](centralizing-logs.md)
+ [

# Distributed tracing
](distributed-tracing.md)
+ [

# Log analysis on AWS
](log-analysis-on-aws.md)
+ [

# Other options for analysis
](other-options-for-analysis.md)

# Monitoring
<a name="monitoring"></a>

 CloudWatch offers system-wide visibility into resource utilization, application performance, and operational health. In a microservices architecture, custom metrics monitoring through CloudWatch is beneficial, as developers can choose which metrics to collect. Dynamic scaling can also be based on these custom metrics. 

 CloudWatch Container Insights extends this functionality, automatically collecting metrics for many resources like CPU, memory, disk, and network. It helps in diagnosing container-related issues, streamlining resolution. 

 For Amazon EKS, an often-preferred choice is Prometheus, an open-source platform providing comprehensive monitoring and alerting capabilities. It's typically coupled with Grafana for intuitive metrics visualization. [Amazon Managed Service for Prometheus (AMP)](https://aws.amazon.com/prometheus/) offers a monitoring service fully compatible with Prometheus, letting you oversee containerized applications effortlessly. Additionally, [Amazon Managed Grafana (AMG)](https://aws.amazon.com/grafana/) simplifies the analysis and visualization of your metrics, eliminating the need for managing underlying infrastructure. 

![\[Diagram showing a serverless architecture with monitoring components\]](http://docs.aws.amazon.com/whitepapers/latest/microservices-on-aws/images/serverless-arch-with-monitoring.png)


![\[A container-based architecture with monitoring components\]](http://docs.aws.amazon.com/whitepapers/latest/microservices-on-aws/images/container-arch-with-monitoring.png)


# Centralizing logs
<a name="centralizing-logs"></a>

 Logging is key to pinpoint and resolve issues. With microservices, you can release more frequently and experiment with new features. AWS provides services like Amazon S3, CloudWatch Logs, and Amazon OpenSearch Service to centralize log files. Amazon EC2 uses a daemon for sending logs to CloudWatch, while Lambda and Amazon ECS natively send their log output there. For Amazon EKS, either [Fluent Bit or Fluentd can be used](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html) to forward logs to CloudWatch for reporting using OpenSearch and Kibana. However, due to the smaller footprint and [performance advantages](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html), Fluent Bit is recommended over Fluentd. 

 Figure 12 illustrates how logs from various AWS services are directed to Amazon S3 and CloudWatch. These centralized logs can be further analyzed using Amazon OpenSearch Service, inclusive of Kibana for data visualization. Also, Amazon Athena can be employed for ad hoc queries against the logs stored in Amazon S3. 

![\[Diagram showing logging capabilities of AWS services\]](http://docs.aws.amazon.com/whitepapers/latest/microservices-on-aws/images/logging-capabilities.png)


# Distributed tracing
<a name="distributed-tracing"></a>

 Microservices often work together to handle requests. AWS X-Ray uses correlation IDs to track requests across these services. X-Ray works with Amazon EC2, Amazon ECS, Lambda, and Elastic Beanstalk. 

![\[Diagram showing AWS X-Ray service map\]](http://docs.aws.amazon.com/whitepapers/latest/microservices-on-aws/images/xray-service-map.png)


 [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/?otel-blogs.sort-by=item.additionalFields.createdDate&otel-blogs.sort-order=desc) is part of the OpenTelemetry project and provides open-source APIs and agents to gather distributed traces and metrics, improving your application monitoring. It sends metrics and traces to multiple AWS and partner monitoring solutions. By collecting metadata from your AWS resources, it aligns application performance with the underlying infrastructure data, accelerating problem-solving. Plus, it's compatible with a variety of AWS services and can be used on-premises. 

# Log analysis on AWS
<a name="log-analysis-on-aws"></a>

 Amazon CloudWatch Logs Insights allows for real-time log exploration, analysis, and visualization. For further log file analysis, Amazon OpenSearch Service, which includes Kibana, is a powerful tool. CloudWatch Logs can stream log entries to OpenSearch Service in real time. Kibana, seamlessly integrated with OpenSearch, visualizes this data and offers an intuitive search interface. 

![\[Diagram showing log analysis with Amazon OpenSearch Service\]](http://docs.aws.amazon.com/whitepapers/latest/microservices-on-aws/images/log-analysis-with-opensearch-service.png)


# Other options for analysis
<a name="other-options-for-analysis"></a>

 For further log analysis, Amazon Redshift, a fully-managed data warehouse service, and [Quick](https://aws.amazon.com/quicksight/), a scalable business intelligence service, offer effective solutions. QuickSight provides easy connectivity to various AWS data services such as Redshift, RDS, Aurora, EMR, DynamoDB, Amazon S3, and Kinesis, simplifying data access. 

 CloudWatch Logs can stream log entries to Amazon Data Firehose, a service for delivering real-time streaming data. QuickSight then uses the data stored in Redshift for comprehensive analysis, reporting, and visualization. 

![\[Diagram showing log analysis with Amazon Redshift and Quick\]](http://docs.aws.amazon.com/whitepapers/latest/microservices-on-aws/images/log-analysis-redshift-quicksight.png)


 Moreover, when logs are stored in S3 buckets, an object storage service, the data can be loaded into services like Redshift or EMR, a cloud-based big data platform, allowing for thorough analysis of the stored log data. 

![\[Diagram showing streamlining log analysis: From AWS services to QuickSight\]](http://docs.aws.amazon.com/whitepapers/latest/microservices-on-aws/images/streamlining-log-analysis.png)