# Spans Spans sent to X-Ray are ingested and managed in a log group called `aws/spans`. This topic describes which CloudWatch Logs features are available for transaction spans. **Available features** The following CloudWatch Logs features are available for transaction spans. + [Metric filters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringLogData.html) – Use metric filters to extract custom metrics from spans. + [Subscriptions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html) – Use subscriptions to access a real-time feed of span events from CloudWatch Logs. + [Log anomaly detection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/LogsAnomalyDetection.html) – Use log anomaly detection to establish a baseline for spans sent to the `aws/spans` log group. + [Contributor Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContributorInsights.html) – Use Contributor Insights to analyze span data and create a time series displaying contributor data. **Unsupported features** The following are features not supported for transaction spans. + Spans cannot be sent to CloudWatch Logs with the `PutLogEvents` API. + Span data cannot be [enriched or transformed](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch-Logs-Transformation.html). **Note** Span ingestion is charged separately from log ingestion. For information about pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). # Searching and analyzing spans Transaction Search provides you with a visual editor to search and analyze all ingested spans using attributes. You can use the visual editor to narrow down transaction spans and create interactive visualizations to troubleshoot issues in your distributed applications. You can also use the CloudWatch Logs Insights query language to analyze your spans. This topic describes how to access and use the visual editor. ## The visual editor The following procedure describes how to access the visual editor. **To access the visual editor** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. From the navigation pane, choose **Application Signals**, and then choose **Transaction Search**. Use span attributes, such as service name, span duration, and span status to narrow down transaction spans quickly. You can access these filters and more on the right side of the visual editor under **Select filters**. This visual editor suggests a list of attributes in the span. These attributes include attributes added through auto-instrumentation and custom attributes added through custom instrumentation. ![\[Filter spans by attributes\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/filter3.png) Select a span key, and enter a value to refine span results. You can filter spans using various operations, such as "Equals," "Does Not Equal," and more. ![\[Filter spans with operators\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/filter4.png) ### Query formats You can run queries in the visual editor using different formats. This section describes each of these formats. #### List View spans or span events in a list format, which displays information about each span. Use this type of analysis to analyze individual spans, understand specific transactions, or identify unique patterns in transaction events. Other use cases include the following: **Use cases** + Troubleshoot customer support tickets + Locate APIs or dependencies, such as database queries taking longer than 1000 milliseconds to execute + Locate spans with errors The following screenshots show how to troubleshoot a customer support ticket with this type of analysis. **Example scenario** In the visual editor, filter on all transaction spans with a particular customer issue. Before you run your query, choose **List** from the **Visualize as** dropdown. ![\[Locate spans with List\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/list1.png) The results show a list of spans where you can choose a trace ID to get the end-to-end journey for the transaction and determine the root cause of the issue. ![\[List results\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/list2.png) #### Timeseries View spans or span events over time. Use this type of analysis to look at trends and spikes in transaction activity. Other use cases include the following: + Visualize latency + Visualize frequency of spans + Visualize performance The following screenshots show how you can view p99 latency trends for an API with this type of analysis. **Example scenario** In the visual editor, filter on the service and API you want to analyze. ![\[Filtering on a service\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/timeseries1.png) Before you run your query, choose **Time series** from the **Visualize as** dropdown. Choose **P99** for the duration statistic from the **Show span as** dropdown. ![\[Filtering\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/timeseries2.png) The results show a latency trend for the service, with the x-axis of the graph being time and y-axis being p99 duration. ![\[Locate spans with time series\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/timeseries3.png) You can choose a point on the chart to view correlated spans and span events. ![\[Time series results\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/timeseries4.png) #### Group analysis Aggregate spans or span events based on specific attributes, such as account IDs and status codes, to display statistical metrics. Use this type of analysis to analyze spans in clusters, compare different groups, and uncover trends at the macro level. Other use cases include the following: **Use cases** + Identify top customers impacted by a service outage + Identify availability zones with the most errors + Identify the top slowest database queries The following screenshots show how you can view the top customers impacted by a service outage with this type of analysis. **Example scenario** In the visual editor, you filter on the service experiencing issues. ![\[Filter by service issue\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/groupanalysis1.png) Before you run your query, choose **Group Analysis** from the **Visualize as** dropdown. Group your query results by `account.id`, and limit the number of results to 10.. ![\[Locate spans by group analysis\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/groupanalysis2.png) The results show the top 10 customers who experienced the most number of errors. ![\[Group analysis results\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/groupanalysis3.png) ## CloudWatch Logs Insights You can use [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html) to analyze your spans. **Example query** The following query shows the top five slowest database queries. ``` STATS pct(durationNano, 99) as `p99` by attributes.db.statement | SORT p99 ASC | LIMIT 5 | DISPLAY p99,attributes.db.statement ``` **Example query** The following query shows which top five services are throwing errors. ``` FILTER `attributes.http.response.status_code` >= 500 | STATS count(*) as `count` by attributes.aws.local.service as service | SORT count ASC | LIMIT 5 | DISPLAY count,service ``` # Ingesting spans for complete visibility Recording all transaction spans provides comprehensive visibility into application issues. It enables you to debug customer support tickets or troubleshoot rarely occurring p99 API latency spikes, which is crucial when identifying the root cause of issues in customer-facing and mission-critical applications. You can create a cost-effective strategy to start capturing 100% of trace spans in CloudWatch by configuring the head sampling rate and then adjusting a lower span indexing rate. ## Setting up head sampling Head sampling is a tracing technique capturing requests at the beginning of a trace, which is based on a set rate or condition. When the head sampling rate is set to 100%, it captures the beginning of every trace without skips, guaranteeing complete visibility into all incoming requests, and that no transaction data is missed. You can configure head sampling if you're using X-Ray or AWS Distro for OpenTelemetry SDKs or the OpenTelemetry SDK. **If you're using X-Ray or AWS Distro for OpenTelemetry SDKs** Navigate to your sampling rules in the console, and set the fixed sampling rate to 100%. This guarantees all trace spans are captured and ingested into CloudWatch logs. For more information, see [Configuring sampling rules](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-sampling.html#xray-console-config) **If you're using the OpenTelemetry SDK** To record 100% of spans and get complete visibility, set your sampling configuration to [https://opentelemetry.io/docs/languages/java/sdk/#sampler](https://opentelemetry.io/docs/languages/java/sdk/#sampler). For more information, see [Language APIs & SDKs]() on the OpenTelemetry website. ### Features unlocked with head sampling When you enable Transaction Search, all spans collected from your application through head sampling are ingested as structured logs in CloudWatch. This provides you with the following features: + The ability to search span attributes and analyze span events in a visual editor. + The ability to visualize traces containing up to 10,000 spans. + Total support for OpenTelemetry, which includes the ability to embed business events into spans for analysis and use span links to define connections between traces for end-to-end viewing. + Access to application dashboards, metrics, and topology with CloudWatch Application Signals enabled for all spans sent to CloudWatch. **Note** Because spans are available in a log group called `aws/spans`, you can use CloudWatch Logs features with transaction spans. For more information, see [The span log group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-ingesting-span-log-groups.html). ## Setting up span indexing with trace summaries Trace summaries can help you debug transactions and are beneficial for asynchronous processes. You only need to index a small percentage of spans as trace summaries. You configure span indexing when you enable Transaction Search in the console or with the API. To enable Transaction Search, see [Getting started with Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html). ### Features unlocked with trace summaries The key features of X-Ray trace summaries include the following: + [Trace summary search](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-traces.html?icmpid=docs_xray_help_panel_traces#xray-console-traces-view) – Search and find traces from trace summaries. + [Trace summary analytics](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-analytics.html) – Interpret trace data. + [Trace insights](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-insights.html) – Analyze trace data to identify application issues. # Monitoring spans across accounts Spans sent to X-Ray are ingested and managed in a log group called `aws/spans`. To monitor spans across multiple accounts, you must [enable Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html) across all source and monitoring accounts and [enable cross-account observability](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html) for logs and traces. When you enable cross-account observability, you can search up to 10,000 accounts and get visibility into traces across accounts. This feature is provided at no extra cost for the `aws/spans` log group. If you enable cross-account observability for trace summaries, the first trace summary copy is free. For more information about pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/).