# Query metrics with PromQL
**Topics**
+ [What is Prometheus Query Language (PromQL)?](#CloudWatch-PromQL-WhatIs)
+ [PromQL limits and restrictions](#CloudWatch-PromQL-Limits)
+ [Supported AWS Regions](#CloudWatch-PromQL-Regions)
+ [IAM permissions for PromQL](#CloudWatch-PromQL-IAM)
+ [PromQL querying](CloudWatch-PromQL-Querying.md)
+ [Running PromQL queries in Query Studio (Preview)](CloudWatch-PromQL-QueryStudio.md)
+ [Using PromQL in alarms](CloudWatch-PromQL-Alarms.md)
**Note**
OTLP metrics ingestion, PromQL querying, OTel enrichment of vended AWS metrics, and Query Studio are in public preview release, are free of charge, and subject to change.
## What is Prometheus Query Language (PromQL)?
Prometheus Query Language (PromQL) is a functional query language that lets you select, aggregate, and transform time series data in real time. PromQL was originally designed for Prometheus and has become a popular query language for metrics.
Amazon CloudWatch supports PromQL for querying metrics including metrics ingested via OpenTelemetry Line Protocol (OTLP) and [AWS enriched vended metrics](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html). When you ingest OTLP metrics, CloudWatch preserves the full semantic structure of your telemetry, including resource attributes, instrumentation scope, datapoint attributes, and AWS-specific metadata, and exposes them as queryable PromQL labels.
With PromQL you can do the following:
+ Select time series by metric name and label matchers.
+ Apply mathematical functions and operators across time series.
+ Aggregate metrics across dimensions such as service, region, or account.
+ Compute rates, histograms, quantiles, and moving averages.
You can use PromQL queries interactively in [Running PromQL queries in Query Studio (Preview)](CloudWatch-PromQL-QueryStudio.md) and also to create CloudWatch Alarms. For more information, see [PromQL querying](CloudWatch-PromQL-Querying.md) and [Using PromQL in alarms](CloudWatch-PromQL-Alarms.md).
**Note**
CloudWatch uses PromQL based on the Prometheus 3.0 specification. This includes support for UTF-8 metric names and label names.
The following concepts are fundamental to working with PromQL in CloudWatch.
| Concept | Description |
| --- | --- |
| **Time series** | A stream of timestamped values identified by a metric name and a set of key-value pairs called *labels*. Each unique combination of metric name and labels forms a distinct time series. |
| **Instant vector** | A set of time series containing a single sample for each series, all sharing the same timestamp. Returned by queries like `{"http.server.active_requests", "@resource.service.name"="myservice"}`. |
| **Range vector** | A set of time series containing a range of data points over time for each series. Created by appending a time duration selector in brackets, for example, `avg_over_time({"http.server.active_requests", "@resource.service.name"="myservice"}[5m])`. |
| **Label** | A key-value pair attached to a time series. In OTLP-ingested metrics, labels are derived from resource attributes, instrumentation scope, datapoint attributes, and AWS-specific metadata. |
| **Label matcher** | An expression in curly braces that filters time series by label value. Supports exact match (`=`), not equal (`!=`), regex match (`=~`), and negative regex match (`!~`). |
| **Aggregation operator** | A function that combines multiple time series into fewer series. Common operators include `sum`, `avg`, `min`, `max`, `count`, and `topk`. |
## PromQL limits and restrictions
The following table lists the limits and restrictions for PromQL:
| Limit | Value | Additional information | Error code |
| --- | --- | --- | --- |
| Max TPS for query requests per account | 300 | Maximum number of query requests (/query, /query\$1range) per second allowed per account. | 422 |
| Max TPS for discovery requests per account | 10 | Maximum number of discovery requests (/series, /label, /label\$1values) per second allowed per account. | 422 |
| Max concurrent query requests per account | 30 | Maximum number of queries (/query, /query\$1range) an account can have actively executing at the same time. | 429 |
| Max concurrent discovery requests per account | 30 | Maximum number of discovery requests (/series, /labels, /label\$1values) an account can have actively executing at the same time. | 429 |
| Max series returned per query request | 500 | Maximum number of unique time series a query request (/query, /query\$1range) can return. | 200 - truncated response |
| Max labels returned per discovery request | 10,000 | Maximum number of unique labels a discovery request (/series, /labels, /label\$1values) can return. | 200 - truncated response |
| Max range per request | 7 days | Maximum time range a query can span, including range parameters and lookback periods. | 422 |
| Max series scanned per 24h window | 100,000 | Maximum number of unique time series that can be scanned per 24-hour window of query execution. | 422 |
| Max samples scanned per 24h window | 300,000,000 | Maximum number of samples that can be scanned per 24-hour window of query execution. | 422 |
| Max samples processed per 24h window | 3,000,000,000 | Maximum number of samples that can be processed per 24-hour window of query execution. | 422 |
| Execution timeout | 20 seconds | Maximum time the engine can spend evaluating a query, excluding time spent in queue and fetching data from storage. | 422 |
## Supported AWS Regions
The following table lists the AWS Regions where OTLP metrics ingestion, PromQL querying, and Query Studio are available.
| Region name | Region code | OTLP metrics ingest | PromQL query | Query Studio |
| --- | --- | --- | --- | --- |
| US East (N. Virginia) | us-east-1 | ✓ | ✓ | ✓ |
| US West (Oregon) | us-west-2 | ✓ | ✓ | ✓ |
| Europe (Ireland) | eu-west-1 | ✓ | ✓ | ✓ |
| Asia Pacific (Singapore) | ap-southeast-1 | ✓ | ✓ | ✓ |
| Asia Pacific (Sydney) | ap-southeast-2 | ✓ | ✓ | ✓ |
## IAM permissions for PromQL
To execute PromQL queries, you need both `cloudwatch:GetMetricData` and `cloudwatch:ListMetrics` permissions. The following table lists the new PromQL API operations and their required IAM actions:
| API operation | Required actions |
| --- | --- |
| ExecuteMetricQueryPost | `cloudwatch:GetMetricData`, `cloudwatch:ListMetrics` |
| ExecuteMetricQueryGet | `cloudwatch:GetMetricData`, `cloudwatch:ListMetrics` |
| ExecuteMetricRangeQuery | `cloudwatch:GetMetricData`, `cloudwatch:ListMetrics` |
| ExecuteMetricRangeQueryGet | `cloudwatch:GetMetricData`, `cloudwatch:ListMetrics` |
| ExecuteMetricSeriesPost | `cloudwatch:ListMetrics` |
| ExecuteMetricSeriesGet | `cloudwatch:ListMetrics` |
| ExecuteMetricLabelsPost | `cloudwatch:ListMetrics` |
| ExecuteMetricLabelsGet | `cloudwatch:ListMetrics` |
| ExecuteMetricLabelValuesGet | `cloudwatch:ListMetrics` |
# PromQL querying
When you ingest OpenTelemetry metrics into CloudWatch via OpenTelemetry Protocol (OTLP), the hierarchical OTLP data model is flattened into PromQL-compatible labels. This section describes the label structure, the PromQL syntax for querying these labels, and the UTF-8 support in PromQL.
**Note**
PromQL in Prometheus 3 supports full UTF-8 characters in metric names and label names. This is particularly important for OTLP metrics, because OpenTelemetry semantic conventions use dots in attribute names such as `service.name`. Previously, these dots were replaced with underscores during translation, causing discrepancies between what was defined in OTel conventions and what was queryable in Prometheus.
When using PromQL in CloudWatch, the `@` prefix convention distinguishes OTLP-scoped labels from standard Prometheus labels. Fields within each scope use a double-`@` prefix (for example, `@resource.@schema_url`), while attributes use a single-`@` scope prefix, for example, `@resource.service.name`. Datapoint attributes also support bare (un-prefixed) access for backward compatibility with standard PromQL queries, for example, `{"http.server.active_requests"}` and `{"@datapoint.@name"="http.server.active_requests"}` are equivalent.
A PromQL expression is enclosed in curly braces, specifying the metric name and an optional set of label matchers. The following example selects all time series for the `http.server.active_requests` metric:
```
{"http.server.active_requests"}
```
The following example selects all time series for the metric `http.server.active_requests` where the OpenTelemetry resource attribute `service.name` equals `myservice`:
```
{"http.server.active_requests", "@resource.service.name"="myservice"}
```
You can combine multiple label matchers in a single query. The following example selects all time series for the `http.server.active_requests` metric where the OpenTelemetry resource attribute `service.name` equals `myservice` across all US regions:
```
{"http.server.active_requests",
"@resource.service.name"="myservice",
"@aws.region"=~"us-.*"}
```
The following example shows a range query. It calculates the average value of all datapoints within a specified time range for each time series:
```
avg_over_time(
{"http.server.active_requests",
"@resource.service.name"="myservice"}[5m]
)
```
The following table summarizes the prefix conventions for each OTLP scope:
| OTLP scope | Fields prefix | Attributes prefix | Example |
| --- | --- | --- | --- |
| Resource | `@resource.@` | `@resource.` | `@resource.service.name="myservice"` |
| Instrumentation Scope | `@instrumentation.@` | `@instrumentation.` | `@instrumentation.@name="otel-go/metrics"` |
| Datapoint | `@datapoint.@` | `@datapoint.` or bare | `cpu="cpu0"` or `@datapoint.cpu="cpu0"` |
| AWS-reserved | N/A | `@aws.` | `@aws.account_id="123456789"` |
## Querying vended AWS metrics with PromQL
To be able to query vended AWS metrics in PromQL, you first need to enable OTel enrichment of vended metrics. See: [Enabling vended metrics in PromQL](CloudWatch-OTelEnrichment.md).
After you enable OTel enrichment, vended AWS metrics become queryable via PromQL with additional labels. The metric name is the same as the original CloudWatch metric name, and the original CloudWatch dimensions are available as datapoint attributes. The following labels are available (the example below is for an EC2 instance):
| PromQL Label | Description | Example |
| --- | --- | --- |
| `InstanceId` | Original CloudWatch dimension, as a datapoint attribute | `i-0123456789abcdef0` |
| `"@resource.cloud.resource_id"` | Full ARN of the resource | `arn:aws:ec2:us-east-1:123456789012:instance/i-0123456789abcdef0` |
| `"@resource.cloud.provider"` | Cloud provider | `aws` |
| `"@resource.cloud.region"` | AWS Region where this metric originated | `us-east-1` |
| `"@resource.cloud.account.id"` | AWS account ID where this metric originated | `123456789012` |
| `"@instrumentation.@name"` | Instrumentation scope name identifying the source service | `cloudwatch.aws/ec2` |
| `"@instrumentation.cloudwatch.source"` | Source service identifier | `aws.ec2` |
| `"@instrumentation.cloudwatch.solution"` | Enrichment solution identifier | `CloudWatchOTelEnrichment` |
| `"@aws.tag.Environment"` | AWS resource tag | `production` |
| `"@aws.account"` | AWS account where this metric was ingested (system label) | `123456789012` |
| `"@aws.region"` | AWS Region where this metric was ingested (system label) | `us-east-1` |
The following example selects `Invocations` for a specific Lambda function:
```
histogram_sum({Invocations, FunctionName="my-api-handler"})
```
The following example selects Lambda `Errors` for all functions tagged with a specific team:
```
histogram_sum(
{Errors, "@instrumentation.@name"="cloudwatch.aws/lambda", "@aws.tag.Team"="backend"}
)
```
The following example computes the total Lambda `Invocations` grouped by team:
```
sum by ("@aws.tag.Team")(
{Invocations, "@instrumentation.@name"="cloudwatch.aws/lambda"}
)
```
The following example selects all time series for the EC2 `CPUUtilization` metric. The usage of `"@instrumentation.@name"="cloudwatch.aws/ec2"` is to exclusively match CPUUtilization from EC2 and not from other AWS services such as Amazon Relational Database Service:
```
histogram_avg({CPUUtilization, "@instrumentation.@name"="cloudwatch.aws/ec2"})
```
# Running PromQL queries in Query Studio (Preview)
**Note**
Query Studio is currently available as a Preview feature at no additional charge. For supported regions, see [Supported AWS Regions](CloudWatch-PromQL.md#CloudWatch-PromQL-Regions).
Query Studio is an interactive query environment in the CloudWatch console where you can write, run, and visualize PromQL queries against your CloudWatch metrics. You can use Query Studio to explore metrics ingested via OTLP and AWS vended metrics, create visualizations, set up alarms, and add widgets to your CloudWatch dashboards.
You can run PromQL queries programmatically using the CloudWatch API, or interactively in Query Studio.
## Running a PromQL query in Query Studio
**To run a PromQL query in Query Studio**
1. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Query Studio (Preview)**.
1. In the query editor tool, select **PromQL** from the drop-down menu.
1. Use the **Builder** mode to browse and select metric names, labels, and aggregation functions.
1. Or enter your PromQL query via the **Editor** mode, for example `{"http.server.active_requests"}`.
1. (Optional) Adjust the time range using the time interval selector at the top of the page.
1. Choose **Run** to execute the query and view the results.
Query Studio will display the results as a time series graph. You can switch between graph and table views to analyze your data.
## Creating alarms from Query Studio
After running a PromQL query that returns a single time series, you can create a CloudWatch Alarm directly from Query Studio. Choose **Create alarm** from the actions menu to configure alarm thresholds, evaluation periods, and notification actions based on your query results. For more information, see [Using PromQL in alarms](CloudWatch-PromQL-Alarms.md).
## Adding visualizations to dashboards
You can add any Query Studio visualization to a CloudWatch dashboard. After running a query and viewing the results, choose **Add to dashboard** to save the visualization as a dashboard widget. The widget continues to run the PromQL query at the dashboard's refresh interval, keeping your dashboard up to date.
# Using PromQL in alarms
You can create CloudWatch alarms that use PromQL queries to monitor your metrics. PromQL alarms evaluate a PromQL expression and trigger alarm state changes based on the query results.
For information about alarm concepts, see [Concepts](alarm-concepts.md).
For information about alarm data queries, see [Alarm data queries](alarm-data-queries.md).
For information about alarm actions, see [Alarm actions](alarm-actions.md).
For information about alarm limits, see [Limits](alarm-limits.md).
## Creating a PromQL alarm
You can create a PromQL alarm from the CloudWatch console, the AWS CLI, or the CloudWatch API.
**To create a PromQL alarm from the console**
1. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Alarms**, **All alarms**.
1. Choose **Create alarm**.
1. Choose **Select metric**, then choose the **PromQL** tab.
1. Enter your PromQL query. The query must return a single time series for the alarm to evaluate.
1. Configure the alarm conditions, including the threshold, evaluation period, and datapoints to alarm.
1. Configure the alarm actions, such as Amazon SNS notifications.
1. Enter a name and description for the alarm, then choose **Create alarm**.
You can also create a PromQL alarm directly from [Running PromQL queries in Query Studio (Preview)](CloudWatch-PromQL-QueryStudio.md) after running a query that returns a single time series.
## Creating a CloudWatch alarm using PromQL for anomaly detection
You can create a PromQL alarm that triggers when a metric breaches an expected range defined by statistical bounds. The alarm query combines upper and lower bounds into a single expression that returns only the anomalous data points. Any time series returned by the query is considered breaching.
The following example expression detects when an ad request metric exceeds 3 standard deviations from the median over a 60-minute window:
```
1 * {"app.ads.ad_requests"} > quantile_over_time(0.5, {"app.ads.ad_requests"}[60m] offset 1m)
+ 3 * stddev_over_time({"app.ads.ad_requests"}[60m] offset 1m)
or
1 * {"app.ads.ad_requests"} < clamp_min(
quantile_over_time(0.5, {"app.ads.ad_requests"}[60m] offset 1m)
- 3 * stddev_over_time({"app.ads.ad_requests"}[60m] offset 1m),
0)
```
This expression works across multiple label values, so the alarm can track anomalies across your entire fleet. Each breaching time series is tracked as a separate contributor. For more information about how PromQL alarms evaluate contributors, see [PromQL alarms](alarm-promql.md).
You can adjust the multiplier and time window to match your metric's behavior. A higher multiplier produces wider bounds with fewer false positives. A longer time window smooths out short-term spikes. The `clamp_min` function prevents the lower bound from going negative for metrics that can't have negative values.
For more information about building anomaly detection bands with PromQL, see [Anomaly detection using PromQL](CloudWatch_Anomaly_Detection.md#anomaly_detection_promql).