# About alert rules
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
An alerting rule is a set of evaluation criteria that determines whether an alert instance will fire. The rule consists of one or more queries and expressions, a condition, the frequency of evaluation, and the duration over which the condition needs to be met to start firing.
While queries and expressions select the data set to evaluate, a *condition* sets the threshold that the data must meet or exceed to create an alert.
An *interval* specifies how frequently an alerting rule is evaluated. *Duration*, when configured, indicates how long a condition must be met. The alert rules can also define alerting behavior in the absence of data.
**Topics**
+ [Alert rule types](v12-alerting-explore-rules-types.md)
+ [Recording rules](v12-alerting-explore-rule-recording.md)
+ [Queries and conditions](v12-alerting-explore-rules-queries.md)
+ [Alert instances](v12-alerting-rules-instances.md)
+ [Namespaces, folders and groups](v12-alerting-rules-grouping.md)
+ [Alert rule evaluation](v12-alerting-rules-evaluation.md)
+ [State and health of alerting rules](v12-alerting-explore-state.md)
+ [Notification templating](v12-alerting-rules-notification-templates.md)
# Alert rule types
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Grafana supports several alert rule types. Learn more about each of the alert rule types, how they work, and decide which one is best for your use case.
## Grafana managed rules
Grafana managed rules are the most flexible alert rule type. They allow you to create alerts that can act on data from any of your existing data sources.
In addition to supporting multiple data sources, you can add [expressions](v12-panels-query-xform-expressions.md) to transform your data and express alert conditions.
In Grafana managed alerting:
+ Alert rules are created within Grafana, based on one or more data sources.
+ Alert rules are evaluated by the alert rule evaluation engine from within Grafana.
+ Alerts are delivered using the internal Grafana Alertmanager.
**Note**
You can also configure alerts to be delivered using an external Alertmanager, or use both internal and external Alertmanagers. For more information, see [Add an external alertmanager](v12-alerting-setup-alertmanager.md).
## Data source managed rules
To create data source managed alert rules you must have a compatible Prometheus or Loki data source. You can check if your data source supports rule creation via Grafana by testing the data source and observing if the Ruler API is supported.
In data source managed alerting:
+ Alert rules are created and stored within the data source itself.
+ Alert rules can only be created based on Prometheus data.
+ Alert rule evaluation and delivery is distributed across multiple nodes for high-availability and fault tolerance.
## Choose an alert rule type
When choosing which alert rule type to use, consider the following comparison between Grafana managed alert rules and data source managed alert rules.
| Feature | Grafana-managed alert rule | Loki/Mimir-managed alert rule |
| --- | --- | --- |
| Create alert rules based on data from any of our supported data sources | Yes | No: You can only create alert rules that are based on Prometheus data. The data source must have the Ruler API enabled. |
| Mix and match data sources | Yes | No |
| Includes support for recording rules | No | Yes |
| Add expressions to transform your data and set alert conditions | Yes | No |
| Use images in alert notifications | Yes | No |
| Scaling | More resource intensive, depend on the database, and are likely to suffer from transient errors. They only scale vertically. | Store alert rules within the data source itself and allow for “infinite” scaling. Generate and send alert notifications from the location of your data. |
| Alert rule evaluation and delivery | Alert rule evaluation and delivery is done from within Grafana, using an external Alertmanager; or both. | Alert rule evaluation and alert delivery is distributed, meaning there is no single point of failure. |
# Recording rules
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Recording rules are available for compatible Prometheus or Loki data sources, and for Grafana-managed alerts. Recording rules allow you to precompute frequently used queries for better performance.
A recording rule allows you to pre-compute frequently needed or computationally expensive expressions and save their result as a new set of time series. This is useful if you want to run alerts on aggregated data or if you have dashboards that query computationally expensive expressions repeatedly.
Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh.
Read more about [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) in Prometheus.
# Queries and conditions
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
In Grafana, queries play a vital role in fetching and transforming data from supported data sources, which include databases like MySQL and PostgreSQL, time series databases like Prometheus, InfluxDB and Graphite, and services like OpenSearch, Amazon CloudWatch, Azure Monitor and Google Cloud Monitoring.
For more information on supported data sources, see [Data sources and Grafana alerting](v12-alerting-overview-datasources.md).
The process of executing a query involves defining the data source, specifying the desired data to retrieve, and applying relevant filters or transformations. Query languages or syntaxes specific to the chosen data source are utilized for constructing these queries.
In Alerting, you define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
An alert rule consists of one or more queries and expressions that select the data you want to measure.
For more information on queries and expressions, see [Query and transform data](v12-panels-query-xform.md).
## Data source queries
Queries in Grafana can be applied in various ways, depending on the data source and query language being used. Each data source’s query editor provides a customized user interface that helps you write queries that take advantage of its unique capabilities.
Because of the differences between query languages, each data source query editor looks and functions differently. Depending on your data source, the query editor might provide auto-completion features, metric names, variable suggestions, or a visual query-building interface.
Some common types of query components include:
**Metrics or data fields** – Specify the specific metrics or data fields you want to retrieve, such as CPU usage, network traffic, or sensor readings.
**Time range** – Define the time range for which you want to fetch data, such as the last hour, a specific day, or a custom time range.
**Filters** – Apply filters to narrow down the data based on specific criteria, such as filtering data by a specific tag, host, or application.
**Aggregations** – Perform aggregations on the data to calculate metrics like averages, sums, or counts over a given time period.
**Grouping** – Group the data by specific dimensions or tags to create aggregated views or breakdowns.
**Note**
Grafana does not support alert queries with template variables. More information is available [here](https://community.grafana.com/t/template-variables-are-not-supported-in-alert-queries-while-setting-up-alert/2514) in the Grafana Labs forums.
## Expression queries
In Grafana, an expression is used to perform calculations, transformations, or aggregations on the data source queried data. It allows you to create custom metrics or modify existing metrics based on mathematical operations, functions, or logical expressions.
By leveraging expression queries, users can perform tasks such as calculating the percentage change between two values, applying functions like logarithmic or trigonometric functions, aggregating data over specific time ranges or dimensions, and implementing conditional logic to handle different scenarios.
In Alerting, you can only use expressions for Grafana-managed alert rules. For each expression, you can choose from the math, reduce, and resample expressions. These are called multi-dimensional rules, because they generate a separate alert for each series.
You can also use a classic condition, which creates an alert rule that triggers a single alert when its condition is met. As a result, Grafana sends only a single alert even when alert conditions are met for multiple series.
**Note**
Classic conditions exist mainly for compatibility reasons and should be avoided if possible.
**Reduce**
Aggregates time series values in the selected time range into a single value.
**Math**
Performs free-form math functions/operations on time series and number data. Can be used to preprocess time series data or to define an alert condition for number data.
**Resample**
Realigns a time range to a new set of timestamps, this is useful when comparing time series data from different data sources where the timestamps would otherwise not align.
**Threshold**
Checks if any time series data matches the threshold condition.
The threshold expression allows you to compare two single values. It returns `0` when the condition is false and `1` if the condition is true. The following threshold functions are available:
+ Is above (x > y)
+ Is below (x < y)
+ Is within range (x > y1 AND x < y2)
+ Is outside range (x < y1 AND x > y2)
**Classic condition**
Checks if any time series data matches the alert condition.
**Note**
Classic condition expression queries always produce one alert instance only, no matter how many time series meet the condition. Classic conditions exist mainly for compatibility reasons and should be avoided if possible.
## Aggregations
Grafana Alerting provides the following aggregation functions to enable you to further refine your query.
These functions are available for **Reduce** and **Classic condition** expressions only.
| Function | Expression | What it does |
| --- | --- | --- |
| avg | Reduce / Classic | Displays the average of the values |
| min | Reduce / Classic | Displays the lowest value |
| max | Reduce / Classic | Displays the highest value |
| sum | Reduce / Classic | Displays the sum of all values |
| count | Reduce / Classic | Counts the number of values in the result |
| last | Reduce / Classic | Displays the last value |
| median | Reduce / Classic | Displays the median value |
| diff | Classic | Displays the difference between the newest and oldest value |
| diff\$1abs | Classic | Displays the absolute value of diff |
| percent\$1diff | Classic | Displays the percentage value of the difference between newest and oldest value |
| percent\$1diff\$1abs | Classic | Displays the absolute value of percent\$1diff |
| count\$1non\$1null | Classic | Displays a count of values in the result set that aren’t null |
## Alert condition
An alert condition is the query or expression that determines whether the alert will fire or not depending on the value it yields. There can be only one condition which will determine the triggering of the alert.
After you have defined your queries and/or expressions, choose one of them as the alert rule condition.
When the queried data satisfies the defined condition, Grafana triggers the associated alert, which can be configured to send notifications through various channels like email, Slack, or PagerDuty. The notifications inform you about the condition being met, allowing you to take appropriate actions or investigate the underlying issue.
By default, the last expression added is used as the alert condition.
## Recovery threshold
To reduce the noise of flapping alerts, you can set a recovery threshold different to the alert threshold.
Flapping alerts occur when a metric hovers around the alert threshold condition and may lead to frequent state changes, resulting in too many notifications being generated.
Grafana-managed alert rules are evaluated for a specific interval of time. During each evaluation, the result of the query is checked against the threshold set in the alert rule. If the value of a metric is above the threshold, an alert rule fires and a notification is sent. When the value goes below the threshold and there is an active alert for this metric, the alert is resolved, and another notification is sent.
It can be tricky to create an alert rule for a noisy metric. That is, when the value of a metric continually goes above and below a threshold. This is called flapping and results in a series of firing - resolved - firing notifications and a noisy alert state history.
For example, if you have an alert for latency with a threshold of 1000ms and the number fluctuates around 1000 (say 980 ->1010 -> 990 -> 1020, and so on) then each of those will trigger a notification.
To solve this problem, you can set a (custom) recovery threshold, which basically means having two thresholds instead of one. An alert is triggered when the first threshold is crossed and is resolved only when the second threshold is crossed.
For example, you could set a threshold of 1000ms and a recovery threshold of 900ms. This way, an alert rule will only stop firing when it goes under 900ms and flapping is reduced.
# Alert instances
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Grafana managed alerts support multi-dimensional alerting. Each alert rule can create multiple alert instances. This is powerful if you are observing multiple series in a single expression.
Consider the following PromQL expression:
```
sum by(cpu) (
rate(node_cpu_seconds_total{mode!="idle"}[1m])
)
```
A rule using this expression will create as many alert instances as the amount of CPUs we are observing after the first evaluation, allowing a single rule to report the status of each CPU.
# Namespaces, folders and groups
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Alerts can be organized using folders for Grafana managed rules and namespaces for Mimir, Loki, or Prometheus rules and group names.
**Namespaces and folders**
When creating Grafana-managed rules, the folder can be used to perform access control and grant or deny access to all rules within a specific folder.
A namespace contains one or more groups. The rules within a group are run sequentially at a regular interval. The default interval is one minute. You can rename Grafana Mimi or Loki rule namespaces and groups, and edit group evaluation intervals.
**Groups**
The rules within a group are run sequentially at a regular interval, meaning no rules will be evaluated at the same time, and in order of appearance. The default interval is one minute. You can rename Grafana Mimir or Loki rule namespaces or Loki rule namespaces and groups, and edit group evaluation intervals.
**Tip**
If you want rules to be evaluated concurrently and with different intervals, consider storing them in different groups.
**Note**
Grafana managed alert rules are evaluated concurrently instead of sequentially.
# Alert rule evaluation
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.
## Evaluation group
Every alert rule is part of an evaluation group. Each evaluation group contains an evaluation interval that determines how frequently the alert rule is checked.
**Data-source managed** alert rules within the same group are evaluated one after the other, while alert rules in different groups can be evaluated simultaneously. This feature is especially useful when you want to ensure that recording rules are evaluated before any alert rules.
**Grafana-managed** alert rules are evaluated at the same time, regardless of alert rule group. The default evaluation interval is set at 10 seconds, which means that Grafana-managed alert rules are evaluated every 10 seconds to the closest 10-second window on the clock, for example, 10:00:00, 10:00:10, 10:00:20, and so on. You can also configure your own evaluation interval, if required.
**Note**
Evaluation groups and alerts grouping in notification policies are two separate things. Grouping in notification policies allows multiple alerts sharing the same labels to be sent in the same time message.
## Pending period
By setting a pending period, you can avoid unnecessary alerts for temporary problems.
In the pending period, you select the period in which an alert rule can be in breach of the condition until it fires.
**Example**
Imagine you have an alert rule evaluation interval set at every 30 seconds and the pending period to 90 seconds.
Evaluation will occur as follows:
[00:30] First evaluation - condition not met.
[01:00] Second evaluation - condition breached. Pending counter starts.**Alert starts pending.**
[01:30] Third evaluation - condition breached. Pending counter = 30s. **Pending state.**
[02:00] Fourth evaluation - condition breached. Pending counter = 60s **Pending state.**
[02:30] Fifth evaluation - condition breached. Pending counter = 90s. **Alert starts firing**
If the alert rule has a condition that needs to be in breach for a certain amount of time before it takes action, then its state changes as follows:
+ When the condition is first breached, the rule goes into a "pending" state.
+ The rule stays in the "pending" state until the condition has been broken for the required amount of time - pending period.
+ Once the required time has passed, the rule goes into a "firing" state.
+ If the condition is no longer broken during the pending period, the rule goes back to its normal state.
**Note**
If you want to skip the pending state, you can simply set the pending period to 0. This effectively skips the pending period and your alert rule will start firing as soon as the condition is breached.
When an alert rule fires, alert instances are produced, which are then sent to the Alertmanager.
# State and health of alerting rules
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
The state and health of alerting rules help you understand several key status indicators about your alerts.
There are three key components: *alert rule state*, *alert instance state*, and *alert rule health*. Although related, each component conveys subtly different information.
**Alert rule state**
An alert rule can be in one of the following states:
| State | Description |
| --- | --- |
| Normal | None of the time series returned by the evaluation engine is in a `Pending` or `Firing` state. |
| Pending | At least one time series returned by the evaluation engine is `Pending`. |
| Firing | At least one time series returned by the evaluation engine is `Firing`. |
| Recovering | The alert condition is no longer firing but has not yet returned to normal. |
**Note**
Alerts will transition first to `pending` and then `firing`, thus it will take at least two evaluation cycles before an alert is fired.
**Alert instance state**
An alert instance can be in one of the following states:
| State | Description |
| --- | --- |
| Normal | The state of an alert that is neither firing nor pending, everything is working correctly. |
| Pending | The state of an alert that has been active for less than the configured threshold duration. |
| Alerting | The state of an alert that has been active for longer than the configured threshold duration. |
| Recovering | The state of an alert that was previously firing but the alert condition is no longer met. The alert has not yet returned to normal. |
| NoData | No data has been received for the configured time window. |
| Error | The error that occurred when attempting to evaluate an alerting rule. |
**Keep last state**
An alert rule can be configured to keep the last state when `NoData` or `Error` state is encountered. This will both prevent alerts from firing, and from resolving and re-firing. Just like normal evaluation, the alert rule will transition from `Pending` to `Firing` after the pending period has elapsed.
**Alert rule health**
An alert rule can have one the following health statuses:
| State | Description |
| --- | --- |
| Ok | No error when evaluating an alerting rule. |
| Error | An error occurred when evaluating an alerting rule. |
| NoData | The absence of data in at least one time series returned during a rule evaluation. |
**Special alerts for `NoData` and `Error`**
When evaluation of an alerting rule produces state `NoData` or `Error`, Grafana Alerting will generate alert instances that have the following additional labels:
| Label | Description |
| --- | --- |
| alertname | Either `DatasourceNoData` or `DatasourceError` depending on the state. |
| datasource\$1uid | The UID of the data source that caused the state. |
You can handle these alerts the same way as regular alerts by adding a silence, route to a contact point, and so on.
# Notification templating
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Notifications sent via contact points are built using notification templates. Grafana’s default templates are based on the [Go templating system](https://golang.org/pkg/text/template) where some fields are evaluated as text, while others are evaluated as HTML (which can affect escaping).
The default template [default\$1template.go](https://github.com/grafana/alerting/blob/main/templates/default_template.go) is a useful reference for custom templates.
Since most of the contact point fields can be templated, you can create reusable custom templates and use them in multiple contact points. To learn about custom notifications using templates, see [Customize notifications](v12-alerting-manage-notifications.md).
**Nested templates**
You can embed templates within other templates.
For example, you can define a template fragment using the `define` keyword.
```
{{ define "mytemplate" }}
{{ len .Alerts.Firing }} firing. {{ len .Alerts.Resolved }} resolved.
{{ end }}
```
You can then embed custom templates within this fragment using the `template` keyword. For example:
```
Alert summary:
{{ template "mytemplate" . }}
```
You can use any of the following built-in template options to embed custom templates.
| Name | Notes |
| --- | --- |
| `default.title` | Displays high-level status information. |
| `default.message` | Provides a formatted summary of firing and resolved alerts. |
| `teams.default.message` | Similar to `default.messsage`, formatted for Microsoft Teams. |
**HTML in notification templates**
HTML in alerting notification templates is escaped. We do not support rendering of HTML in the resulting notification.
Some notifiers support alternative methods of changing the look and feel of the resulting notification. For example, Grafana installs the base template for alerting emails to `/public/emails/ng_alert_notification.html`. You can edit this file to change the appearance of all alerting emails.