# Overview
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Whether you’re just starting out or you’re a more experienced user of Grafana Alerting, learn more about the fundamentals and available features that help you create, manage, and respond to alerts; and improve your team’s ability to resolve issues quickly.
## Principles
In Prometheus-based alerting systems, you have an alert generator that creates alerts and an alert receiver that receives alerts. For example, Prometheus is an alert generator and is responsible for evaluating alert rules, while Alertmanager is an alert receiver and is responsible for grouping, inhibiting, silencing, and sending notifications about firing and resolved alerts.
Grafana Alerting is built on the Prometheus model of designing alerting systems. It has an internal alert generator responsible for scheduling and evaluating alert rules, as well as an internal alert receiver responsible for grouping, inhibiting, silencing, and sending notifications. Grafana doesn’t use Prometheus as its alert generator because Grafana Alerting needs to work with many other data sources in addition to Prometheus. However, it does use Alertmanager as its alert receiver.
Alerts are sent to the alert receiver where they are routed, grouped, inhibited, silenced and notified. In Grafana Alerting, the default alert receiver is the Alertmanager embedded inside Grafana, and is referred to as the Grafana Alertmanager. However, you can use other Alertmanagers too, and these are referred to as [External Alertmanagers](v12-alerting-setup-alertmanager.md).
## Fundamentals
The following provides an overview of the different parts of Grafana alerting.
### Alert rules
An alert rule is a set of criteria that determine when an alert should fire. It consists of one or more queries and expressions, a condition which needs to be met, an interval which determines how often the alert rule is evaluated, and a duration over which the condition must be met for an alert to fire.
Alert rules are evaluated over their interval, and each alert rule can have zero, one, or any number of alerts firing at a time. The state of the alert rule is determined by its most `severe` alert, which can be one of Normal, Pending, or Firing. For example, if at least one of an alert rule’s alerts are firing then the alert rule is also firing. The health of an alert rule is determined by the status of its most recent evaluation. These can be OK, Error, and NoData.
Alert rules can be configured to keep their last state when a query returns no data or an error, preventing unnecessary alert state changes. Alert rules also maintain a version history, allowing you to track changes and revert to previous configurations. Alert rules support a Recovering state that indicates when an alert condition is no longer firing but has not yet returned to normal.
A very important feature of alert rules is that they support custom annotations and labels. These allow you to instrument alerts with additional metadata such as summaries and descriptions, and add additional labels to route alerts to specific notification policies.
### Alerts
Alerts are uniquely identified by sets of key/value pairs called Labels. Each key is a label name and each value is a label value. For example, one alert might have the labels `foo=bar` and another alert might have the labels `foo=baz`. An alert can have many labels such as `foo=bar,bar=baz` but it cannot have the same label twice such as `foo=bar,foo=baz`. Two alerts cannot have the same labels either, and if two alerts have the same labels such as `foo=bar,bar=baz` and `foo=bar,bar=baz` then one of the alerts will be discarded. Alerts are resolved when the condition in the alert rule is no longer met, or the alert rule is deleted.
In Grafana Managed Alerts, alerts can be in Normal, Pending, Alerting, No Data or Error states. In Data source Managed Alerts, such as Mimir and Loki, alerts can be in Normal, Pending and Alerting, but not NoData or Error.
### Contact points
Contact points determine where notifications are sent. For example, you might have a contact point that sends notifications to an email address, to Slack, to an incident management system (IRM) such as Grafana OnCall or Pagerduty, or to a webhook.
The notifications that are sent from contact points can be customized using notification templates. You can use notification templates to change the title, message, and structure of the notification. Notification templates are not specific to individual integrations or contact points.
### Notification policies
Notification policies group alerts and then route them to contact points. They determine when notifications are sent, and how often notifications should be repeated.
Alerts are matched to notification policies using label matchers. These are human-readable expressions that assert if the alert’s labels exactly match, do not exactly match, contain, or do not contain some expected text. For example, the matcher `foo=bar` matches alerts with the label `foo=bar` while the matcher `foo=~[a-zA-Z]+` matches alerts with any label called foo with a value that matches the regular expression `[a-zA-Z]+`.
By default, an alert can only match one notification policy. However, with the `continue` feature alerts can be made to match any number of notification policies at the same time. For more information about notification policies, see [Notification Policies](v12-alerting-explore-notifications-policies-details.md).
### Silences and mute timings
Silences and mute timings allow you to pause notifications for specific alerts or even entire notification policies. Use a silence to pause notifications on an ad-hoc basis, such as while working on a fix for an alert; and use mute timings to pause notifications at regular intervals, such as during regularly scheduled maintenance windows.
**Topics**
+ [Principles](#v12-alerting-overview-principles)
+ [Fundamentals](#v12-alerting-overview-fundamentals)
+ [Data sources and Grafana alerting](v12-alerting-overview-datasources.md)
+ [Alerting on numeric data](v12-alerting-overview-numeric.md)
+ [Labels and annotations](v12-alerting-overview-labels.md)
+ [About alert rules](v12-alerting-explore-rules.md)
+ [Alertmanager](v12-alerting-explore-alertmanager.md)
+ [Contact points](v12-alerting-explore-contacts.md)
+ [Notifications](v12-alerting-explore-notifications.md)
+ [Alerting high availability](v12-alerting-explore-high-availability.md)
# Data sources and Grafana alerting
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
There are a number of data sources that are compatible with Grafana Alerting. Each data source is supported by a plugin. Grafana alerting requires that data source plugins be *backend* plugins, in order to evaluate rules using the data source, because the evaluation engine runs on the backend. Plugins must also specify that they are compatible with Grafana alerting.
Data sources are added and updated over time. The following data sources are known to be compatible with Grafana alerting.
+ [Connect to an Amazon CloudWatch data source](using-amazon-cloudwatch-in-AMG.md)
+ [Connect to an Azure Monitor data source](using-azure-monitor-in-AMG.md)
+ [Connect to an Amazon OpenSearch Service data source](using-Amazon-OpenSearch-in-AMG.md)
+ [Connect to a Google Cloud Monitoring data source](using-google-cloud-monitoring-in-grafana.md)
+ [Connect to a Graphite data source](using-graphite-in-AMG.md)
+ [Connect to an InfluxDB data source](using-influxdb-in-AMG.md)
+ [Connect to a Loki data source](using-loki-in-AMG.md)
+ [Connect to a Microsoft SQL Server data source](using-microsoft-sql-server-in-AMG.md)
+ [Connect to a MySQL data source](using-mysql-in-AMG.md)
+ [Connect to an OpenTSDB data source](using-opentsdb-in-AMG.md)
+ [Connect to a PostgreSQL data source](using-postgresql-in-AMG.md)
+ [Connect to Amazon Managed Service for Prometheus and open-source Prometheus data sources](prometheus-data-source.md)
+ [Connect to a Jaeger data source](jaeger-data-source.md)
+ [Connect to a Zipkin data source](zipkin-data-source.md)
+ [Connect to a Tempo data source](tempo-data-source.md)
+ [Configure a TestData data source for testing](testdata-data-source.md)
For more detailed information about data sources and data source plugins in Amazon Managed Grafana, see [Connect to data sources](AMG-data-sources.md).
# Alerting on numeric data
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
This topic describes how Grafana handles alerting on numeric rather than time series data.
Among certain data sources, numeric data that is not time series can be directly alerted on, or passed into Server Side Expressions (SSE). This allows for more processing and resulting efficiency within the data source, and it can also simplify alert rules. When alerting on numeric data instead of time series data, there is no need to reduce each labeled time series into a single number. Instead labeled numbers are returned to Grafana instead.
## Tabular data
This feature is supported with backend data sources that query tabular data:
+ SQL data sources such as MySQL, Postgres, MSSQL, and Oracle.
+ The Azure Kusto based services: Azure Monitor (Logs), Azure Monitor (Azure Resource Graph), and Azure Data Explorer.
A query with Grafana managed alerts or SSE is considered numeric with these data sources, if:
+ The “Format AS” option is set to “Table” in the data source query.
+ The table response returned to Grafana from the query includes only one numeric (e.g. int, double, float) column, and optionally additional string columns.
If there are string columns, then those columns become labels. The name of a column becomes the label name, and the value for each row becomes the value of the corresponding label. If multiple rows are returned, then each row should be uniquely identified their labels.
## Example
For a MySQL table called “DiskSpace”:
| Time | Host | Disk | PercentFree |
| --- | --- | --- | --- |
| 2021-June-7 | web1 | /etc | 3 |
| 2021-June-7 | web2 | /var | 4 |
| 2021-June-7 | web3 | /var | 8 |
| ... | ... | ... | ... |
You can query the data filtering on time, but without returning the time series to Grafana. For example, an alert that would trigger per Host, Disk when there is less than 5% free space:
```
SELECT Host , Disk , CASE WHEN PercentFree < 5.0 THEN PercentFree ELSE 0 END FROM (
SELECT
Host,
Disk,
Avg(PercentFree)
FROM DiskSpace
Group By
Host,
Disk
Where __timeFilter(Time)
```
This query returns the following Table response to Grafana:
| Host | Disk | PercentFree |
| --- | --- | --- |
| web1 | /etc | 3 |
| web2 | /var | 4 |
| web3 | /var | 0 |
When this query is used as the **condition** in an alert rule, then the non-zero will be alerting. As a result, three alert instances are produced:
| Labels | Status |
| --- | --- |
| \$1Host=web1,disk=/etc\$1 | Alerting |
| \$1Host=web2,disk=/var\$1 | Alerting |
| \$1Host=web3,disk=/var\$1 | Normal |
# Labels and annotations
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Labels and annotations contain information about an alert. Both labels and annotations have the same structure: a set of named values; however their intended uses are different. An example of label, or the equivalent annotation, might be `alertname="test"`.
The main difference between a label and an annotation is that labels are used to differentiate an alert from all other alerts, while annotations are used to add additional information to an existing alert.
For example, consider two high CPU alerts: one for `server1` and another for `server2`. In such an example, we might have a label called `server` where the first alert has the label `server="server1"` and the second alert has the label `server="server2"`. However, we might also want to add a description to each alert such as `"The CPU usage for server1 is above 75%."`, where `server1` and `75%` are replaced with the name and CPU usage of the server (please refer to the documentation on [Templating labels and annotations](v12-alerting-overview-labels-templating.md) for how to do this). This kind of description would be more suitable as an annotation.
## Labels
Labels contain information that identifies an alert. An example of a label might be `server=server1`. Each alert can have more than one label, and the complete set of labels for an alert is called its label set. It is this label set that identifies the alert.
For example, an alert might have the label set `{alertname="High CPU usage",server="server1"}` while another alert might have the label set `{alertname="High CPU usage",server="server2"}`. These are two separate alerts because although their `alertname` labels are the same, their `server` labels are different.
The label set for an alert is a combination of the labels from the datasource, custom labels from the alert rule, and a number of reserved labels such as `alertname`.
**Custom Labels**
Custom labels are additional labels from the alert rule. Like annotations, custom labels must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired. Documentation on how to template custom labels can be found [here](v12-alerting-overview-labels-templating.md).
When using custom labels with templates, it is important to make sure that the label value does not change between consecutive evaluations of the alert rule as this will end up creating large numbers of distinct alerts. However, it is OK for the template to produce different label values for different alerts. For example, do not put the value of the query in a custom label as this will end up creating a new set of alerts each time the value changes. Instead use annotations.
It is also important to make sure that the label set for an alert does not have two or more labels with the same name. If a custom label has the same name as a label from the datasource then it will replace that label. However, should a custom label have the same name as a reserved label then the custom label will be omitted from the alert.
## Annotations
Annotations are named pairs that add additional information to existing alerts. There are a number of suggested annotations in Grafana such as `description`, `summary`, `runbook_url`, `dashboardUId` and `panelId`. Like custom labels, annotations must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired. If an annotation contains template code, the template is evaluated once when the alert is fired. It is not re-evaluated, even when the alert is resolved. Documentation on how to template annotations can be found [here](v12-alerting-overview-labels-templating.md).
**Topics**
+ [Labels](#v12-alerting-overview-labels-labels)
+ [Annotations](#v12-alerting-overview-labels-annotations)
+ [How label matching works](v12-alerting-overview-labels-matching.md)
+ [Labels in Grafana Alerting](v12-alerting-overview-labels-alerting.md)
+ [Templating labels and annotations](v12-alerting-overview-labels-templating.md)
# How label matching works
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Use labels and label matchers to link alert rules to notification policies and silences. This allows for a very flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
A label matcher consists of 3 distinct parts, the **label**, the **value** and the **operator**.
+ The **Label** field is the name of the label to match. It must exactly match the label name.
+ The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Operator** value.
+ The **Operator** field is the operator to match against the label value. The available operators are:
| Operator | Description |
| --- | --- |
| `=` | Select labels that are exactly equal to the value. |
| `!=` | Select labels that are not equal to the value. |
| `=~` | Select labels that regex-match the value. |
| `!~` | Select labels that do not regex-match the value. |
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
## Example
If you define the following set of labels for your alert:
```
{ foo=bar, baz=qux, id=12 }
```
then:
+ A label matcher defined as `foo=bar` matches this alert rule.
+ A label matcher defined as `foo!=bar` does *not* match this alert rule.
+ A label matcher defined as `id=~[0-9]+` matches this alert rule.
+ A label matcher defined as `baz!~[0-9]+` matches this alert rule.
+ Two label matchers defined as `foo=bar` and `id=~[0-9]+` match this alert rule.
## Exclude labels
You can also write label matchers to exclude labels.
Here is an example that shows how to exclude the label `team`. You can choose between any of these values to exclude the label.
+ `team=""`
+ `team!~.+`
+ `team=~^$`
# Labels in Grafana Alerting
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
This topic explains why labels are a fundamental component of alerting.
+ The complete set of labels for an alert is what uniquely identifies an alert within Grafana alerts.
+ The Alertmanager uses labels to match alerts for silences and alert groups in notification policies.
+ The alerting UI shows labels for every alert instance generated during evaluation of that rule.
+ Contact points can access labels to dynamically generate notifications that contain information specific to the alert that is resulting in a notification.
+ You can add labels to an [alerting rule](v12-alerting-configure.md). Labels are manually configurable, use template functions, and can reference other labels. Labels added to an alerting rule take precedence in the event of a collision between labels (except in the case of Grafana reserved labels, see below for more information).
## External Alertmanager compatibility
Grafana’s built-in Alertmanager supports both Unicode label keys and values. If you are using an external Prometheus Alertmanager, label keys must be compatible with their [data model](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). This means that label keys must only contain **ASCII letters**, **numbers**, as well as **underscores** and match the regex `[a-zA-Z_][a-zA-Z0-9_]*`. Any invalid characters will be removed or replaced by the Grafana alerting engine before being sent to the external Alertmanager according to the following rules:
+ `Whitespace` will be removed.
+ `ASCII characters` will be replaced with `_`.
+ `All other characters` will be replaced with their lower-case hex representation. If this is the first character it will be prefixed with `_`.
**Note**
If multiple label keys are sanitized to the same value, the duplicates will have a short hash of the original label appended as a suffix.
## Grafana reserved labels
**Note**
Labels prefixed with `grafana_` are reserved by Grafana for special use. If a manually configured label is added beginning with `grafana_` it will be overwritten in case of collision.
Grafana reserved labels can be used in the same way as manually configured labels. The current list of available reserved labels are:
| Label | Description |
| --- | --- |
| grafana\$1folder | Title of the folder containing the alert. |
# Templating labels and annotations
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
You can use templates to include data from queries and expressions in labels and annotations. For example, you might want to set the severity label for an alert based on the value of the query, or use the instance label from the query in a summary annotation so you know which server is experiencing high CPU usage.
All templates should be written in [text/template](https://pkg.go.dev/text/template). Regardless of whether you are templating a label or an annotation, you should write each template inline inside the label or annotation that you are templating. This means you cannot share templates between labels and annotations, and instead you will need to copy templates wherever you want to use them.
Each template is evaluated whenever the alert rule is evaluated, and is evaluated for every alert separately. For example, if your alert rule has a templated summary annotation, and the alert rule has 10 firing alerts, then the template will be executed 10 times, once for each alert. You should try to avoid doing expensive computations in your templates as much as possible.
## Examples
Rather than write a complete tutorial on text/template, the following examples attempt to show the most common use-cases we have seen for templates. You can use these examples verbatim, or adapt them as necessary for your use case. For more information about how to write text/template see the [text/template](https://pkg.go.dev/text/template) documentation.
**Print all labels, comma separated**
To print all labels, comma separated, print the `$labels` variable:
```
{{ $labels }}
```
For example, given an alert with the labels `alertname=High CPU usage`, `grafana_folder=CPU alerts` and `instance=server1`, this would print:
```
alertname=High CPU usage, grafana_folder=CPU alerts, instance=server1
```
**Note**
If you are using classic conditions then `$labels` will not contain any labels from the query. Refer to [the \$1labels variable](#v12-alerting-overview-labels-templating-the-labels-variable) for more information.
**Print all labels, one per line**
To print all labels, one per line, use a `range` to iterate over each key/value pair and print them individually. Here `$k` refers to the name and `$v` refers to the value of the current label:
```
{{ range $k, $v := $labels -}}
{{ $k }}={{ $v }}
{{ end }}
```
For example, given an alert with the labels `alertname=High CPU usage`, `grafana_folder=CPU alerts` and `instance=server1`, this would print:
```
alertname=High CPU usage
grafana_folder=CPU alerts
instance=server1
```
**Note**
If you are using classic conditions then `$labels` will not contain any labels from the query. Refer to [the \$1labels variable](#v12-alerting-overview-labels-templating-the-labels-variable) for more information.
**Print an individual label**
To print an individual label use the `index` function with the `$labels` variable:
```
The host {{ index $labels "instance" }} has exceeded 80% CPU usage for the last 5 minutes
```
For example, given an alert with the label `instance=server1`, this would print:
```
The host server1 has exceeded 80% CPU usage for the last 5 minutes
```
**Note**
If you are using classic conditions then `$labels` will not contain any labels from the query. Refer to [the \$1labels variable](#v12-alerting-overview-labels-templating-the-labels-variable) for more information.
**Print the value of a query**
To print the value of an instant query you can print its Ref ID using the `index` function and the `$values` variable:
```
{{ index $values "A" }}
```
For example, given an instant query that returns the value 81.2345, this will print:
```
81.2345
```
To print the value of a range query you must first reduce it from a time series to an instant vector with a reduce expression. You can then print the result of the reduce expression by using its Ref ID instead. For example, if the reduce expression takes the average of A and has the Ref ID B you would write:
```
{{ index $values "B" }}
```
**Print the humanized value of a query**
To print the humanized value of an instant query use the `humanize` function:
```
{{ humanize (index $values "A").Value }}
```
For example, given an instant query that returns the value 81.2345, this will print:
```
81.234
```
To print the humanized value of a range query you must first reduce it from a time series to an instant vector with a reduce expression. You can then print the result of the reduce expression by using its Ref ID instead. For example, if the reduce expression takes the average of A and has the Ref ID B you would write:
```
{{ humanize (index $values "B").Value }}
```
**Print the value of a query as a percentage**
To print the value of an instant query as a percentage use the `humanizePercentage` function:
```
{{ humanizePercentage (index $values "A").Value }}
```
This function expects the value to be a decimal number between 0 and 1. If the value is instead a decimal number between 0 and 100 you can divide it by 100 either in your query or using a math expression. If the query is a range query you must first reduce it from a time series to an instant vector with a reduce expression.
**Set a severity from the value of a query**
To set a severity label from the value of a query use an if statement and the greater than comparison function. Make sure to use decimals (`80.0`, `50.0`, `0.0`, etc) when doing comparisons against `$values` as text/template does not support type coercion. You can find a list of all the supported comparison functions [here](https://pkg.go.dev/text/template#hdr-Functions).
```
{{ if (gt $values.A.Value 80.0) -}}
high
{{ else if (gt $values.A.Value 50.0) -}}
medium
{{ else -}}
low
{{- end }}
```
**Print all labels from a classic condition**
You cannot use `$labels` to print labels from the query if you are using classic conditions, and must use `$values` instead. The reason for this is classic conditions discard these labels to enforce uni-dimensional behavior (at most one alert per alert rule). If classic conditions didn’t discard these labels, then queries that returned many time series would cause alerts to flap between firing and resolved constantly as the labels would change every time the alert rule was evaluated.
Instead, the `$values` variable contains the reduced values of all time series for all conditions that are firing. For example, if you have an alert rule with a query A that returns two time series, and a classic condition B with two conditions, then `$values` would contain `B0`, `B1`, `B2` and `B3`. If the classic condition B had just one condition, then `$values` would contain just `B0` and `B1`.
To print all labels of all firing time series use the following template (make sure to replace `B` in the regular expression with the Ref ID of the classic condition if it’s different):
```
{{ range $k, $v := $values -}}
{{ if (match "B[0-9]+" $k) -}}
{{ $k }}: {{ $v.Labels }}{{ end }}
{{ end }}
```
For example, a classic condition for two time series exceeding a single condition would print:
```
B0: instance=server1
B1: instance=server2
```
If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then its labels will be duplicated for each condition that is exceeded:
```
B0: instance=server1
B1: instance=server2
B2: instance=server1
B3: instance=server2
```
If you need to print unique labels you should consider changing your alert rules from uni-dimensional to multi-dimensional instead. You can do this by replacing your classic condition with reduce and math expressions.
**Print all values from a classic condition**
To print all values from a classic condition take the previous example and replace `$v.Labels` with `$v.Value`:
```
{{ range $k, $v := $values -}}
{{ if (match "B[0-9]+" $k) -}}
{{ $k }}: {{ $v.Value }}{{ end }}
{{ end }}
```
For example, a classic condition for two time series exceeding a single condition would print:
```
B0: 81.2345
B1: 84.5678
```
If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then `$values` will contain the values of all conditions:
```
B0: 81.2345
B1: 92.3456
B2: 84.5678
B3: 95.6789
```
## Variables
The following variables are available to you when templating labels and annotations:
### The labels variable
The `$labels` variable contains all labels from the query. For example, suppose you have a query that returns CPU usage for all of your servers, and you have an alert rule that fires when any of your servers have exceeded 80% CPU usage for the last 5 minutes. You want to add a summary annotation to the alert that tells you which server is experiencing high CPU usage. With the `$labels` variable you can write a template that prints a human-readable sentence such as:
```
CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes
```
**Note**
If you are using a classic condition then `$labels` will not contain any labels from the query. Classic conditions discard these labels in order to enforce uni-dimensional behavior (at most one alert per alert rule). If you want to use labels from the query in your template then follow the previous *Print all labels from a classic condition* example.
### The value variable
The `$value` variable is a string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule. It does not contain the results of range queries, as these can return anywhere from 10s to 10,000s of rows or metrics. If it did, for especially large queries a single alert could use 10s of MBs of memory and Grafana would run out of memory very quickly.
To print the `$value` variable in the summary you would write something like this:
```
CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ $value }}
```
And it would look something like this:
```
CPU usage for instance1 has exceeded 80% for the last 5 minutes: [ var='A' labels={instance=instance1} value=81.234 ]
```
Here `var='A'` refers to the instant query with Ref ID A, `labels={instance=instance1}` refers to the labels, and `value=81.234` refers to the average CPU usage over the last 5 minutes.
If you want to print just some of the string instead of the full string then use the `$values` variable. It contains the same information as `$value`, but in a structured table, and is much easier to use then writing a regular expression to match just the text you want.
### The values variable
The `$values` variable is a table containing the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs.
To print the value of the instant query with Ref ID A:
```
CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ index $values "A" }}
```
For example, given an alert with the labels `instance=server1` and an instant query with the value `81.2345`, this would print:
```
CPU usage for instance1 has exceeded 80% for the last 5 minutes: 81.2345
```
If the query in Ref ID A is a range query rather than an instant query then add a reduce expression with Ref ID B and replace `(index $values "A")` with `(index $values "B")`:
```
CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ index $values "B" }}
```
## Functions
The following functions are available to you when templating labels and annotations:
**args**
The `args` function translates a list of objects to a map with keys arg0, arg1 etc. This is intended to allow multiple arguments to be passed to templates.
```
{{define "x"}}{{.arg0}} {{.arg1}}{{end}}{{template "x" (args 1 "2")}}
```
```
1 2
```
**externalURL**
The `externalURL` function returns the external URL of the Grafana server.
```
{{ externalURL }}
```
```
https://example.com/grafana
```
**graphLink**
The `graphLink` function returns the path to the graphical view in [Explore in Grafana version 12](v12-explore.md) for the given expression and data source.
```
{{ graphLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }}
```
```
/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":false,"range":true}]
```
**humanize**
The `humanize` function humanizes decimal numbers.
```
{{ humanize 1000.0 }}
```
```
1k
```
**humanize1024**
The `humanize1024` works similar to `humanize` but uses 1024 as the base rather than 1000.
```
{{ humanize1024 1024.0 }}
```
```
1ki
```
**humanizeDuration**
The `humanizeDuration` function humanizes a duration in seconds.
```
{{ humanizeDuration 60.0 }}
```
```
1m 0s
```
**humanizePercentage**
The `humanizePercentage` function humanizes a ratio value to a percentage.
```
{{ humanizePercentage 0.2 }}
```
```
20%
```
**humanizeTimestamp**
The `humanizeTimestamp` function humanizes a Unix timestamp.
```
{{ humanizeTimestamp 1577836800.0 }}
```
```
2020-01-01 00:00:00 +0000 UTC
```
**match**
The `match` function matches the text against a regular expression pattern.
```
{{ match "a.*" "abc" }}
```
```
true
```
**pathPrefix**
The `pathPrefix` function returns the path of the Grafana server.
```
{{ pathPrefix }}
```
```
/grafana
```
**tableLink**
The `tableLink` function returns the path to the tabular view in [Explore in Grafana version 12](v12-explore.md) for the given expression and data source.
```
{{ tableLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }}
```
```
/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":true,"range":false}]
```
**title**
The `title` function capitalizes the first character of each word.
```
{{ title "hello, world!" }}
```
```
Hello, World!
```
**toLower**
The `toLower` function returns all text in lowercase.
```
{{ toLower "Hello, world!" }}
```
```
hello, world!
```
**toUpper**
The `toUpper` function returns all text in uppercase.
```
{{ toUpper "Hello, world!" }}
```
```
HELLO, WORLD!
```
**reReplaceAll**
The `reReplaceAll` function replaces text matching the regular expression.
```
{{ reReplaceAll "localhost:(.*)" "example.com:$1" "localhost:8080" }}
```
```
example.com:8080
```
# About alert rules
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
An alerting rule is a set of evaluation criteria that determines whether an alert instance will fire. The rule consists of one or more queries and expressions, a condition, the frequency of evaluation, and the duration over which the condition needs to be met to start firing.
While queries and expressions select the data set to evaluate, a *condition* sets the threshold that the data must meet or exceed to create an alert.
An *interval* specifies how frequently an alerting rule is evaluated. *Duration*, when configured, indicates how long a condition must be met. The alert rules can also define alerting behavior in the absence of data.
**Topics**
+ [Alert rule types](v12-alerting-explore-rules-types.md)
+ [Recording rules](v12-alerting-explore-rule-recording.md)
+ [Queries and conditions](v12-alerting-explore-rules-queries.md)
+ [Alert instances](v12-alerting-rules-instances.md)
+ [Namespaces, folders and groups](v12-alerting-rules-grouping.md)
+ [Alert rule evaluation](v12-alerting-rules-evaluation.md)
+ [State and health of alerting rules](v12-alerting-explore-state.md)
+ [Notification templating](v12-alerting-rules-notification-templates.md)
# Alert rule types
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Grafana supports several alert rule types. Learn more about each of the alert rule types, how they work, and decide which one is best for your use case.
## Grafana managed rules
Grafana managed rules are the most flexible alert rule type. They allow you to create alerts that can act on data from any of your existing data sources.
In addition to supporting multiple data sources, you can add [expressions](v12-panels-query-xform-expressions.md) to transform your data and express alert conditions.
In Grafana managed alerting:
+ Alert rules are created within Grafana, based on one or more data sources.
+ Alert rules are evaluated by the alert rule evaluation engine from within Grafana.
+ Alerts are delivered using the internal Grafana Alertmanager.
**Note**
You can also configure alerts to be delivered using an external Alertmanager, or use both internal and external Alertmanagers. For more information, see [Add an external alertmanager](v12-alerting-setup-alertmanager.md).
## Data source managed rules
To create data source managed alert rules you must have a compatible Prometheus or Loki data source. You can check if your data source supports rule creation via Grafana by testing the data source and observing if the Ruler API is supported.
In data source managed alerting:
+ Alert rules are created and stored within the data source itself.
+ Alert rules can only be created based on Prometheus data.
+ Alert rule evaluation and delivery is distributed across multiple nodes for high-availability and fault tolerance.
## Choose an alert rule type
When choosing which alert rule type to use, consider the following comparison between Grafana managed alert rules and data source managed alert rules.
| Feature | Grafana-managed alert rule | Loki/Mimir-managed alert rule |
| --- | --- | --- |
| Create alert rules based on data from any of our supported data sources | Yes | No: You can only create alert rules that are based on Prometheus data. The data source must have the Ruler API enabled. |
| Mix and match data sources | Yes | No |
| Includes support for recording rules | No | Yes |
| Add expressions to transform your data and set alert conditions | Yes | No |
| Use images in alert notifications | Yes | No |
| Scaling | More resource intensive, depend on the database, and are likely to suffer from transient errors. They only scale vertically. | Store alert rules within the data source itself and allow for “infinite” scaling. Generate and send alert notifications from the location of your data. |
| Alert rule evaluation and delivery | Alert rule evaluation and delivery is done from within Grafana, using an external Alertmanager; or both. | Alert rule evaluation and alert delivery is distributed, meaning there is no single point of failure. |
# Recording rules
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Recording rules are available for compatible Prometheus or Loki data sources, and for Grafana-managed alerts. Recording rules allow you to precompute frequently used queries for better performance.
A recording rule allows you to pre-compute frequently needed or computationally expensive expressions and save their result as a new set of time series. This is useful if you want to run alerts on aggregated data or if you have dashboards that query computationally expensive expressions repeatedly.
Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh.
Read more about [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) in Prometheus.
# Queries and conditions
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
In Grafana, queries play a vital role in fetching and transforming data from supported data sources, which include databases like MySQL and PostgreSQL, time series databases like Prometheus, InfluxDB and Graphite, and services like OpenSearch, Amazon CloudWatch, Azure Monitor and Google Cloud Monitoring.
For more information on supported data sources, see [Data sources and Grafana alerting](v12-alerting-overview-datasources.md).
The process of executing a query involves defining the data source, specifying the desired data to retrieve, and applying relevant filters or transformations. Query languages or syntaxes specific to the chosen data source are utilized for constructing these queries.
In Alerting, you define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
An alert rule consists of one or more queries and expressions that select the data you want to measure.
For more information on queries and expressions, see [Query and transform data](v12-panels-query-xform.md).
## Data source queries
Queries in Grafana can be applied in various ways, depending on the data source and query language being used. Each data source’s query editor provides a customized user interface that helps you write queries that take advantage of its unique capabilities.
Because of the differences between query languages, each data source query editor looks and functions differently. Depending on your data source, the query editor might provide auto-completion features, metric names, variable suggestions, or a visual query-building interface.
Some common types of query components include:
**Metrics or data fields** – Specify the specific metrics or data fields you want to retrieve, such as CPU usage, network traffic, or sensor readings.
**Time range** – Define the time range for which you want to fetch data, such as the last hour, a specific day, or a custom time range.
**Filters** – Apply filters to narrow down the data based on specific criteria, such as filtering data by a specific tag, host, or application.
**Aggregations** – Perform aggregations on the data to calculate metrics like averages, sums, or counts over a given time period.
**Grouping** – Group the data by specific dimensions or tags to create aggregated views or breakdowns.
**Note**
Grafana does not support alert queries with template variables. More information is available [here](https://community.grafana.com/t/template-variables-are-not-supported-in-alert-queries-while-setting-up-alert/2514) in the Grafana Labs forums.
## Expression queries
In Grafana, an expression is used to perform calculations, transformations, or aggregations on the data source queried data. It allows you to create custom metrics or modify existing metrics based on mathematical operations, functions, or logical expressions.
By leveraging expression queries, users can perform tasks such as calculating the percentage change between two values, applying functions like logarithmic or trigonometric functions, aggregating data over specific time ranges or dimensions, and implementing conditional logic to handle different scenarios.
In Alerting, you can only use expressions for Grafana-managed alert rules. For each expression, you can choose from the math, reduce, and resample expressions. These are called multi-dimensional rules, because they generate a separate alert for each series.
You can also use a classic condition, which creates an alert rule that triggers a single alert when its condition is met. As a result, Grafana sends only a single alert even when alert conditions are met for multiple series.
**Note**
Classic conditions exist mainly for compatibility reasons and should be avoided if possible.
**Reduce**
Aggregates time series values in the selected time range into a single value.
**Math**
Performs free-form math functions/operations on time series and number data. Can be used to preprocess time series data or to define an alert condition for number data.
**Resample**
Realigns a time range to a new set of timestamps, this is useful when comparing time series data from different data sources where the timestamps would otherwise not align.
**Threshold**
Checks if any time series data matches the threshold condition.
The threshold expression allows you to compare two single values. It returns `0` when the condition is false and `1` if the condition is true. The following threshold functions are available:
+ Is above (x > y)
+ Is below (x < y)
+ Is within range (x > y1 AND x < y2)
+ Is outside range (x < y1 AND x > y2)
**Classic condition**
Checks if any time series data matches the alert condition.
**Note**
Classic condition expression queries always produce one alert instance only, no matter how many time series meet the condition. Classic conditions exist mainly for compatibility reasons and should be avoided if possible.
## Aggregations
Grafana Alerting provides the following aggregation functions to enable you to further refine your query.
These functions are available for **Reduce** and **Classic condition** expressions only.
| Function | Expression | What it does |
| --- | --- | --- |
| avg | Reduce / Classic | Displays the average of the values |
| min | Reduce / Classic | Displays the lowest value |
| max | Reduce / Classic | Displays the highest value |
| sum | Reduce / Classic | Displays the sum of all values |
| count | Reduce / Classic | Counts the number of values in the result |
| last | Reduce / Classic | Displays the last value |
| median | Reduce / Classic | Displays the median value |
| diff | Classic | Displays the difference between the newest and oldest value |
| diff\$1abs | Classic | Displays the absolute value of diff |
| percent\$1diff | Classic | Displays the percentage value of the difference between newest and oldest value |
| percent\$1diff\$1abs | Classic | Displays the absolute value of percent\$1diff |
| count\$1non\$1null | Classic | Displays a count of values in the result set that aren’t null |
## Alert condition
An alert condition is the query or expression that determines whether the alert will fire or not depending on the value it yields. There can be only one condition which will determine the triggering of the alert.
After you have defined your queries and/or expressions, choose one of them as the alert rule condition.
When the queried data satisfies the defined condition, Grafana triggers the associated alert, which can be configured to send notifications through various channels like email, Slack, or PagerDuty. The notifications inform you about the condition being met, allowing you to take appropriate actions or investigate the underlying issue.
By default, the last expression added is used as the alert condition.
## Recovery threshold
To reduce the noise of flapping alerts, you can set a recovery threshold different to the alert threshold.
Flapping alerts occur when a metric hovers around the alert threshold condition and may lead to frequent state changes, resulting in too many notifications being generated.
Grafana-managed alert rules are evaluated for a specific interval of time. During each evaluation, the result of the query is checked against the threshold set in the alert rule. If the value of a metric is above the threshold, an alert rule fires and a notification is sent. When the value goes below the threshold and there is an active alert for this metric, the alert is resolved, and another notification is sent.
It can be tricky to create an alert rule for a noisy metric. That is, when the value of a metric continually goes above and below a threshold. This is called flapping and results in a series of firing - resolved - firing notifications and a noisy alert state history.
For example, if you have an alert for latency with a threshold of 1000ms and the number fluctuates around 1000 (say 980 ->1010 -> 990 -> 1020, and so on) then each of those will trigger a notification.
To solve this problem, you can set a (custom) recovery threshold, which basically means having two thresholds instead of one. An alert is triggered when the first threshold is crossed and is resolved only when the second threshold is crossed.
For example, you could set a threshold of 1000ms and a recovery threshold of 900ms. This way, an alert rule will only stop firing when it goes under 900ms and flapping is reduced.
# Alert instances
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Grafana managed alerts support multi-dimensional alerting. Each alert rule can create multiple alert instances. This is powerful if you are observing multiple series in a single expression.
Consider the following PromQL expression:
```
sum by(cpu) (
rate(node_cpu_seconds_total{mode!="idle"}[1m])
)
```
A rule using this expression will create as many alert instances as the amount of CPUs we are observing after the first evaluation, allowing a single rule to report the status of each CPU.
# Namespaces, folders and groups
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Alerts can be organized using folders for Grafana managed rules and namespaces for Mimir, Loki, or Prometheus rules and group names.
**Namespaces and folders**
When creating Grafana-managed rules, the folder can be used to perform access control and grant or deny access to all rules within a specific folder.
A namespace contains one or more groups. The rules within a group are run sequentially at a regular interval. The default interval is one minute. You can rename Grafana Mimi or Loki rule namespaces and groups, and edit group evaluation intervals.
**Groups**
The rules within a group are run sequentially at a regular interval, meaning no rules will be evaluated at the same time, and in order of appearance. The default interval is one minute. You can rename Grafana Mimir or Loki rule namespaces or Loki rule namespaces and groups, and edit group evaluation intervals.
**Tip**
If you want rules to be evaluated concurrently and with different intervals, consider storing them in different groups.
**Note**
Grafana managed alert rules are evaluated concurrently instead of sequentially.
# Alert rule evaluation
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.
## Evaluation group
Every alert rule is part of an evaluation group. Each evaluation group contains an evaluation interval that determines how frequently the alert rule is checked.
**Data-source managed** alert rules within the same group are evaluated one after the other, while alert rules in different groups can be evaluated simultaneously. This feature is especially useful when you want to ensure that recording rules are evaluated before any alert rules.
**Grafana-managed** alert rules are evaluated at the same time, regardless of alert rule group. The default evaluation interval is set at 10 seconds, which means that Grafana-managed alert rules are evaluated every 10 seconds to the closest 10-second window on the clock, for example, 10:00:00, 10:00:10, 10:00:20, and so on. You can also configure your own evaluation interval, if required.
**Note**
Evaluation groups and alerts grouping in notification policies are two separate things. Grouping in notification policies allows multiple alerts sharing the same labels to be sent in the same time message.
## Pending period
By setting a pending period, you can avoid unnecessary alerts for temporary problems.
In the pending period, you select the period in which an alert rule can be in breach of the condition until it fires.
**Example**
Imagine you have an alert rule evaluation interval set at every 30 seconds and the pending period to 90 seconds.
Evaluation will occur as follows:
[00:30] First evaluation - condition not met.
[01:00] Second evaluation - condition breached. Pending counter starts.**Alert starts pending.**
[01:30] Third evaluation - condition breached. Pending counter = 30s. **Pending state.**
[02:00] Fourth evaluation - condition breached. Pending counter = 60s **Pending state.**
[02:30] Fifth evaluation - condition breached. Pending counter = 90s. **Alert starts firing**
If the alert rule has a condition that needs to be in breach for a certain amount of time before it takes action, then its state changes as follows:
+ When the condition is first breached, the rule goes into a "pending" state.
+ The rule stays in the "pending" state until the condition has been broken for the required amount of time - pending period.
+ Once the required time has passed, the rule goes into a "firing" state.
+ If the condition is no longer broken during the pending period, the rule goes back to its normal state.
**Note**
If you want to skip the pending state, you can simply set the pending period to 0. This effectively skips the pending period and your alert rule will start firing as soon as the condition is breached.
When an alert rule fires, alert instances are produced, which are then sent to the Alertmanager.
# State and health of alerting rules
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
The state and health of alerting rules help you understand several key status indicators about your alerts.
There are three key components: *alert rule state*, *alert instance state*, and *alert rule health*. Although related, each component conveys subtly different information.
**Alert rule state**
An alert rule can be in one of the following states:
| State | Description |
| --- | --- |
| Normal | None of the time series returned by the evaluation engine is in a `Pending` or `Firing` state. |
| Pending | At least one time series returned by the evaluation engine is `Pending`. |
| Firing | At least one time series returned by the evaluation engine is `Firing`. |
| Recovering | The alert condition is no longer firing but has not yet returned to normal. |
**Note**
Alerts will transition first to `pending` and then `firing`, thus it will take at least two evaluation cycles before an alert is fired.
**Alert instance state**
An alert instance can be in one of the following states:
| State | Description |
| --- | --- |
| Normal | The state of an alert that is neither firing nor pending, everything is working correctly. |
| Pending | The state of an alert that has been active for less than the configured threshold duration. |
| Alerting | The state of an alert that has been active for longer than the configured threshold duration. |
| Recovering | The state of an alert that was previously firing but the alert condition is no longer met. The alert has not yet returned to normal. |
| NoData | No data has been received for the configured time window. |
| Error | The error that occurred when attempting to evaluate an alerting rule. |
**Keep last state**
An alert rule can be configured to keep the last state when `NoData` or `Error` state is encountered. This will both prevent alerts from firing, and from resolving and re-firing. Just like normal evaluation, the alert rule will transition from `Pending` to `Firing` after the pending period has elapsed.
**Alert rule health**
An alert rule can have one the following health statuses:
| State | Description |
| --- | --- |
| Ok | No error when evaluating an alerting rule. |
| Error | An error occurred when evaluating an alerting rule. |
| NoData | The absence of data in at least one time series returned during a rule evaluation. |
**Special alerts for `NoData` and `Error`**
When evaluation of an alerting rule produces state `NoData` or `Error`, Grafana Alerting will generate alert instances that have the following additional labels:
| Label | Description |
| --- | --- |
| alertname | Either `DatasourceNoData` or `DatasourceError` depending on the state. |
| datasource\$1uid | The UID of the data source that caused the state. |
You can handle these alerts the same way as regular alerts by adding a silence, route to a contact point, and so on.
# Notification templating
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Notifications sent via contact points are built using notification templates. Grafana’s default templates are based on the [Go templating system](https://golang.org/pkg/text/template) where some fields are evaluated as text, while others are evaluated as HTML (which can affect escaping).
The default template [default\$1template.go](https://github.com/grafana/alerting/blob/main/templates/default_template.go) is a useful reference for custom templates.
Since most of the contact point fields can be templated, you can create reusable custom templates and use them in multiple contact points. To learn about custom notifications using templates, see [Customize notifications](v12-alerting-manage-notifications.md).
**Nested templates**
You can embed templates within other templates.
For example, you can define a template fragment using the `define` keyword.
```
{{ define "mytemplate" }}
{{ len .Alerts.Firing }} firing. {{ len .Alerts.Resolved }} resolved.
{{ end }}
```
You can then embed custom templates within this fragment using the `template` keyword. For example:
```
Alert summary:
{{ template "mytemplate" . }}
```
You can use any of the following built-in template options to embed custom templates.
| Name | Notes |
| --- | --- |
| `default.title` | Displays high-level status information. |
| `default.message` | Provides a formatted summary of firing and resolved alerts. |
| `teams.default.message` | Similar to `default.messsage`, formatted for Microsoft Teams. |
**HTML in notification templates**
HTML in alerting notification templates is escaped. We do not support rendering of HTML in the resulting notification.
Some notifiers support alternative methods of changing the look and feel of the resulting notification. For example, Grafana installs the base template for alerting emails to `/public/emails/ng_alert_notification.html`. You can edit this file to change the appearance of all alerting emails.
# Alertmanager
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Alertmanager enables you to quickly and efficiently manage and respond to alerts. It receives alerts, handles mutings, inhibition, grouping, and routing by sending notifications out via your channel of choice, for example, email or Slack.
In Grafana, you can use the Grafana Alertmanager or an external Alertmanager. You can also run multiple alertmanagers; your decision depends on your set up and where your alerts are being generated.
**Grafana Alertmanager**
Grafana Alertmanager is an internal Alertmanager that is pre-configured and available for selection by default.
The Grafana Alertmanager can receive alerts from Grafana, but it cannot receive alerts from outside Grafana, for example, from Mimir or Loki.
**Note**
Inhibition rules are not supported in the Grafana Alertmanager.
**External Alertmanager**
If you want to use a single alertmanager to receive all your Grafana, Loki, Mimir, and Prometheus alerts, you can set up Grafana to use an external Alertmanager. This external Alertmanager can be configured and administered from within Grafana itself.
Here are two examples of when you might want to configure your own external alertmanager and send your alerts there instead of the Grafana Alertmanager:
1. You already have alertmanagers on-premise in your own Cloud infrastructure that you have set up and still want to use, because you have other alert generators, such as Prometheus.
1. You want to use both Prometheus on-premise and hosted Grafana to send alerts to the same alertmanager that runs in your Cloud infrastructure.
Alertmanagers are visible from the dropdown menu on the Alerting Contact Points, and Notification Policies pages.
If you are provisioning your data source, set the flag `handleGrafanaManagedAlerts` in the `jsonData` field to `true` to send Grafana-managed alerts to this Alertmanager.
# Contact points
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Contact points contain the configuration for sending notifications. A contact point is a list of integrations, each of which sends a notification to a particular email address, service or URL. Contact points can have multiple integrations of the same kind, or a combination of integrations of different kinds. For example, a contact point could contain a Pagerduty integration; an Amazon SNS and Slack integration; or a Pagerduty integration, a Slack integration, and two Amazon SNS integrations. You can also configure a contact point with no integrations; in which case no notifications are sent.
A contact point cannot send notifications until it has been added to a notification policy. A notification policy can only send alerts to one contact point, but a contact point can be added to a number of notification policies at the same time. When an alert matches a notification policy, the alert is sent to the contact point in that notification policy, which then sends a notification to each integration in its configuration.
Contact points can be configured for the Grafana Alertmanager as well as external alertmanagers.
You can also use notification templating to customize notification messages for contact point types.
**Supported contact point types**
The following table lists the contact point types supported by Grafana.
| Name | Type |
| --- | --- |
| Amazon SNS | `sns` |
| OpsGenie | `opsgenie` |
| Pager Duty | `pagerduty` |
| Slack | `slack` |
| VictorOps | `victorops` |
For more information about contact points, see [Configure contact points](v12-alerting-configure-contactpoints.md) and [Customize notifications](v12-alerting-manage-notifications.md).
# Notifications
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Choosing how, when, and where to send your alert notifications is an important part of setting up your alerting system. These decisions will have a direct impact on your ability to resolve issues quickly and not miss anything important.
As a first step, define your [contact points](v12-alerting-explore-contacts.md), which define where to send your alert notifications. A contact point is a set of one or more integrations that are used to deliver notifications. Add notification templates to contact points for reuse and consistent messaging in your notifications.
Next, create a notification policy which is a set of rules for where, when and how your alerts are routed to contact points. In a notification policy, you define where to send your alert notifications by choosing one of the contact points you created.
## Alertmanagers
Grafana uses Alertmanagers to send notifications for firing and resolved alerts. Grafana has its own Alertmanager, referred to as **Grafana** in the user interface, but also supports sending notifications from other Alertmanagers too, such as the [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/). The Grafana Alertmanager uses notification policies and contact points to configure how and where a notification is sent; how often a notification should be sent; and whether alerts should all be sent in the same notification, sent in grouped notifications based on a set of labels, or as separate notifications.
## Notification policies
Notification policies control when and where notifications are sent. A notification policy can choose to send all alerts together in the same notification, send alerts in grouped notifications based on a set of labels, or send alerts as separate notifications. You can configure each notification policy to control how often notifications should be sent as well as having one or more mute timings to inhibit notifications at certain times of the day and on certain days of the week.
Notification policies are organized in a tree structure where at the root of the tree there is a notification policy called the default policy. There can be only one default policy and the default policy cannot be deleted.
Specific routing policies are children of the root policy and can be used to match either all alerts or a subset of alerts based on a set of matching labels. A notification policy matches an alert when its matching labels match the labels in the alert.
A nested policy can have its own nested policies, which allow for additional matching of alerts. An example of a nested policy could be sending infrastructure alerts to the Ops team; while a child policy might send high priority alerts to Pagerduty and low priority alerts to Slack.
All alerts, irrespective of their labels, match the default policy. However, when the default policy receives an alert it looks at each nested policy and sends the alert to the first nested policy that matches the alert. If the nested policy has further nested policies, then it can attempt to match the alert against one of its nested policies. If no nested policies match the alert then the policy itself is the matching policy. If there are no nested policies, or no nested policies match the alert, then the default policy is the matching policy.
For more detailed information about notification policies, see [Notification policies](v12-alerting-explore-notifications-policies-details.md).
## Notification templates
You can customize notifications with templates. For example, templates can be used to change the title and message of notifications sent to Slack.
Templates are not limited to an individual integration or contact point, but instead can be used in a number of integrations in the same contact point and even integrations across different contact points. For example, a Grafana user can create a template called `custom_subject_or_title` and use it for both templating subjects in Pager Duty and titles of Slack messages without having to create two separate templates.
All notifications templates are written in [Go’s templating language](https://pkg.go.dev/text/template), and are in the Contact points tab on the Alerting page.
For more detailed information about customizing notifications, see [Customize notifications](v12-alerting-manage-notifications.md).
## Silences
You can use silences to mute notifications from one or more firing rules. Silences do not stop alerts from firing or being resolved, or hide firing alerts in the user interface. A silence lasts as long as its duration which can be configured in minutes, hours, days, months or years.
For more detailed information about using silences, see [Silencing alert notifications](v12-alerting-silences.md).
# Notification policies
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Notification policies provide you with a flexible way of routing alerts to various different receivers. Using label matchers, you can modify alert notification delivery without having to update every individual alert rule.
In this section, you will learn more about how notification policies work and are structured, so that you can make the most out of setting up your notification policies.
## Policy tree
Notification policies are *not* a list, but rather are structured according to a tree structure. This means that each policy can have child policies, and so on. The root of the notification policy tree is called the **Default notification policy**.
Each policy consists of a set of label matchers (0 or more) that specify which labels they are or aren’t interested in handling.
For more information about label matching, see [How label matching works](v12-alerting-overview-labels-matching.md).
**Note**
If you haven’t configured any label matchers for your notification policy, your notification policy will match *all* alert instances. This may prevent child policies from being evaluated unless you have enabled **Continue matching siblings** on the notification policy.
## Routing
To determine which notification policy will handle which alert instances, you have to start by looking at the existing set of notification policies, starting with the default notification policy.
If no policies other than the default policy are configured, the default policy will handle the alert instance.
If policies other than the default policy are defined, it will evaluate those notification policies in the order they are displayed.
If a notification policy has label matchers that match the labels of the alert instance, it will descend in to its child policies and, if there are any, will continue to look for any child policies that might have label matchers that further narrow down the set of labels, and so forth until no more child policies have been found.
If no child policies are defined in a notification policy or if none of the child policies have any label matchers that match the alert instance’s labels, the parent notification policy is used.
As soon as a matching policy is found, the system does not continue to look for other matching policies. If you want to continue to look for other policies that may match, enable **Continue matching siblings** on that particular policy.
Lastly, if none of the notification policies are selected the default notification policy is used.
### Routing example
Here is an example of a relatively simple notification policy tree and some alert instances.
![\[An image showing a set of notification policies in a tree structure, and a set of alert instances with different labels to match to the policies.\]](http://docs.aws.amazon.com/grafana/latest/userguide/images/notification-routing.png)
Here’s a breakdown of how these policies are selected:
**Pod stuck in CrashLoop** does not have a `severity` label, so none of its child policies are matched. It does have a `team=operations` label, so the first policy is matched.
The `team=security` policy is not evaluated since we already found a match and **Continue matching siblings** was not configured for that policy.
**Disk Usage – 80%** has both a `team` and `severity` label, and matches a child policy of the operations team.
**Unauthorized log entry** has a `team` label but does not match the first policy (`team=operations`) since the values are not the same, so it will continue searching and match the `team=security` policy. It does not have any child policies, so the additional `severity=high` label is ignored.
## Inheritance
In addition to child policies being a useful concept for routing alert instances, they also inherit properties from their parent policy. This also applies to any policies that are child policies of the default notification policy.
The following properties are inherited by child policies:
+ Contact point
+ Grouping options
+ Timing options
+ Mute timings
Each of these properties can be overwritten by an individual policy should you wish to override the inherited properties.
To inherit a contact point from the parent policy, leave it blank. To override the inherited grouping options, enable **Override grouping**. To override the inherited timing options, enable **Override general timings**.
### Inheritance example
The example below shows how the notification policy tree from our previous example allows the child policies of the `team=operations` to inherit its contact point.
In this way, we can avoid having to specify the same contact point multiple times for each child policy.
![\[An image showing a set of notification policies in a tree structure, with contact points assigned to some of the policies, but with some child policies inheriting the contact points of their parents, rather than defining their own.\]](http://docs.aws.amazon.com/grafana/latest/userguide/images/notification-inheritance.png)
## Additional configuration options
### Grouping
Grouping is an important feature of Grafana Alerting as it allows you to batch relevant alerts together into a smaller number of notifications. This is particularly important if notifications are delivered to first-responders, such as engineers on-call, where receiving lots of notifications in a short period of time can be overwhelming and in some cases can negatively impact a first-responders ability to respond to an incident. For example, consider a large outage where many of your systems are down. In this case, grouping can be the difference between receiving 1 phone call and 100 phone calls.
You choose how alerts are grouped together using the Group by option in a notification policy. By default, notification policies in Grafana group alerts together by alert rule using the `alertname` and `grafana_folder` labels (since alert names are not unique across multiple folders). Should you wish to group alerts by something other than the alert rule, change the grouping to any other combination of labels.
#### Disable grouping
Should you wish to receive every alert as a separate notification, you can do so by grouping by a special label called `...`. This is useful when your alerts are being delivered to an automated system instead of a first-responder.
#### A single group for all alerts
Should you wish to receive all alerts together in a single notification, you can do so by leaving Group by empty.
### Timing options
The timing options decide how often notifications are sent for each group of alerts. There are three timers that you need to know about: Group wait, Group interval, and Repeat interval.
#### Group wait
Group wait is the amount of time Grafana waits before sending the first notification for a new group of alerts. The longer Group wait is the more time you have for other alerts to arrive. The shorter Group wait is the earlier the first notification will be sent, but at the risk of sending incomplete notifications. You should always choose a Group wait that makes the most sense for your use case.
**Default** 30 seconds
#### Group interval
Once the first notification has been sent for a new group of alerts, Grafana starts the Group interval timer. This is the amount of time Grafana waits before sending notifications about changes to the group. For example, another firing alert might have just been added to the group while an existing alert might have resolved. If an alert was too late to be included in the first notification due to Group wait, it will be included in subsequent notifications after Group interval. Once Group interval has elapsed, Grafana resets the Group interval timer. This repeats until there are no more alerts in the group after which the group is deleted.
**Default** 5 minutes
#### Repeat interval
Repeat interval decides how often notifications are repeated if the group has not changed since the last notification. You can think of these as reminders that some alerts are still firing. Repeat interval is closely related to Group interval, which means your Repeat interval must not only be greater than or equal to Group interval, but also must be a multiple of Group interval. If Repeat interval is not a multiple of Group interval it will be coerced into one. For example, if your Group interval is 5 minutes, and your Repeat interval is 9 minutes, the Repeat interval will be rounded up to the nearest multiple of 5 which is 10 minutes.
**Default** 4 hours
# Alerting high availability
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Amazon Managed Grafana is configured for high availability, including running multiple instances across multiple availability zones for each workspace that you create.
Grafana Alerting uses the Prometheus model of separating the evaluation of alert rules from the delivering of notifications. In this model the evaluation of alert rules is done in the alert generator and the delivering of notifications is done in the alert receiver. In Grafana Alerting, the alert generator is the Scheduler and the receiver is the Alertmanager.
With high availability configurations, all alert rules are evaluated on all instances. You can think of the evaluation of alert rules as being duplicated. This is how Grafana Alerting makes sure that as long as at least one Grafana instance is working, alert rules will still be evaluated and notifications for alerts will still be sent. You will see this duplication in state history, and is a good way to tell if you are using high availability.