# Labels and annotations
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Labels and annotations contain information about an alert. Both labels and annotations have the same structure: a set of named values; however their intended uses are different. An example of label, or the equivalent annotation, might be `alertname="test"`.
The main difference between a label and an annotation is that labels are used to differentiate an alert from all other alerts, while annotations are used to add additional information to an existing alert.
For example, consider two high CPU alerts: one for `server1` and another for `server2`. In such an example, we might have a label called `server` where the first alert has the label `server="server1"` and the second alert has the label `server="server2"`. However, we might also want to add a description to each alert such as `"The CPU usage for server1 is above 75%."`, where `server1` and `75%` are replaced with the name and CPU usage of the server (please refer to the documentation on [Templating labels and annotations](v12-alerting-overview-labels-templating.md) for how to do this). This kind of description would be more suitable as an annotation.
## Labels
Labels contain information that identifies an alert. An example of a label might be `server=server1`. Each alert can have more than one label, and the complete set of labels for an alert is called its label set. It is this label set that identifies the alert.
For example, an alert might have the label set `{alertname="High CPU usage",server="server1"}` while another alert might have the label set `{alertname="High CPU usage",server="server2"}`. These are two separate alerts because although their `alertname` labels are the same, their `server` labels are different.
The label set for an alert is a combination of the labels from the datasource, custom labels from the alert rule, and a number of reserved labels such as `alertname`.
**Custom Labels**
Custom labels are additional labels from the alert rule. Like annotations, custom labels must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired. Documentation on how to template custom labels can be found [here](v12-alerting-overview-labels-templating.md).
When using custom labels with templates, it is important to make sure that the label value does not change between consecutive evaluations of the alert rule as this will end up creating large numbers of distinct alerts. However, it is OK for the template to produce different label values for different alerts. For example, do not put the value of the query in a custom label as this will end up creating a new set of alerts each time the value changes. Instead use annotations.
It is also important to make sure that the label set for an alert does not have two or more labels with the same name. If a custom label has the same name as a label from the datasource then it will replace that label. However, should a custom label have the same name as a reserved label then the custom label will be omitted from the alert.
## Annotations
Annotations are named pairs that add additional information to existing alerts. There are a number of suggested annotations in Grafana such as `description`, `summary`, `runbook_url`, `dashboardUId` and `panelId`. Like custom labels, annotations must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired. If an annotation contains template code, the template is evaluated once when the alert is fired. It is not re-evaluated, even when the alert is resolved. Documentation on how to template annotations can be found [here](v12-alerting-overview-labels-templating.md).
**Topics**
+ [Labels](#v12-alerting-overview-labels-labels)
+ [Annotations](#v12-alerting-overview-labels-annotations)
+ [How label matching works](v12-alerting-overview-labels-matching.md)
+ [Labels in Grafana Alerting](v12-alerting-overview-labels-alerting.md)
+ [Templating labels and annotations](v12-alerting-overview-labels-templating.md)
# How label matching works
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Use labels and label matchers to link alert rules to notification policies and silences. This allows for a very flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
A label matcher consists of 3 distinct parts, the **label**, the **value** and the **operator**.
+ The **Label** field is the name of the label to match. It must exactly match the label name.
+ The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Operator** value.
+ The **Operator** field is the operator to match against the label value. The available operators are:
| Operator | Description |
| --- | --- |
| `=` | Select labels that are exactly equal to the value. |
| `!=` | Select labels that are not equal to the value. |
| `=~` | Select labels that regex-match the value. |
| `!~` | Select labels that do not regex-match the value. |
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
## Example
If you define the following set of labels for your alert:
```
{ foo=bar, baz=qux, id=12 }
```
then:
+ A label matcher defined as `foo=bar` matches this alert rule.
+ A label matcher defined as `foo!=bar` does *not* match this alert rule.
+ A label matcher defined as `id=~[0-9]+` matches this alert rule.
+ A label matcher defined as `baz!~[0-9]+` matches this alert rule.
+ Two label matchers defined as `foo=bar` and `id=~[0-9]+` match this alert rule.
## Exclude labels
You can also write label matchers to exclude labels.
Here is an example that shows how to exclude the label `team`. You can choose between any of these values to exclude the label.
+ `team=""`
+ `team!~.+`
+ `team=~^$`
# Labels in Grafana Alerting
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
This topic explains why labels are a fundamental component of alerting.
+ The complete set of labels for an alert is what uniquely identifies an alert within Grafana alerts.
+ The Alertmanager uses labels to match alerts for silences and alert groups in notification policies.
+ The alerting UI shows labels for every alert instance generated during evaluation of that rule.
+ Contact points can access labels to dynamically generate notifications that contain information specific to the alert that is resulting in a notification.
+ You can add labels to an [alerting rule](v12-alerting-configure.md). Labels are manually configurable, use template functions, and can reference other labels. Labels added to an alerting rule take precedence in the event of a collision between labels (except in the case of Grafana reserved labels, see below for more information).
## External Alertmanager compatibility
Grafana’s built-in Alertmanager supports both Unicode label keys and values. If you are using an external Prometheus Alertmanager, label keys must be compatible with their [data model](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). This means that label keys must only contain **ASCII letters**, **numbers**, as well as **underscores** and match the regex `[a-zA-Z_][a-zA-Z0-9_]*`. Any invalid characters will be removed or replaced by the Grafana alerting engine before being sent to the external Alertmanager according to the following rules:
+ `Whitespace` will be removed.
+ `ASCII characters` will be replaced with `_`.
+ `All other characters` will be replaced with their lower-case hex representation. If this is the first character it will be prefixed with `_`.
**Note**
If multiple label keys are sanitized to the same value, the duplicates will have a short hash of the original label appended as a suffix.
## Grafana reserved labels
**Note**
Labels prefixed with `grafana_` are reserved by Grafana for special use. If a manually configured label is added beginning with `grafana_` it will be overwritten in case of collision.
Grafana reserved labels can be used in the same way as manually configured labels. The current list of available reserved labels are:
| Label | Description |
| --- | --- |
| grafana\$1folder | Title of the folder containing the alert. |
# Templating labels and annotations
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
You can use templates to include data from queries and expressions in labels and annotations. For example, you might want to set the severity label for an alert based on the value of the query, or use the instance label from the query in a summary annotation so you know which server is experiencing high CPU usage.
All templates should be written in [text/template](https://pkg.go.dev/text/template). Regardless of whether you are templating a label or an annotation, you should write each template inline inside the label or annotation that you are templating. This means you cannot share templates between labels and annotations, and instead you will need to copy templates wherever you want to use them.
Each template is evaluated whenever the alert rule is evaluated, and is evaluated for every alert separately. For example, if your alert rule has a templated summary annotation, and the alert rule has 10 firing alerts, then the template will be executed 10 times, once for each alert. You should try to avoid doing expensive computations in your templates as much as possible.
## Examples
Rather than write a complete tutorial on text/template, the following examples attempt to show the most common use-cases we have seen for templates. You can use these examples verbatim, or adapt them as necessary for your use case. For more information about how to write text/template see the [text/template](https://pkg.go.dev/text/template) documentation.
**Print all labels, comma separated**
To print all labels, comma separated, print the `$labels` variable:
```
{{ $labels }}
```
For example, given an alert with the labels `alertname=High CPU usage`, `grafana_folder=CPU alerts` and `instance=server1`, this would print:
```
alertname=High CPU usage, grafana_folder=CPU alerts, instance=server1
```
**Note**
If you are using classic conditions then `$labels` will not contain any labels from the query. Refer to [the \$1labels variable](#v12-alerting-overview-labels-templating-the-labels-variable) for more information.
**Print all labels, one per line**
To print all labels, one per line, use a `range` to iterate over each key/value pair and print them individually. Here `$k` refers to the name and `$v` refers to the value of the current label:
```
{{ range $k, $v := $labels -}}
{{ $k }}={{ $v }}
{{ end }}
```
For example, given an alert with the labels `alertname=High CPU usage`, `grafana_folder=CPU alerts` and `instance=server1`, this would print:
```
alertname=High CPU usage
grafana_folder=CPU alerts
instance=server1
```
**Note**
If you are using classic conditions then `$labels` will not contain any labels from the query. Refer to [the \$1labels variable](#v12-alerting-overview-labels-templating-the-labels-variable) for more information.
**Print an individual label**
To print an individual label use the `index` function with the `$labels` variable:
```
The host {{ index $labels "instance" }} has exceeded 80% CPU usage for the last 5 minutes
```
For example, given an alert with the label `instance=server1`, this would print:
```
The host server1 has exceeded 80% CPU usage for the last 5 minutes
```
**Note**
If you are using classic conditions then `$labels` will not contain any labels from the query. Refer to [the \$1labels variable](#v12-alerting-overview-labels-templating-the-labels-variable) for more information.
**Print the value of a query**
To print the value of an instant query you can print its Ref ID using the `index` function and the `$values` variable:
```
{{ index $values "A" }}
```
For example, given an instant query that returns the value 81.2345, this will print:
```
81.2345
```
To print the value of a range query you must first reduce it from a time series to an instant vector with a reduce expression. You can then print the result of the reduce expression by using its Ref ID instead. For example, if the reduce expression takes the average of A and has the Ref ID B you would write:
```
{{ index $values "B" }}
```
**Print the humanized value of a query**
To print the humanized value of an instant query use the `humanize` function:
```
{{ humanize (index $values "A").Value }}
```
For example, given an instant query that returns the value 81.2345, this will print:
```
81.234
```
To print the humanized value of a range query you must first reduce it from a time series to an instant vector with a reduce expression. You can then print the result of the reduce expression by using its Ref ID instead. For example, if the reduce expression takes the average of A and has the Ref ID B you would write:
```
{{ humanize (index $values "B").Value }}
```
**Print the value of a query as a percentage**
To print the value of an instant query as a percentage use the `humanizePercentage` function:
```
{{ humanizePercentage (index $values "A").Value }}
```
This function expects the value to be a decimal number between 0 and 1. If the value is instead a decimal number between 0 and 100 you can divide it by 100 either in your query or using a math expression. If the query is a range query you must first reduce it from a time series to an instant vector with a reduce expression.
**Set a severity from the value of a query**
To set a severity label from the value of a query use an if statement and the greater than comparison function. Make sure to use decimals (`80.0`, `50.0`, `0.0`, etc) when doing comparisons against `$values` as text/template does not support type coercion. You can find a list of all the supported comparison functions [here](https://pkg.go.dev/text/template#hdr-Functions).
```
{{ if (gt $values.A.Value 80.0) -}}
high
{{ else if (gt $values.A.Value 50.0) -}}
medium
{{ else -}}
low
{{- end }}
```
**Print all labels from a classic condition**
You cannot use `$labels` to print labels from the query if you are using classic conditions, and must use `$values` instead. The reason for this is classic conditions discard these labels to enforce uni-dimensional behavior (at most one alert per alert rule). If classic conditions didn’t discard these labels, then queries that returned many time series would cause alerts to flap between firing and resolved constantly as the labels would change every time the alert rule was evaluated.
Instead, the `$values` variable contains the reduced values of all time series for all conditions that are firing. For example, if you have an alert rule with a query A that returns two time series, and a classic condition B with two conditions, then `$values` would contain `B0`, `B1`, `B2` and `B3`. If the classic condition B had just one condition, then `$values` would contain just `B0` and `B1`.
To print all labels of all firing time series use the following template (make sure to replace `B` in the regular expression with the Ref ID of the classic condition if it’s different):
```
{{ range $k, $v := $values -}}
{{ if (match "B[0-9]+" $k) -}}
{{ $k }}: {{ $v.Labels }}{{ end }}
{{ end }}
```
For example, a classic condition for two time series exceeding a single condition would print:
```
B0: instance=server1
B1: instance=server2
```
If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then its labels will be duplicated for each condition that is exceeded:
```
B0: instance=server1
B1: instance=server2
B2: instance=server1
B3: instance=server2
```
If you need to print unique labels you should consider changing your alert rules from uni-dimensional to multi-dimensional instead. You can do this by replacing your classic condition with reduce and math expressions.
**Print all values from a classic condition**
To print all values from a classic condition take the previous example and replace `$v.Labels` with `$v.Value`:
```
{{ range $k, $v := $values -}}
{{ if (match "B[0-9]+" $k) -}}
{{ $k }}: {{ $v.Value }}{{ end }}
{{ end }}
```
For example, a classic condition for two time series exceeding a single condition would print:
```
B0: 81.2345
B1: 84.5678
```
If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then `$values` will contain the values of all conditions:
```
B0: 81.2345
B1: 92.3456
B2: 84.5678
B3: 95.6789
```
## Variables
The following variables are available to you when templating labels and annotations:
### The labels variable
The `$labels` variable contains all labels from the query. For example, suppose you have a query that returns CPU usage for all of your servers, and you have an alert rule that fires when any of your servers have exceeded 80% CPU usage for the last 5 minutes. You want to add a summary annotation to the alert that tells you which server is experiencing high CPU usage. With the `$labels` variable you can write a template that prints a human-readable sentence such as:
```
CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes
```
**Note**
If you are using a classic condition then `$labels` will not contain any labels from the query. Classic conditions discard these labels in order to enforce uni-dimensional behavior (at most one alert per alert rule). If you want to use labels from the query in your template then follow the previous *Print all labels from a classic condition* example.
### The value variable
The `$value` variable is a string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule. It does not contain the results of range queries, as these can return anywhere from 10s to 10,000s of rows or metrics. If it did, for especially large queries a single alert could use 10s of MBs of memory and Grafana would run out of memory very quickly.
To print the `$value` variable in the summary you would write something like this:
```
CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ $value }}
```
And it would look something like this:
```
CPU usage for instance1 has exceeded 80% for the last 5 minutes: [ var='A' labels={instance=instance1} value=81.234 ]
```
Here `var='A'` refers to the instant query with Ref ID A, `labels={instance=instance1}` refers to the labels, and `value=81.234` refers to the average CPU usage over the last 5 minutes.
If you want to print just some of the string instead of the full string then use the `$values` variable. It contains the same information as `$value`, but in a structured table, and is much easier to use then writing a regular expression to match just the text you want.
### The values variable
The `$values` variable is a table containing the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs.
To print the value of the instant query with Ref ID A:
```
CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ index $values "A" }}
```
For example, given an alert with the labels `instance=server1` and an instant query with the value `81.2345`, this would print:
```
CPU usage for instance1 has exceeded 80% for the last 5 minutes: 81.2345
```
If the query in Ref ID A is a range query rather than an instant query then add a reduce expression with Ref ID B and replace `(index $values "A")` with `(index $values "B")`:
```
CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ index $values "B" }}
```
## Functions
The following functions are available to you when templating labels and annotations:
**args**
The `args` function translates a list of objects to a map with keys arg0, arg1 etc. This is intended to allow multiple arguments to be passed to templates.
```
{{define "x"}}{{.arg0}} {{.arg1}}{{end}}{{template "x" (args 1 "2")}}
```
```
1 2
```
**externalURL**
The `externalURL` function returns the external URL of the Grafana server.
```
{{ externalURL }}
```
```
https://example.com/grafana
```
**graphLink**
The `graphLink` function returns the path to the graphical view in [Explore in Grafana version 12](v12-explore.md) for the given expression and data source.
```
{{ graphLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }}
```
```
/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":false,"range":true}]
```
**humanize**
The `humanize` function humanizes decimal numbers.
```
{{ humanize 1000.0 }}
```
```
1k
```
**humanize1024**
The `humanize1024` works similar to `humanize` but uses 1024 as the base rather than 1000.
```
{{ humanize1024 1024.0 }}
```
```
1ki
```
**humanizeDuration**
The `humanizeDuration` function humanizes a duration in seconds.
```
{{ humanizeDuration 60.0 }}
```
```
1m 0s
```
**humanizePercentage**
The `humanizePercentage` function humanizes a ratio value to a percentage.
```
{{ humanizePercentage 0.2 }}
```
```
20%
```
**humanizeTimestamp**
The `humanizeTimestamp` function humanizes a Unix timestamp.
```
{{ humanizeTimestamp 1577836800.0 }}
```
```
2020-01-01 00:00:00 +0000 UTC
```
**match**
The `match` function matches the text against a regular expression pattern.
```
{{ match "a.*" "abc" }}
```
```
true
```
**pathPrefix**
The `pathPrefix` function returns the path of the Grafana server.
```
{{ pathPrefix }}
```
```
/grafana
```
**tableLink**
The `tableLink` function returns the path to the tabular view in [Explore in Grafana version 12](v12-explore.md) for the given expression and data source.
```
{{ tableLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }}
```
```
/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":true,"range":false}]
```
**title**
The `title` function capitalizes the first character of each word.
```
{{ title "hello, world!" }}
```
```
Hello, World!
```
**toLower**
The `toLower` function returns all text in lowercase.
```
{{ toLower "Hello, world!" }}
```
```
hello, world!
```
**toUpper**
The `toUpper` function returns all text in uppercase.
```
{{ toUpper "Hello, world!" }}
```
```
HELLO, WORLD!
```
**reReplaceAll**
The `reReplaceAll` function replaces text matching the regular expression.
```
{{ reReplaceAll "localhost:(.*)" "example.com:$1" "localhost:8080" }}
```
```
example.com:8080
```