# Manage your alerts **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). Once you have set up your alert rules, contact points, and notification policies, you can use Grafana alerting to manage your alerts in practice. **Topics** + [Customize notifications](v12-alerting-manage-notifications.md) + [Manage contact points](v12-alerting-manage-contactpoints.md) + [Silencing alert notifications](v12-alerting-silences.md) + [View and filter alert rules](v12-alerting-manage-rules-viewfilter.md) + [Mute timings](v12-alerting-manage-muting.md) + [View the state and health of alert rules](v12-alerting-manage-rulestate.md) + [View and filter by alert groups](v12-alerting-manage-viewfiltergroups.md) + [View notification errors](v12-alerting-manage-viewnotificationerrors.md) # Customize notifications **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). Customize your notifications with notifications templates. You can use notification templates to change the title, message, and format of the message in your notifications. Notification templates are not tied to specific contact point integrations, such as Amazon SNS or Slack. However, you can choose to create separate notification templates for different contact point integrations. You can use notification templates to: + Add, remove, or re-order information in the notification including the summary, description, labels and annotations, values, and links + Format text in bold and italic, and add or remove line breaks You cannot use notification templates to: + Change the design of notifications in instant messaging services such as Slack and Microsoft Teams **Topics** + [Using Go’s templating language](v12-alerting-notifications-go-templating.md) + [Create notification templates](v12-alerting-create-templates.md) + [Using notification templates](#v12-alerting-use-notification-templates) + [Template reference](v12-alerting-template-reference.md) # Using Go’s templating language **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). You write notification templates in Go’s templating language, [text/template](https://pkg.go.dev/text/template). This section provides an overview of Go’s templating language and writing templates in text/template. ## Dot In text/template there is a special cursor called dot, and is written as `.`. You can think of this cursor as a variable whose value changes depending where in the template it is used. For example, at the start of a notification template `.` refers to the `ExtendedData` object, which contains a number of fields including `Alerts`, `Status`, `GroupLabels`, `CommonLabels`, `CommonAnnotations` and `ExternalURL`. However, dot might refer to something else when used in a `range` over a list, when used inside a `with`, or when writing feature templates to be used in other templates. You can see examples of this in [Create notification templates](v12-alerting-create-templates.md), and all data and functions in the [Template reference](v12-alerting-template-reference.md). ## Opening and closing tags In text/template, templates start with `{{` and end with `}}` irrespective of whether the template prints a variable or runs control structures such as if statements. This is different from other templating languages such as Jinja where printing a variable uses `{{` and `}}` and control structures use `{%` and `%}`. ## Print To print the value of something use `{{` and `}}`. You can print the value of dot, a field of dot, the result of a function, and the value of a [variable](#v12-go-variables). For example, to print the `Alerts` field where dot refers to `ExtendedData` you would write the following: ``` {{ .Alerts }} ``` ## Iterate over alerts To print just the labels of each alert, rather than all information about the alert, you can use a `range` to iterate the alerts in `ExtendedData`: ``` {{ range .Alerts }} {{ .Labels }} {{ end }} ``` Inside the range dot no longer refers to `ExtendedData`, but to an `Alert`. You can use `{{ .Labels }}` to print the labels of each alert. This works because `{{ range .Alerts }}` changes dot to refer to the current alert in the list of alerts. When the range is finished dot is reset to the value it had before the start of the range, which in this example is `ExtendedData`: ``` {{ range .Alerts }} {{ .Labels }} {{ end }} {{/* does not work, .Labels does not exist here */}} {{ .Labels }} {{/* works, cursor was reset */}} {{ .Status }} ``` ## Iterate over annotations and labels Let’s write a template to print the labels of each alert in the format `The name of the label is $name, and the value is $value`, where `$name` and `$value` contain the name and value of each label. Like in the previous example, use a range to iterate over the alerts in `.Alerts` such that dot refers to the current alert in the list of alerts, and then use a second range on the sorted labels so dot is updated a second time to refer to the current label. Inside the second range use `.Name` and `.Value` to print the name and value of each label: ``` {{ range .Alerts }} {{ range .Labels.SortedPairs }} The name of the label is {{ .Name }}, and the value is {{ .Value }} {{ end }} {{ range .Annotations.SortedPairs }} The name of the annotation is {{ .Name }}, and the value is {{ .Value }} {{ end }} {{ end }} ``` ## The index functions To print a specific annotation or label use the `index` function. ``` {{ range .Alerts }} The name of the alert is {{ index .Labels "alertname" }} {{ end }} ``` ## If statements You can use if statements in templates. For example, to print `There are no alerts` if there are no alerts in `.Alerts` you would write the following: ``` {{ if .Alerts }} There are alerts {{ else }} There are no alerts {{ end }} ``` ## With With is similar to if statements, however unlike if statements, `with` updates dot to refer to the value of the with: ``` {{ with .Alerts }} There are {{ len . }} alert(s) {{ else }} There are no alerts {{ end }} ``` ## Variables Variables in text/template must be created within the template. For example, to create a variable called `$variable` with the current value of dot you would write the following: ``` {{ $variable := . }} ``` You can use `$variable` inside a range or `with` and it will refer to the value of dot at the time the variable was defined, not the current value of dot. For example, you cannot write a template that use `{{ .Labels }}` in the second range because here dot refers to the current label, not the current alert: ``` {{ range .Alerts }} {{ range .Labels.SortedPairs }} {{ .Name }} = {{ .Value }} {{/* does not work because in the second range . is a label not an alert */}} There are {{ len .Labels }} {{ end }} {{ end }} ``` You can fix this by defining a variable called `$alert` in the first range and before the second range: ``` {{ range .Alerts }} {{ $alert := . }} {{ range .Labels.SortedPairs }} {{ .Name }} = {{ .Value }} {{/* works because $alert refers to the value of dot inside the first range */}} There are {{ len $alert.Labels }} {{ end }} {{ end }} ``` ## Range with index You can get the index of each alert within a range by defining index and value variables at the start of the range: ``` {{ $num_alerts := len .Alerts }} {{ range $index, $alert := .Alerts }} This is alert {{ $index }} out of {{ $num_alerts }} {{ end }} ``` ## Define templates You can define templates that can be used within other templates, using `define` and the name of the template in double quotes. You should not define templates with the same name as other templates, including default templates such as `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`. Where a template has been created with the same name as a default template, or a template in another notification template, Grafana might use either template. Grafana does not prevent, or show an error message, when there are two or more templates with the same name. ``` {{ define "print_labels" }} {{ end }} ``` ## Execute templates You can execute defined template within your template using `template`, the name of the template in double quotes, and the cursor that should be passed to the template: ``` {{ template "print_labels" . }} ``` ## Pass data to templates Within a template dot refers to the value that is passed to the template. For example, if a template is passed a list of firing alerts then dot refers to that list of firing alerts: ``` {{ template "print_alerts" .Alerts }} ``` If the template is passed the sorted labels for an alert then dot refers to the list of sorted labels: ``` {{ template "print_labels" .SortedLabels }} ``` This is useful when writing reusable templates. For example, to print all alerts you might write the following: ``` {{ template "print_alerts" .Alerts }} ``` Then to print just the firing alerts you could write this: ``` {{ template "print_alerts" .Alerts.Firing }} ``` This works because both `.Alerts` and `.Alerts.Firing` are lists of alerts. ``` {{ define "print_alerts" }} {{ range . }} {{ template "print_labels" .SortedLabels }} {{ end }} {{ end }} ``` ## Comments You can add comments with `{{/*` and `*/}}`: ``` {{/* This is a comment */}} ``` To prevent comments from adding line breaks use: ``` {{- /* This is a comment with no leading or trailing line breaks */ -}} ``` ## Indentation You can use indentation, both tabs and spaces, and line breaks, to make templates more readable: ``` {{ range .Alerts }} {{ range .Labels.SortedPairs }} {{ .Name }} = {{ .Value }} {{ end }} {{ end }} ``` However, indentation in the template will also be present in the text. Next we will see how to remove it. ## Remove spaces and line breaks In text/template use `{{-` and `-}}` to remove leading and trailing spaces and line breaks. For example, when using indentation and line breaks to make a template more readable: ``` {{ range .Alerts }} {{ range .Labels.SortedPairs }} {{ .Name }} = {{ .Value }} {{ end }} {{ end }} ``` The indentation and line breaks will also be present in the text: ``` alertname = "Test" grafana_folder = "Test alerts" ``` You can remove the indentation and line breaks from the text changing `}}` to `-}}` at the start of each range: ``` {{ range .Alerts -}} {{ range .Labels.SortedPairs -}} {{ .Name }} = {{ .Value }} {{ end }} {{ end }} ``` The indentation and line breaks in the template are now absent from the text: ``` alertname = "Test" grafana_folder = "Test alerts" ``` # Create notification templates **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). Create reusable notification templates to send to your contact points. You can add one or more templates to your notification template. Your notification template name must be unique. You cannot have two templates with the same name in the same notification template or in different notification templates. Avoid defining templates with the same name as default templates, such as: `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`. In the Contact points tab, you can see a list of your notification templates. ## Creating notification templates **To create a notification template** 1. Choose **Alerting**, **Contact points**. 1. Choose the **Notification Templates** tab, and then **\$1 Add notification template**. 1. Choose a name for the notification template, such as `email.subject`. 1. Write the content of the template in the content field. For example: ``` {{ if .Alerts.Firing -}} {{ len .Alerts.Firing }} firing alerts {{ end }} {{ if .Alerts.Resolved -}} {{ len .Alerts.Resolved }} resolved alerts {{ end }} ``` 1. Save your changes. `{{ define "email.subject" }}` (where `email.subject` is the name of your template) and `{{ end }}` is automatically added to the start and end of the content. **To create a notification template that contains more than one template** 1. Choose **Alerting**, **Contact points**. 1. Choose the **Notification Templates** tab, and then **\$1 Add notification template**. 1. Enter a name for the overall notification template. For example, `email`. 1. Write each template in the Content field, including `{{ define "name-of-template" }}` and `{{ end }}` at the start and end of each template. You can use descriptive names for each of the templates in the notification template, for example, `email.subject` or `email.message`. In this case, do not reuse the name of the notification template you entered above. Later sections show detailed examples for templates you might create. 1. Click Save. ## Preview notification templates Preview how your notification templates will look before using them in your contact points, helping you understand the result of the template you are creating as well as giving you a chance to fix any errors before saving the template. **Note** Notification previews are only available for Grafana Alertmanager. **To preview your notification templates** 1. Choose **Alerting**, **Contact points**. 1. Choose the **Notification Templates** tab, and then **\$1 Add notification template**, or edit an existing template. 1. Add or update your template content. Default data is provided and you can add or edit alert data to it as well as alert instances. You can add alert data directly in the Payload data window itself, or click **Select alert instances** or **Add custom alerts**. 1. [Optional] To add alert data from existing alert instances: 1. Choose **Select alert instances**. 1. Hover over the alert instances to view more information about each alert instance/ 1. Choose **Confirm** to add the alert instance to the payload. 1. [Optional] To add alert data using the Alert data editor, choose **Add custom data**: 1. Add annotations, custom labels, or set a dashboard or panel. 1. Toggle Firing or resolved, depending on whether you want to add firing or resolved alerts to your notification. 1. Choose **Add alert data**. 1. Choose **Refresh preview** to see what your template content will look like and the corresponding payload data. If there are any errors in your template, they are displayed in the Preview and you can correct them before saving. 1. Save your changes. ## Creating a template for the subject of message Create a template for the subject of an email that contains the number of firing and resolved alerts, as in this example: ``` 1 firing alerts, 0 resolved alerts ``` **To create a template for the subject of an email** 1. Create a template called `email.subject` with the following content: ``` {{ define "email.subject" }} {{ len .Alerts.Firing }} firing alerts, {{ len .Alerts.Resolved }} resolved alerts {{ end }} ``` 1. Use the template when creating your contact point integration by putting it into the **Subject** field with the `template` keyword. ``` {{ template "email.subject" . }} ``` ## Creating a template for the message of an email Create a template for the message of an email that contains a summary of all firing and resolved alerts, as in this example: ``` There are 2 firing alerts, and 1 resolved alerts Firing alerts: - alertname=Test 1 grafana_folder=GrafanaCloud has value(s) B=1 - alertname=Test 2 grafana_folder=GrafanaCloud has value(s) B=2 Resolved alerts: - alertname=Test 3 grafana_folder=GrafanaCloud has value(s) B=0 ``` **To create a template for the message of an email** 1. Create a notification template called `email` with two templates in the content: `email.message_alert` and `email.message`. The `email.message_alert` template is used to print the labels and values for each firing and resolved alert while the `email.message` template contains the structure of the email. ``` {{- define "email.message_alert" -}} {{- range .Labels.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }} has value(s) {{- range $k, $v := .Values }} {{ $k }}={{ $v }}{{ end }} {{- end -}} {{ define "email.message" }} There are {{ len .Alerts.Firing }} firing alerts, and {{ len .Alerts.Resolved }} resolved alerts {{ if .Alerts.Firing -}} Firing alerts: {{- range .Alerts.Firing }} - {{ template "email.message_alert" . }} {{- end }} {{- end }} {{ if .Alerts.Resolved -}} Resolved alerts: {{- range .Alerts.Resolved }} - {{ template "email.message_alert" . }} {{- end }} {{- end }} {{ end }} ``` 1. Use the template when creating your contact point integration by putting it into the **Text Body** field with the `template` keyword. ``` {{ template "email.message" . }} ``` ## Creating a template for the title of a Slack message Create a template for the title of a Slack message that contains the number of firing and resolved alerts, as in the following example: ``` 1 firing alerts, 0 resolved alerts ``` **To create a template for the title of a Slack message** 1. Create a template called `slack.title` with the following content: ``` {{ define "slack.title" }} {{ len .Alerts.Firing }} firing alerts, {{ len .Alerts.Resolved }} resolved alerts {{ end }} ``` 1. Execute the template from the title field in your contact point integration. ``` {{ template "slack.title" . }} ``` ## Creating a template for the content of a Slack message Create a template for the content of a Slack message that contains a description of all firing and resolved alerts, including their labels, annotations, and Dashboard URL. **Note** This template is for Grafana managed alerts only. To use the template for data source managed alerts, delete the references to DashboardURL and SilenceURL. For more information about configuring Prometheus notifications, see the [Prometheus documentation on notifications](https://prometheus.io/docs/alerting/latest/notifications/). ``` 1 firing alerts: [firing] Test1 Labels: - alertname: Test1 - grafana_folder: GrafanaCloud Annotations: - description: This is a test alert Go to dashboard: https://example.com/d/dlhdLqF4z?orgId=1 1 resolved alerts: [firing] Test2 Labels: - alertname: Test2 - grafana_folder: GrafanaCloud Annotations: - description: This is another test alert Go to dashboard: https://example.com/d/dlhdLqF4z?orgId=1 ``` **To create a template for the content of a Slack message** 1. Create a template called `slack` with two templates in the content: `slack.print_alert` and `slack.message`. The `slack.print_alert` template is used to print the labels, annotations, and DashboardURL while the `slack.message` template contains the structure of the notification. ``` {{ define "slack.print_alert" -}} [{{.Status}}] {{ .Labels.alertname }} Labels: {{ range .Labels.SortedPairs -}} - {{ .Name }}: {{ .Value }} {{ end -}} {{ if .Annotations -}} Annotations: {{ range .Annotations.SortedPairs -}} - {{ .Name }}: {{ .Value }} {{ end -}} {{ end -}} {{ if .DashboardURL -}} Go to dashboard: {{ .DashboardURL }} {{- end }} {{- end }} {{ define "slack.message" -}} {{ if .Alerts.Firing -}} {{ len .Alerts.Firing }} firing alerts: {{ range .Alerts.Firing }} {{ template "slack.print_alert" . }} {{ end -}} {{ end }} {{ if .Alerts.Resolved -}} {{ len .Alerts.Resolved }} resolved alerts: {{ range .Alerts.Resolved }} {{ template "slack.print_alert" .}} {{ end -}} {{ end }} {{- end }} ``` 1. Execute the template from the text body field in your contact point integration: ``` {{ template "slack.message" . }} ``` ## Template both email and Slack with shared templates Instead of creating separate notification templates for each contact point, such as email and Slack, you can share the same template. For example, if you want to send an email with this subject and Slack message with this title `1 firing alerts, 0 resolved alerts`, you can create a shared template. **To create a shared template** 1. Create a template called `common.subject_title` with the following content: ``` {{ define "common.subject_title" }} {{ len .Alerts.Firing }} firing alerts, {{ len .Alerts.Resolved }} resolved alerts {{ end }} ``` 1. For email, run the template from the subject field in your email contact point integration: ``` {{ template "common.subject_title" . }} ``` 1. For Slack, run the template from the title field in your Slack contact point integration: ``` {{ template "common.subject_title" . }} ``` ## Using notification templates Use templates in contact points to customize your notifications. **To use a template when creating a contact point** 1. From the **Alerting** menu, choose the **Contact points** tab to see a list of existing contact points. 1. Choose **New**. Alternately, you can edit an existing contact point by choosing the **Edit** icon. 1. Enter the templates you wish to use in a field, such as **Message** or **Subject**. To enter a template, use the form `{{ template "template_name" . }}`, replacing *template\$1name* with the name of the template you want to use. 1. Choose **Save contact point**. # Template reference **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). This section provides reference information for creating your templates. **Alert (type)** The alert type contains the following data. | Name | Kind | Description | Example | | --- | --- | --- | --- | | Status | string | `firing` or `resolved`. | \$1\$1 .Status \$1\$1 | | Labels | KeyValue | A set of labels attached to the alert. | \$1\$1 .Labels \$1\$1 | | Annotations | KeyValue | A set of annotations attached to the alert. | \$1\$1 .Annotations \$1\$1 | | Values | KeyValue | The values of all expressions, including Classic Conditions | \$1\$1 .Values \$1\$1 | | StartsAt | time.Time | Time the alert started firing. | \$1\$1 .StartsAt \$1\$1 | | EndsAt | time.Time | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received. | \$1\$1 .EndsAt \$1\$1 | | GeneratorURL | string | A back link to Grafana or external Alertmanager. | \$1\$1 .GeneratorURL \$1\$1 | | SilenceURL | string | A link to silence the alert (with labels for this alert pre-filled). Only for Grafana managed alerts. | \$1\$1 .SilenceURL\$1\$1 | | DashboardURL | string | Link to grafana dashboard, if alert rule belongs to one. Only for Grafana managed alerts. | \$1\$1 .DashboardURL \$1\$1 | | PanelURL | string | Link to grafana dashboard panel, if alert rule belongs to one. Only for Grafana managed alerts. | \$1\$1 .PanelURL \$1\$1 | | Fingerprint | string | Fingerprint that can be used to identify the alert. | \$1\$1 .Fingerprint \$1\$1 | | ValueString | string | A string that contains the labels and value of each reduced expression in the alert. | \$1\$1 .ValueString \$1\$1 | **ExtendedData** The ExtendedData object contains the following properties. | Name | Kind | Description | Example | | --- | --- | --- | --- | | Receiver | `string` | The name of the contact point sending the notification. | `{{ .Receiver }}` | | Status | `string` | The status is `firing` if at least one alert is firing, otherwise `resolved`. | `{{ .Status }}` | | Alerts | `[]Alert` | List of all firing and resolved alerts in this notification. | `There are {{ len .Alerts }} alerts` | | Firing alerts | `[]Alert` | List of all firing alerts in this notification. | `There are {{ len .Alerts.Firing }} firing alerts` | | Resolved alerts | `[]Alert` | List of all resolved alerts in this notification. | `There are {{ len .Alerts.Resolved }} resolved alerts` | | GroupLabels | `KeyValue` | The labels that group these alerts int his notification. | `{{ .GroupLabels }}` | | CommonLabels | `KeyValue` | The labels common to all alerts in this notification. | `{{ .CommonLabels }}` | | CommonAnnotations | `KeyValue` | The annotations common to all alerts in this notification. | `{{ .CommonAnnotations }}` | | ExternalURL | `string` | A link to the Grafana workspace or Alertmanager that sent this notification. | `{{ .ExternalURL }}` | **KeyValue type** The `KeyValue` type is a set of key/value string pairs that represent labels and annotations. In addition to direct access of the data stored as a `KeyValue`, there are also methods for sorting, removing, and transforming the data. | Name | Arguments | Returns | Notes | Example | | --- | --- | --- | --- | --- | | SortedPairs | | Sorted list of key and value string pairs | | `{{ .Annotations.SortedPairs }}` | | Remove | []string | KeyValue | Returns a copy of the Key/Value map without the given keys. | `{{ .Annotations.Remove "summary" }}` | | Names | | []string | List of names | `{{ .Names }}` | | Values | | []string | List of values | `{{ .Values }}` | **Time** Time is from the Go [https://pkg.go.dev/time#Time](https://pkg.go.dev/time#Time) package. You can print a time in a number of different formats. For example, to print the time that an alert fired in the format `Monday, 1st January 2022 at 10:00AM`, you write the following template: ``` {{ .StartsAt.Format "Monday, 2 January 2006 at 3:04PM" }} ``` You can find a reference for Go’s time format [here](https://pkg.go.dev/time#pkg-constants). # Manage contact points **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). The **Contact points** list view lists all existing contact points and notification templates. On the **Contact points** tab, you can: + Search for names and types of contact points and integrations. + View all existing contact points and integrations. + View how many notification policies each contact point is being used for, and navigate directly to the linked notification policies. + View the status of notification deliveries. + Export individual contact points or all contact points in JSON, YAML, or Terraform format. + Delete contact points that are not in use by a notification policy. On the **Notification templates** tab, you can: + View, edit, copy, or delete existing notification templates. # Silencing alert notifications **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). You can suppress alert notifications with a *silence*. A silence only stops notifications from being created: Silences do not prevent alert rules from being evaluated, and they do not stop alerting instances from being shown in the user interface. When you silence an alert, you specify a window of time for it to be suppressed. **Note** To suppress alert notifications at regular time intervals, for example, during regular maintenance periods, use [Mute timings](v12-alerting-manage-muting.md) rather than silences. **To add a silence** 1. From your Grafana console, in the Grafana menu, choose **Alerting**. 1. Choose **Silences**. 1. Choose an Alertmanager from the **Alertmanager** dropdown. 1. Choose **Create Silence**. 1. Select the start and end date in **Silence start and end** to indicate when the silence should go into effect and when it should end. 1. As an alternative to setting an end time, in **Duration**, specify how long the silence is enforced. This automatically updates the end time in the **Silence start and end** field. 1. In the **Label** and **Value** fields, enter one or more *Matching Labels*. Matchers determine which rules the silence applies to. Any matching alerts (in firing state), will show in the **Affected alerts instances** field. 1. Optionally, add a **Comment** describing the silence. 1. Choose **Submit**. **To edit a silence** 1. From your Grafana console, in the Grafana menu, choose **Alerting**. 1. Choose **Silences** to view the list of existing silences. 1. Find the silence you want to edit, then choose **Edit** (pen icon). 1. Make any desired changes, then choose **Submit** to save your changes. You can edit an existing silence by choosing the **Edit** icon (pen). **To create a URL link to a silence form** When linking to a silence form, provide the default matching labels and comment via `matcher` and `comment` query parameters. The `matcher` parameter should be in the following format `[label][operator][value]` where the `operator` parameter can be one of the following: `=` (equals, not regex), `!=` (not equals, not regex), `=~` (equals, regex), `!~` (not equals, regex). The URL can contain many query parameters with the key `matcher`. For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/alerting/silence/new?matcher=severity%3Dcritical&matcher=cluster!~europe-*&comment=Silence%20critical%20EU%20alert`. To link to a new silence page for an external Alertmanager, add an `alertmanager` query parameter **To remove a silence** 1. From your Grafana console, in the Grafana menu, choose **Alerting**. 1. Choose **Silences** to view the list of existing silences. 1. Select the silence that you want to end, and choose **Unsilence**. This ends the alert suppression. **Note** Unsilencing ends the alert suppression, as if the end time was set for the current time. Silences that have ended (automatically or manually) are retained and listed for five days. You cannot remove a silence from the list manually. # View and filter alert rules **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). The **Alerting** page lists alerting rules. By default, rules are grouped by types of data sources. The **Grafana** section lists rules managed by Grafana. Alert rules for Prometheus compatible data sources are also listed here. You can view alerting rules for Prometheus compatible data sources but you cannot edit them. The Mimir/Cortex/Loki rules section lists all rules for Mimir, Cortex, or Loki data sources. Cloud alert rules are also listed in this section. When managing large volumes of alerts, you can use extended alert rule search capabilities to filter on folders, evaluation groups, and rules. Additionally, you can filter alert rules by their properties like labels, state, type, and health. ## View alert rules Using Grafana alerts, you can view all of your alerts in one page. **To view alerting details** 1. From your Grafana console, in the Grafana menu, choose **Alerting**, **Alert rules**. By default, the list view is displayed. 1. In **View as**, you can toggle between the Grouped, List, and State views by choosing the option you prefer. 1. Expand the rule row to view the rule labels, annotations, data sources, the rule queries, and a list of alert instances resulting from the rule. From this page, you can also make copies of an alert rule to help you reuse existing rules. ## Export alert rules You can export rules to YAML or JSON in the Grafana workspace. + Choose the **Export rule group** icon next to each alert rule group to export to YAML, JSON, or Terraform. + Choose **Export rules** to export all Grafana managed alert rules to YAML, JSON, or Terraform. + Choose **More**, **Modify export** next to each individual alert rule within a group to edit provisioned alert rules and export a modified version. ## View query definitions for provisioned alerts View read-only query definitions for provisioned alerts. Check quickly if your alert rule queries are correct, without diving into your "as-code" repository for rule definitions. **Grouped view** Grouped view shows Grafana alert rules grouped by folder and Loki or Prometheus alert rules grouped by `namespace` \$1 `group`. This is the default rule list view, intended for managing rules. You can expand each group to view a list of rules in this group. Expand a rule further to view its details. You can also expand action buttons and alerts resulting from the rule to view their details. **State view** State view shows alert rules grouped by state. Use this view to get an overview of which rules are in what state. Each rule can be expanded to view its details. Action buttons and any alerts generated by this rule, and each alert can be further expanded to view its details. ## Filtering alert rules You can filter the alerting rules that appear on the **Alerting** page in several ways. **To filter alert rules** 1. From **Select data sources**, select a data source.. You can see alert rules that query the selected data source. 1. In **Search by label**, enter search criteria using label selectors. For example, `environment=production;region=~US|EU,severity!=warning`. 1. From **Filter alerts by state**, select an alerting state you want to see. You can see alerting rules that match that state. Rules matching other states are hidden. # Mute timings **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). A mute timing is a recurring interval of time when no new notifications for a policy are generated or sent. Use them to prevent alerts from firing a specific and reoccurring period, for example, a regular maintenance period. Similar to silences, mute timings do not prevent alert rules from being evaluated, nor do they stop alert instances from being shown in the user interface. They only prevent notifications from being created. You can configure Grafana managed mute timings as well as mute timings for an external Alertmanager data source. ## Mute timings vs Silences The following table highlights the differences between mute timings and silences. | Mute timing | Silence | | --- | --- | | Uses time interval definitions that can reoccur. | Has a fixed start and end time. | | Is created and then added to notification policies. | Uses labels to match against an alert to determine whether to silence or not. | ## Adding a mute timing You can create mute timings in your Grafana workspace. **To add a mute timing** 1. From your Grafana console, in the Grafana menu, choose **Alerting**. 1. Choose **Notification policies**, and then select the **Mute Timings** tab. 1. From the **Alertmanager** dropdown, select the Alertmanager you want to edit. 1. Choose the **\$1 Add mute timing** button. 1. Fill out the form to create a [time interval](#v12-alerting-manage-muting-interval) to match against for your mute timing. 1. Save your mute timing. ## Adding a mute timing to a notification policy Once you have a mute timing, you use it by adding it to notification policy that you want to mute at regular intervals. **To add a mute timing to a notification policy** 1. From your Grafana console, in the Grafana menu, choose **Alerting**. 1. Choose **Notification policies**, and then select the **Notification Policies** tab. 1. Select the notification policy you would like to add the mute timing to, and choose **...**, **Edit**. 1. From the **Mute timings** dropdown, select the mute timings you would like to add to the policy. 1. Save your changes. ## Time intervals A time interval is a specific duration during which alerts are suppressed. The duration typically consists of a specific time range and the days of the week, month, or year. Support time interval options are: + **Time range** – The time inclusive of the start and exclusive of the end time (in UTC, if no location has been selected, otherwise local time. + **Location** – Sets the location for the timing—the time range is displayed in local time for the location. + **Days of the week** – The day or range of days of the week. For example, `monday:thursday`. + **Days of the month** – The dates within a month. Values can range from `1`-`31`. Negative values specify days of the month in reverse order, so `-1` represents the last day of the month. + **Months** – The months of the year in either numerical of full calendar month name. For example, `1, may:august`. + **Years** – The year or years for the interval. For example, `2023:2024`. Each of these elements can be a list, and at least one item in the element must be satisfied to be a match. Fields also support ranges, using `:`. For example, `monday:thursday`. If a field is left blank, any moment of time will match the field. For an instant of to match a complete time intervale, all fields must match. A mute timing can contain multiple time intervals. If you want to specify an exact duration, specify all the options needed for that duration. For example, if you want to create a time interval for the first Monday of the month, for March, June, September, and December, between the hours of 12:00 and 24:00 UTC, your time interval specification could be: + Time range: + Start time: `12:00` + End time: `24:00` + Days of the week: `monday` + Months: `3, 6, 9, 12` + Days of the month: `1:7` # View the state and health of alert rules **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). The state and health of alert rules gives you several key status indicators about your alerts. There are three components: + [Alert rule state](#v12-alerting-manage-rulestate-state) + [Alert instance state](#v12-alerting-manage-rulestate-instance) + [Alert rule health](#v12-alerting-manage-rulestate-health) Although related, each component conveys subtly different information. **To view the state and health of your alert rules** 1. From your Grafana console, in the Grafana menu, choose **Alerting**. 1. Choose **Alert rules** to view the list of existing alerts. 1. Choose an alert rule to view its state and health. ## Alert rule state An alert rule can be in any of the following states: | State | Description | | --- | --- | | Normal | None of the time series returned by the evaluation engine is in a pending or firing state. | | Pending | At least one time series returned by the evaluation engine is pending. | | Firing | At least one time series returned by the evaluation engine is firing. | **Note** Alerts transition first to `pending` and then `firing`, thus it takes at least two evaluation cycles before an alert is fired. ## Alert instance state An alert instance can be in any of the following states: | State | Description | | --- | --- | | Normal | The state of an alert that is neither pending nor firing. Everything is working as expected. | | Pending | The state of an alert that has been active for less than the configured threshold duration. | | Alerting | The state of an alert that has been active for longer than the configured threshold duration. | | No data | No data has been received for the configured time window. | | Alerting | An error occurred when attempting to evaluate an alerting rule. | ## Keep last state An alert rule can be configured to keep the last state when a `NoData` or `Error` state is encountered. This will both prevent alerts from firing, and from resolving and re-firing. Just like normal evaluation, the alert rule will transition from `pending` to `firing` after the pending period has elapsed. ## Alert rule health An alert rule can have one of the following health statuses. | State | Description | | --- | --- | | Ok | No errors when evaluating the alert rule. | | Error | An error occurred when evaluating the alert rule. | | NoData | The absence of data in at least one time series returned during a rule evaluation. | | \$1status\$1, KeepLast | The rule would have received another status, but was configured to keep the last state of the alert rule. | ## Special alerts for NoData and Error When evaluation of an alert rule produces the state `NoData` or `Error`, Grafana alerting will generate alert instances that have the following additional labels. | Label | Description | | --- | --- | | alertname | Either DatasourceNoData or DatasourceError, depending on the state. | | datasource\$1uid | The UID of the data source that caused the state. | **Note** You will need to set the no data or error handling to `NoData` or `Error` in the alert rule, as described in the [Configure Grafana managed alert rules](v12-alerting-configure-grafanamanaged.md) topic, to generate the additional labels. You can handle these alerts the same way as regular alerts, including adding silences, routing to a contact point, and so on. # View and filter by alert groups **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). Alert groups show grouped alerts from an Alertmanager instance. By default, alert rules are grouped by the label keys for the default policy in notification policies. Grouping common alert rules into a single alert group prevents duplicate alert rules from being fired. You can view alert groups and also filter for alert rules that match specific criteria. **To view alert groups** 1. From your Grafana console, in the Grafana menu, choose **Alerting**. 1. Choose **Groups** to view existing groups. 1. From the **Alertmanager** dropdown, select an external Alertmanager as your data source. 1. From the **Custom group by** dropdown, select a combination of labels to view a grouping other than the default. This is useful for debugging and verifying your grouping of notification policies. If an alert does not contain labels specified either in the grouping of the root policy or the custom grouping, then the alert is added to a catch all group with a header of `No grouping`. You can filter alerts by label or state of the alerts. **To filter by label** + In **Search**, enter an existing label to view alerts matching the label. For example, `environment=production,region=~US|EU,severity!=warning`. **To filter by state** + In **States**, select from Active, Suppressed, or Unprocessed states to view alerts matching your selected state. All other alerts are hidden. # View notification errors **** This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**. For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md). For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md). For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md). View notification errors and understand why they failed to be sent or were not received. **Note** This feature is only supported for Grafana Alertmanager. **To view notification errors** 1. From the left menu, choose **Alerting** then **Contact points**. If any contact points are failing, a message at the right-hand corner of the workspace tells you that there are errors, and how many. 1. Select a contact point to view the details of errors for that contact point. Error details are displayed if you hover over the Error icon. If a contact point has more than one integration, you see all errors for each of the integrations listed. 1. In the Health column, check the status of the notification. This can be either OK, No attempts, or Error.