# Notifications
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Choosing how, when, and where to send your alert notifications is an important part of setting up your alerting system. These decisions will have a direct impact on your ability to resolve issues quickly and not miss anything important.
As a first step, define your [contact points](v12-alerting-explore-contacts.md), which define where to send your alert notifications. A contact point is a set of one or more integrations that are used to deliver notifications. Add notification templates to contact points for reuse and consistent messaging in your notifications.
Next, create a notification policy which is a set of rules for where, when and how your alerts are routed to contact points. In a notification policy, you define where to send your alert notifications by choosing one of the contact points you created.
## Alertmanagers
Grafana uses Alertmanagers to send notifications for firing and resolved alerts. Grafana has its own Alertmanager, referred to as **Grafana** in the user interface, but also supports sending notifications from other Alertmanagers too, such as the [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/). The Grafana Alertmanager uses notification policies and contact points to configure how and where a notification is sent; how often a notification should be sent; and whether alerts should all be sent in the same notification, sent in grouped notifications based on a set of labels, or as separate notifications.
## Notification policies
Notification policies control when and where notifications are sent. A notification policy can choose to send all alerts together in the same notification, send alerts in grouped notifications based on a set of labels, or send alerts as separate notifications. You can configure each notification policy to control how often notifications should be sent as well as having one or more mute timings to inhibit notifications at certain times of the day and on certain days of the week.
Notification policies are organized in a tree structure where at the root of the tree there is a notification policy called the default policy. There can be only one default policy and the default policy cannot be deleted.
Specific routing policies are children of the root policy and can be used to match either all alerts or a subset of alerts based on a set of matching labels. A notification policy matches an alert when its matching labels match the labels in the alert.
A nested policy can have its own nested policies, which allow for additional matching of alerts. An example of a nested policy could be sending infrastructure alerts to the Ops team; while a child policy might send high priority alerts to Pagerduty and low priority alerts to Slack.
All alerts, irrespective of their labels, match the default policy. However, when the default policy receives an alert it looks at each nested policy and sends the alert to the first nested policy that matches the alert. If the nested policy has further nested policies, then it can attempt to match the alert against one of its nested policies. If no nested policies match the alert then the policy itself is the matching policy. If there are no nested policies, or no nested policies match the alert, then the default policy is the matching policy.
For more detailed information about notification policies, see [Notification policies](v12-alerting-explore-notifications-policies-details.md).
## Notification templates
You can customize notifications with templates. For example, templates can be used to change the title and message of notifications sent to Slack.
Templates are not limited to an individual integration or contact point, but instead can be used in a number of integrations in the same contact point and even integrations across different contact points. For example, a Grafana user can create a template called `custom_subject_or_title` and use it for both templating subjects in Pager Duty and titles of Slack messages without having to create two separate templates.
All notifications templates are written in [Go’s templating language](https://pkg.go.dev/text/template), and are in the Contact points tab on the Alerting page.
For more detailed information about customizing notifications, see [Customize notifications](v12-alerting-manage-notifications.md).
## Silences
You can use silences to mute notifications from one or more firing rules. Silences do not stop alerts from firing or being resolved, or hide firing alerts in the user interface. A silence lasts as long as its duration which can be configured in minutes, hours, days, months or years.
For more detailed information about using silences, see [Silencing alert notifications](v12-alerting-silences.md).
# Notification policies
****
This documentation topic is designed for Grafana workspaces that support **Grafana version 12.x**.
For Grafana workspaces that support Grafana version 10.x, see [Working in Grafana version 10](using-grafana-v10.md).
For Grafana workspaces that support Grafana version 9.x, see [Working in Grafana version 9](using-grafana-v9.md).
For Grafana workspaces that support Grafana version 8.x, see [Working in Grafana version 8](using-grafana-v8.md).
Notification policies provide you with a flexible way of routing alerts to various different receivers. Using label matchers, you can modify alert notification delivery without having to update every individual alert rule.
In this section, you will learn more about how notification policies work and are structured, so that you can make the most out of setting up your notification policies.
## Policy tree
Notification policies are *not* a list, but rather are structured according to a tree structure. This means that each policy can have child policies, and so on. The root of the notification policy tree is called the **Default notification policy**.
Each policy consists of a set of label matchers (0 or more) that specify which labels they are or aren’t interested in handling.
For more information about label matching, see [How label matching works](v12-alerting-overview-labels-matching.md).
**Note**
If you haven’t configured any label matchers for your notification policy, your notification policy will match *all* alert instances. This may prevent child policies from being evaluated unless you have enabled **Continue matching siblings** on the notification policy.
## Routing
To determine which notification policy will handle which alert instances, you have to start by looking at the existing set of notification policies, starting with the default notification policy.
If no policies other than the default policy are configured, the default policy will handle the alert instance.
If policies other than the default policy are defined, it will evaluate those notification policies in the order they are displayed.
If a notification policy has label matchers that match the labels of the alert instance, it will descend in to its child policies and, if there are any, will continue to look for any child policies that might have label matchers that further narrow down the set of labels, and so forth until no more child policies have been found.
If no child policies are defined in a notification policy or if none of the child policies have any label matchers that match the alert instance’s labels, the parent notification policy is used.
As soon as a matching policy is found, the system does not continue to look for other matching policies. If you want to continue to look for other policies that may match, enable **Continue matching siblings** on that particular policy.
Lastly, if none of the notification policies are selected the default notification policy is used.
### Routing example
Here is an example of a relatively simple notification policy tree and some alert instances.
![\[An image showing a set of notification policies in a tree structure, and a set of alert instances with different labels to match to the policies.\]](http://docs.aws.amazon.com/grafana/latest/userguide/images/notification-routing.png)
Here’s a breakdown of how these policies are selected:
**Pod stuck in CrashLoop** does not have a `severity` label, so none of its child policies are matched. It does have a `team=operations` label, so the first policy is matched.
The `team=security` policy is not evaluated since we already found a match and **Continue matching siblings** was not configured for that policy.
**Disk Usage – 80%** has both a `team` and `severity` label, and matches a child policy of the operations team.
**Unauthorized log entry** has a `team` label but does not match the first policy (`team=operations`) since the values are not the same, so it will continue searching and match the `team=security` policy. It does not have any child policies, so the additional `severity=high` label is ignored.
## Inheritance
In addition to child policies being a useful concept for routing alert instances, they also inherit properties from their parent policy. This also applies to any policies that are child policies of the default notification policy.
The following properties are inherited by child policies:
+ Contact point
+ Grouping options
+ Timing options
+ Mute timings
Each of these properties can be overwritten by an individual policy should you wish to override the inherited properties.
To inherit a contact point from the parent policy, leave it blank. To override the inherited grouping options, enable **Override grouping**. To override the inherited timing options, enable **Override general timings**.
### Inheritance example
The example below shows how the notification policy tree from our previous example allows the child policies of the `team=operations` to inherit its contact point.
In this way, we can avoid having to specify the same contact point multiple times for each child policy.
![\[An image showing a set of notification policies in a tree structure, with contact points assigned to some of the policies, but with some child policies inheriting the contact points of their parents, rather than defining their own.\]](http://docs.aws.amazon.com/grafana/latest/userguide/images/notification-inheritance.png)
## Additional configuration options
### Grouping
Grouping is an important feature of Grafana Alerting as it allows you to batch relevant alerts together into a smaller number of notifications. This is particularly important if notifications are delivered to first-responders, such as engineers on-call, where receiving lots of notifications in a short period of time can be overwhelming and in some cases can negatively impact a first-responders ability to respond to an incident. For example, consider a large outage where many of your systems are down. In this case, grouping can be the difference between receiving 1 phone call and 100 phone calls.
You choose how alerts are grouped together using the Group by option in a notification policy. By default, notification policies in Grafana group alerts together by alert rule using the `alertname` and `grafana_folder` labels (since alert names are not unique across multiple folders). Should you wish to group alerts by something other than the alert rule, change the grouping to any other combination of labels.
#### Disable grouping
Should you wish to receive every alert as a separate notification, you can do so by grouping by a special label called `...`. This is useful when your alerts are being delivered to an automated system instead of a first-responder.
#### A single group for all alerts
Should you wish to receive all alerts together in a single notification, you can do so by leaving Group by empty.
### Timing options
The timing options decide how often notifications are sent for each group of alerts. There are three timers that you need to know about: Group wait, Group interval, and Repeat interval.
#### Group wait
Group wait is the amount of time Grafana waits before sending the first notification for a new group of alerts. The longer Group wait is the more time you have for other alerts to arrive. The shorter Group wait is the earlier the first notification will be sent, but at the risk of sending incomplete notifications. You should always choose a Group wait that makes the most sense for your use case.
**Default** 30 seconds
#### Group interval
Once the first notification has been sent for a new group of alerts, Grafana starts the Group interval timer. This is the amount of time Grafana waits before sending notifications about changes to the group. For example, another firing alert might have just been added to the group while an existing alert might have resolved. If an alert was too late to be included in the first notification due to Group wait, it will be included in subsequent notifications after Group interval. Once Group interval has elapsed, Grafana resets the Group interval timer. This repeats until there are no more alerts in the group after which the group is deleted.
**Default** 5 minutes
#### Repeat interval
Repeat interval decides how often notifications are repeated if the group has not changed since the last notification. You can think of these as reminders that some alerts are still firing. Repeat interval is closely related to Group interval, which means your Repeat interval must not only be greater than or equal to Group interval, but also must be a multiple of Group interval. If Repeat interval is not a multiple of Group interval it will be coerced into one. For example, if your Group interval is 5 minutes, and your Repeat interval is 9 minutes, the Repeat interval will be rounded up to the nearest multiple of 5 which is 10 minutes.
**Default** 4 hours