|
| Approval for suspension | example-inc:suspension:approval | The link to the approval of account suspension | workload/deprecation |
# Incident management
Tags can play a vital part in all phases of incident management starting from incident logging, prioritization, investigation, communication, resolution to closure.
Tags can detail where an incident should be logged, the team or teams that should be informed of the incident, and the defined escalation priority. It’s important to remember that tags are not encrypted, so consider what information you store in them. Also, in organizations, teams, and reporting lines, responsibilities change, so consider storing a link to a secure portal where this information can be more effectively managed. These tags don’t need to be exclusive. For instance, the application ID could be used to lookup the escalation paths in an IT service management portal. Make sure it is clear in your operational definitions that this tag is being used for multiple purposes.
Operational requirement tags can be detailed as well, to help incident managers and operations personnel further refine their objectives in response to an incident or event.
Relative links (to the knowledge system base URL) for [runbooks](https://wa.aws.amazon.com/wellarchitected/2020-07-02T19-33-23/wat.concept.runbook.en.html) and [playbooks](https://wa.aws.amazon.com/wellarchitected/2020-07-02T19-33-23/wat.concept.playbook.en.html) can be included as tags to assist the responding teams in identifying corresponding process, procedure and documentation.
*Table 9 - Use operational tags to inform incident management*
| Use Case | Tag Key | Rationale | Example Values |
| --- | --- | --- | --- |
| Incident Management | example-inc:incident-management:escalationlog | The system in use by the supporting team to log incidents | jira, servicenow, zendesk |
| Incident Management | example-inc:incident-management:escalationpath | Path of escalation | ops-center, dev-ops, app-team |
| Cost Allocation and Incident Management | example-inc:cost-allocation:CostCenter | Monitor costs by cost center. This is an example of a dual use tag where the cost center is being used as an application code for incident logging | 123-\$1 |
| Backup Schedule | example-inc:backup:schedule | Backup schedule of the resource | Daily |
| Playbook / Incident Management | example-inc:incident-management:playbook | Documented playbook | webapp/incident/playbook |
# Patching
Organizations can automate their patching strategy for mutable compute environments and keep mutable instances in-line with the defined patch baseline of that application environment by using AWS Systems Manager Patch Manager and AWS Lambda. A tagging strategy for mutable instances within these environments can be managed by assigning said instances to **Patch Groups** and **Maintenance Windows**. See the following examples for a Dev → Test → Prod split. AWS prescriptive guidance is available for the [patch management of mutable instances.](https://docs.aws.amazon.com/prescriptive-guidance/latest/patch-management-hybrid-cloud/welcome.html)
*Table 10 - Operational tags can be environment specific*
| Development | Staging | Production |
| --- | --- | --- |
| {
"Tags": [
{
"Key": "Maintenance Window",
"ResourceId": "i-012345678ab9ab111",
"ResourceType": "instance",
"Value": "cron(30 23 ? * TUE#1 *)"
},
{
"Key": "Name",
"ResourceId": "i-012345678ab9ab222",
"ResourceType": "instance",
"Value": "WEBAPP"
},
{
"Key": "Patch Group",
"ResourceId": "i-012345678ab9ab333",
"ResourceType": "instance",
"Value": "WEBAPP-DEV-AL2"
}
]
}
| {
"Tags": [
{
"Key": "Maintenance Window",
"ResourceId": "i-012345678ab9ab444",
"ResourceType": "instance",
"Value": "cron(30 23 ? * TUE#2 *)"
},
{
"Key": "Name",
"ResourceId": "i-012345678ab9ab555",
"ResourceType": "instance",
"Value": "WEBAPP"
},
{
"Key": "Patch Group",
"ResourceId": "i-012345678ab9ab666",
"ResourceType": "instance",
"Value": "WEBAPP-TEST-AL2"
}
]
}
| {
"Tags": [
{
"Key": "Maintenance Window",
"ResourceId": "i-012345678ab9ab777",
"ResourceType": "instance",
"Value": "cron(30 23 ? * TUE#3 *)"
},
{
"Key": "Name",
"ResourceId": "i-012345678ab9ab888",
"ResourceType": "instance",
"Value": "WEBAPP"
},
{
"Key": "Patch Group",
"ResourceId": "i-012345678ab9ab999",
"ResourceType": "instance",
"Value": "WEBAPP-PROD-AL2"
}
]
}
|
Zero-day vulnerabilities can also be managed by having tags defined to complement your patching strategy. Refer to [Avoid zero-day vulnerabilities with same-day security patching using AWS Systems Manager](https://aws.amazon.com/blogs/mt/avoid-zero-day-vulnerabilities-same-day-security-patching-aws-systems-manager/) for detailed guidance.
# Operational observability
Observability is required to gain actionable insights into the performance of your environments and help you to detect and investigate problems. It also has a secondary purpose that allows you to define and measure key performance indicators (KPIs) and service level objectives (SLOs) such as uptime. For most organizations, important operations KPIs are mean time to detect (MTTD) and mean time to recover (MTTR) from an incident.
Throughout observability, context is important, because data is collected and then associated tags are gathered. Regardless of the service, application, or application tier that you are focusing on, you can filter and analyze for that specific dataset. Tags can be used to automate onboarding to CloudWatch Alarms so that the right teams can be alerted when certain metric thresholds are breached. For example, a tag key `example-inc:ops:alarm-tag` and the value on it could indicate creation of the CloudWatch Alarm. A solution demonstrating this is described in [Use tags to create and maintain Amazon CloudWatch alarms for Amazon EC2 instances](https://aws.amazon.com/blogs/mt/use-tags-to-create-and-maintain-amazon-cloudwatch-alarms-for-amazon-ec2-instances-part-1/).
Having too many alarms configured can easily create an alert storm—when a large number of alarms or notifications rapidly overwhelm operators and reduce their overall effectiveness while operators are manually triaging and prioritizing individual alarms. Additional context for the alarms can be provided in the form of tags, which means that rules can be defined within Amazon EventBridge to help ensure that focus is given to the upstream issue rather than downstream dependencies.
The role of operations alongside DevOps is often overlooked, but for many organizations, central operations teams still provide a critical first response outside of normal business hours. (More details can be found about this model in the [Operational Excellence whitepaper](https://docs.aws.amazon.com/wellarchitected/latest/operational-excellence-pillar/separated-aeo-ieo-with-cent-gov-and-partner.html).) Unlike the DevOps team that owns the workload, they typically do not have the same depth of knowledge, so the context that tags provide within dashboards and alerts, can direct them to the correct runbook for the issue, or initiate an automated runbook (refer to the blog post [Automating Amazon CloudWatch Alarms with AWS Systems Manager](https://aws.amazon.com/blogs/mt/automating-amazon-cloudwatch-alarms-with-aws-systems-manager/)).