# Operations
<a name="sap-hana-pacemaker-operations"></a>

**Topics**
+ [Viewing the cluster state](sap-hana-pacemaker-sles-ops-cluster-state.md)
+ [Performing planned maintenance](sap-hana-pacemaker-sles-ops-planned-maint.md)
+ [Post-failure analysis and reset](sap-hana-pacemaker-sles-ops-post-failure.md)
+ [Alerting and monitoring](sap-hana-pacemaker-sles-ops-alert-monitor.md)

# Viewing the cluster state
<a name="sap-hana-pacemaker-sles-ops-cluster-state"></a>

You can view the state of the cluster in two ways - based on your operating system or with a web based console provided by SUSE.

**Topics**
+ [Operating system based](#_operating_system_based)
+ [SUSE Hawk2](#_suse_hawk2)

## Operating system based
<a name="_operating_system_based"></a>

There are multiple operating system commands that can be run as root or as a user with appropriate permissions. The commands enable you to get an overview of the status of the cluster and its services. See the following commands for more details.

```
# crm status
```

Sample output:

```
Cluster Summary:
  * Stack: corosync
  * Current DC: sapsecdb (version 2.0.5+20201202.ba59be712-150300.4.45.2-2.0.5+20201202.ba59be712) - partition with quorum
  * Last updated: Wed Aug 20 14:05:19 2025
  * Last change:  Wed Aug 20 14:04:54 2025 by root via crm_attribute on hanahost01
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Online: [ hanahost01 hanahost02  ]

Full List of Resources:
  * rsc_AWS_STONITH     (stonith:external/ec2):  Started sapsecdb
  * rsc_ip_HDB_HDB00    (ocf::heartbeat:aws-vpc-move-ip):        Started hanahost01
  * Clone Set: cln_SAPHanaTopology_HDB_HDB00 [rsc_SAPHanaTopology_HDB_HDB00]:
    * Started: [ hanahost01 hanahost02 ]
  * Clone Set: msl_SAPHana_HDB_HDB00 [rsc_SAPHana_HDB_HDB00] (promotable):
    * Masters: [ hanahost01 ]
    * Slaves: [ hanahost02 ]
```

The following table provides a list of useful commands.


| Command | Description | 
| --- | --- | 
|   `crm_mon`   |  Display cluster status on the console with updates as they occur  | 
|   `crm_mon -1`   |  Display cluster status on the console just once, and exit  | 
|   `crm_mon -Arnf`   |  -A Display node attributes -n Group resources by node -r Display inactive resources -f Display resource fail counts  | 
|   `crm help`   |  View more options  | 
|   `crm_mon --help-all`   |  View more options  | 

## SUSE Hawk2
<a name="_suse_hawk2"></a>

Hawk2 is a web-based graphical user interface for managing and monitoring pacemaker highly availability clusters. It must be enabled on every node in the cluster, to point your web browser on any node for accessing it. Use the following command to enable Hawk2.

```
# systemctl enable --now hawk
# systemctl status hawk
```

Use the following URL to check security groups for access on port 7630 from your administrative host.

```
https://your-server:7630/

e.g https://hanahost01:7630
```

For more information, see [Configuring and Managing Cluster Resources with Hawk2](https://documentation.suse.com/sle-ha/12-SP5/html/SLE-HA-all/cha-conf-hawk2.html) in the SUSE Documentation.

# Performing planned maintenance
<a name="sap-hana-pacemaker-sles-ops-planned-maint"></a>

When performing maintenance on SAP HANA systems in a cluster environment, it’s important to understand how the cluster interacts with SAP HANA system replication. Planned maintenance activities should be conducted carefully to prevent unnecessary failovers or cluster interventions.

There are different options to perform planned maintenance on nodes, resources, and the cluster.

**Topics**
+ [Maintenance mode](#_maintenance_mode)
+ [Placing a node in standby mode](#_placing_a_node_in_standby_mode)
+ [Moving a resource](#_moving_a_resource)

## Maintenance mode
<a name="_maintenance_mode"></a>

Use maintenance mode if you want to make any changes to the configuration or take control of the resources and nodes in the cluster. In most cases, this is the safest option for administrative tasks.

**Example**  
Use one of the following commands to turn on maintenance mode.  

```
# crm maintenance on
```

```
# crm configure property maintenance-mode="true"
```
Use one of the following commands to turn off maintenance mode.  

```
# crm maintenance off
```

```
# crm configure property maintenance-mode="false"
```

## Placing a node in standby mode
<a name="_placing_a_node_in_standby_mode"></a>

To perform maintenance on the cluster without a full system outage, the recommended method for moving active resources is to place the node you want to remove from the cluster in standby mode.

```
# crm node standby <hostname>
```

The cluster will cleanly relocate resources, and you can perform activities, including reboots on the node in standby mode. When maintenance activities are complete, you can re-introduce the node with the following command.

```
# crm node online <hostname>
```

## Moving a resource
<a name="_moving_a_resource"></a>

Moving individual resources is not recommended because of the migration or move constraints that are created to lock the resource in its new location. These can be cleared as described in the info messages, but this introduces an additional setup.

```
 # crm resource move msl_SAPHanaController_HDB_HDB00 hanahost02
INFO: Move constraint created for msl_SAPHanaController_HDB_HDB00 to hanahost02
INFO: Use `crm resource clear msl_SAPHanaController_HDB_HDB00` to remove this constraint
```

Note: The exact resource name will vary depending on your SAP HANA system ID and instance number. Adjust the commands accordingly.

Use the following command once the resources have relocated to their target location.

```
# crm resource clear msl_SAPHanaController_HDB_HDB00
```

# Post-failure analysis and reset
<a name="sap-hana-pacemaker-sles-ops-post-failure"></a>

A review must be conducted after each failure to understand the source of failure as well the reaction of the cluster. In most scenarios, the cluster prevents an application outage. However, a manual action is often required to reset the cluster to a protective state for any subsequent failures.

**Topics**
+ [Checking the Logs](#_checking_the_logs)
+ [Cleanup crm status](#_cleanup_crm_status)
+ [Restart failed nodes or pacemaker](#_restart_failed_nodes_or_pacemaker)
+ [Further Analysis](#_further_analysis)

## Checking the Logs
<a name="_checking_the_logs"></a>
+ For troubleshooting cluster issues, use journalctl to examine both pacemaker and corosync logs:

  ```
  # journalctl -u pacemaker -u corosync --since "1 hour ago"
  ```
  + Use `--since` to specify time periods (e.g., "2 hours ago", "today")
  + Add `-f` to follow logs in real-time
  + Combine with grep for specific searches
+ System messages and resource agent activity can be found in `/var/log/messages`.
+ For HANA-specific issues, check the HANA trace directory. This can be reached using 'cdtrace' when logged in as <sid>adm. Also consult the DB\$1<tenantdb> directory within the HANA trace directory.

## Cleanup crm status
<a name="_cleanup_crm_status"></a>

If failed actions are reported using the `crm status` command, and if they have already been investigated, then you can clear the reports with the following command.

```
# crm resource cleanup <resource> <hostname>
```

## Restart failed nodes or pacemaker
<a name="_restart_failed_nodes_or_pacemaker"></a>

It is recommended that failed (or fenced) nodes are not automatically restarted. It gives operators a chance to investigate the failure, and ensure that the cluster doesn’t make assumptions about the state of resources.

You need to restart the instance or the pacemaker service based on your approach.

## Further Analysis
<a name="_further_analysis"></a>

For cluster-specific issues, use `hb_report` to generate a targeted analysis of cluster components across all nodes:

```
# hb_report -f "YYYY-MM-DD HH:MM:SS" -t "YYYY-MM-DD HH:MM:SS" /tmp/hb_report
```

For quick analysis of recent events, you can use:

```
# crm history events
# crm history log
```
+ Both `hb_report` and `crm history` commands require passwordless SSH between nodes
+ For more information, see SUSE Documentation - [Usage of hb\$1report for SLES HAE](https://www.suse.com/support/kb/doc/?id=000017501) 

# Alerting and monitoring
<a name="sap-hana-pacemaker-sles-ops-alert-monitor"></a>

This section covers the following topics.

**Topics**
+ [Using Amazon CloudWatch Application Insights](#_using_amazon_cloudwatch_application_insights)
+ [Using the cluster alert agents](#_using_the_cluster_alert_agents)

## Using Amazon CloudWatch Application Insights
<a name="_using_amazon_cloudwatch_application_insights"></a>

For monitoring and visibility of cluster state and actions, Application Insights includes metrics for monitoring enqueue replication state, cluster metrics, and SAP and high availability checks. Additional metrics, such as EFS and CPU monitoring can also help with root cause analysis.

For more information, see [Get started with Amazon CloudWatch Application Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/appinsights-getting-started.html) and [SAP HANA High Availability on Amazon EC2](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/component-configuration-examples-hana-ha.html).

## Using the cluster alert agents
<a name="_using_the_cluster_alert_agents"></a>

Within the cluster configuration, you can call an external program (an alert agent) to handle alerts. This is a *push* notification. It passes information about the event via environment variables.

The agents can then be configured to send emails, log to a file, update a monitoring system, etc. For example, the following script can be used to access Amazon SNS.

```
#!/bin/sh

# alert_sns.sh
# modified from /usr/share/pacemaker/alerts/alert_smtp.sh.sample

##############################################################################
# SETUP
# * Create an SNS Topic and subscribe email or chatbot
# * Note down the ARN for the SNS topic
# * Give the IAM Role attached to both Instances permission to publish to the SNS Topic
# * Ensure the aws cli is installed
# * Copy this file to /usr/share/pacemaker/alerts/alert_sns.sh or other location on BOTH nodes
# * Ensure the permissions allow for hacluster and root to execute the script
# * Run the following as root (modify file location if necessary and replace SNS ARN):
#
# SLES:
# crm configure alert aws_sns_alert /usr/share/pacemaker/alerts/alert_sns.sh meta timeout=30s timestamp-format="%Y-%m-%d_%H:%M:%S" to <{ arn:aws:sns:region:account-id:myPacemakerAlerts  }>
#
# RHEL:
# pcs alert create id=aws_sns_alert path=/usr/share/pacemaker/alerts/alert_sns.sh meta timeout=30s timestamp-format="%Y-%m-%d_%H:%M:%S"
# pcs alert recipient add aws_sns_alert value=arn:aws:sns:region:account-id:myPacemakerAlerts
##############################################################################

# Additional information to send with the alerts
node_name=`uname -n`
sns_body=`env | grep CRM_alert_`

# Required for SNS
TOKEN=$(/usr/bin/curl --noproxy '*' -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Get metadata
REGION=$(/usr/bin/curl --noproxy '*' -w "\n" -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | awk -F\" '{print $4}')

sns_subscription_arn=${CRM_alert_recipient}

# Format depending on alert type
case ${CRM_alert_kind} in
   node)
     sns_subject="${CRM_alert_timestamp} ${cluster_name}: Node '${CRM_alert_node}' is now '${CRM_alert_desc}'"
   ;;
   fencing)
     sns_subject="${CRM_alert_timestamp} ${cluster_name}: Fencing ${CRM_alert_desc}"
   ;;
   resource)
     if [ ${CRM_alert_interval} = "0" ]; then
         CRM_alert_interval=""
     else
         CRM_alert_interval=" (${CRM_alert_interval})"
     fi
     if [ ${CRM_alert_target_rc} = "0" ]; then
         CRM_alert_target_rc=""
     else
         CRM_alert_target_rc=" (target: ${CRM_alert_target_rc})"
     fi
     case ${CRM_alert_desc} in
         Cancelled)
           ;;
         *)
           sns_subject="${CRM_alert_timestamp}: Resource operation '${CRM_alert_task}${CRM_alert_interval}' for '${CRM_alert_rsc}' on '${CRM_alert_node}': ${CRM_alert_desc}${CRM_alert_target_rc}"
           ;;
     esac
     ;;
   attribute)
     sns_subject="${CRM_alert_timestamp}: The '${CRM_alert_attribute_name}' attribute of the '${CRM_alert_node}' node was updated in '${CRM_alert_attribute_value}'"
     ;;
   *)
     sns_subject="${CRM_alert_timestamp}: Unhandled $CRM_alert_kind alert"
     ;;
esac

# Use this information to send the email
aws sns publish --topic-arn "${sns_subscription_arn}" --subject "${sns_subject}" --message "${sns_body}" --region ${REGION}
```