# Operations
<a name="rhel-netweaver-ha-operations"></a>

This section covers the following topics.

**Topics**
+ [Viewing the cluster state](cluster-state-nw-rhel.md)
+ [Performing planned maintenance](planned-maintenance-nw-rhel.md)
+ [Post-failure analysis and reset](analysis-reset-nw-rhel.md)
+ [Alerting and monitoring](alerting-monitoring-nw-rhel.md)

# Viewing the cluster state
<a name="cluster-state-nw-rhel"></a>

You can view the state of the cluster in two ways - based on your operating system or with a web based console provided by Red Hat.

**Topics**
+ [Operating system based](#os-based-nw-rhel)
+ [Red Hat Cockpit](#rhel-cockpit)

## Operating system based
<a name="os-based-nw-rhel"></a>

There are multiple operating system commands that can be run as root or as a user with appropriate permissions. The commands enable you to get an overview of the status of the cluster and its services. See the following commands for more details.

```
# pcs status
```

Sample output:

```
rhxhost01:~ # pcs status
Cluster name: rhx-cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: rhxhost01 (version 2.1.0-8.el8-7c3f660707) - partition with quorum
  * Last updated: Tue Nov  1 13:41:58 2022
  * Last change:  Fri Oct 28 08:55:43 2022 by root via crm_attribute on rhxhost02
  * 2 nodes configured
  * 7 resource instances configured

Node List:
  * Online: [ rhxhost01 rhxhost02 ]

Full List of Resources:
  * Resource Group: grp_RHX_ASCS00:
    * rsc_ip_RHX_ASCS00 (ocf::heartbeat:aws-vpc-move-ip):        Started rhxhost01
    * rsc_sapstart_RHX_ASCS00   (ocf::heartbeat:SAPStartSrv):         Started rhxhost01
    * rsc_sap_RHX_ASCS00        (ocf::heartbeat:SAPInstance):    Started rhxhost01
  * res_AWS_STONITH     (stonith:fence_aws):  Started rhxhost02
  * Resource Group: grp_RHX_ERS10:
    * rsc_ip_RHX_ERS10  (ocf::heartbeat:aws-vpc-move-ip):        Started rhxhost02
    * rsc_sapstart_RHX_ERS10    (ocf::heartbeat:SAPStartSrv):         Started rhxhost02
    * rsc_sap_RHX_ERS10 (ocf::heartbeat:SAPInstance):    Started rhxhost02
```

The following table provides a list of useful commands.


| Command | Description | 
| --- | --- | 
|   `pcs status`   |  Display cluster status on the console  | 
|   `pcs status --full`   |  Display detailed cluster status including inactive resources  | 
|   `pcs status nodes`   |  Display node status and attributes  | 
|   `pcs status resources`   |  Display resource status and fail counts  | 
|   `pcs cluster status`   |  Display cluster daemon status  | 
|   `pcs help`   |  View more options  | 
|   `pcs status --help`   |  View more options  | 

## Red Hat Cockpit
<a name="rhel-cockpit"></a>

Cockpit is a web-based graphical user interface for managing and monitoring Red Hat Enterprise Linux systems, including pacemaker highly availability clusters. It must be enabled on every node in the cluster, to point your web browser on any node for accessing it. Use the following command to enable Cockpit.

```
# systemctl enable --now cockpit.socket
# systemctl status cockpit.socket
```

Use the following URL to check security groups for access on port 9090 from your administrative host.

```
https://your-server:9090/

e.g https://rhxhost01:9090
```

For more information, see [Configuring and Managing High Availability Clusters](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters) in the Red Hat Documentation.

# Performing planned maintenance
<a name="planned-maintenance-nw-rhel"></a>

The cluster connector is designed to integrate the cluster with SAP start framework (`sapstartsrv`), including the rolling kernel switch (RKS) awareness. Stopping and starting the SAP system using `sapcontrol` should not result in any cluster remediation activities as these actions are not interpreted as failures. Validate this scenario when testing your cluster.

There are different options to perform planned maintenance on nodes, resources, and the cluster.

**Topics**
+ [Maintenance mode](#maintenance-mode-nw-rhel)
+ [Placing a node in standby mode](#node-standby-nw-rhel)
+ [Moving a resource](#moving-resource-nw-rhel)

## Maintenance mode
<a name="maintenance-mode-nw-rhel"></a>

Use maintenance mode if you want to make any changes to the configuration or take control of the resources and nodes in the cluster. In most cases, this is the safest option for administrative tasks.

**Example**  
Use one of the following commands to turn on maintenance mode.  

```
# pcs property set maintenance-mode=true
```

```
# pcs cluster maintenance --all
```
Use one of the following commands to turn off maintenance mode.  

```
# pcs property set maintenance-mode=false
```

```
# pcs cluster maintenance --all --wait=60
```

## Placing a node in standby mode
<a name="node-standby-nw-rhel"></a>

To perform maintenance on the cluster without system outage, the recommended method for moving active resources is to place the node you want to remove from the cluster in standby mode.

```
# pcs node standby <hostname>
```

The cluster will cleanly relocate resources, and you can perform activities, including reboots on the node in standby mode. When maintenance activities are complete, you can re-introduce the node with the following command.

```
# pcs node unstandby <hostname>
```

## Moving a resource
<a name="moving-resource-nw-rhel"></a>

Moving individual resources is not recommended because of the migration or move constraints that are created to lock the resource in its new location. These can be cleared as described in the info messages, but this introduces an additional setup.

```
<rhxhost01>:~ pcs resource move grp_<RHX>_ASCS<00> <rhxhost02>
Location constraint to move resource 'grp_<RHX>_ASCS<00>' has been created
Run 'pcs resource clear grp_<RHX>_ASCS<00>' to remove this constraint
```

Use the following command once the resources have relocated to their target location.

```
# pcs resource clear grp_RHX_ASCS00
```

# Post-failure analysis and reset
<a name="analysis-reset-nw-rhel"></a>

A review must be conducted after each failure to understand the source of failure as well the reaction of the cluster. In most scenarios, the cluster prevents an application outage. However, a manual action is often required to reset the cluster to a protective state for any subsequent failures.

**Topics**
+ [Checking the logs](#checking-logs-nw-rhel)
+ [Cleanup pcs status](#cleanup-crm-nw-rhel)
+ [Restart failed nodes or pacemaker](#restart-nodes-nw-rhel)
+ [Further Analysis](#_further_analysis)

## Checking the logs
<a name="checking-logs-nw-rhel"></a>
+ For troubleshooting cluster issues, use journalctl to examine both pacemaker and corosync logs:

  ```
  # journalctl -u pacemaker -u corosync --since "1 hour ago"
  ```
  + Use `--since` to specify time periods (e.g., "2 hours ago", "today")
  + Add `-f` to follow logs in real-time
  + Combine with grep for specific searches
+ System messages and resource agent activity can be found in `/var/log/messages`.

Application based failures can be investigated in the SAP work directory.

## Cleanup pcs status
<a name="cleanup-crm-nw-rhel"></a>

If failed actions are reported using the `pcs status` command, and if they have already been investigated, then you can clear the reports with the following command.

```
# pcs resource cleanup <resource> <hostname>
```

## Restart failed nodes or pacemaker
<a name="restart-nodes-nw-rhel"></a>

It is recommended that failed (or fenced) nodes are not automatically restarted. It gives operators a chance to investigate the failure, and ensure that the cluster doesn’t make assumptions about the state of resources.

You need to restart the instance or the pacemaker service based on your approach.

## Further Analysis
<a name="_further_analysis"></a>

For cluster-specific issues, use `sosreport` to generate a targeted analysis of cluster components:

```
# sosreport --batch --tmp-dir /tmp
```

For quick analysis of recent events, you can use:

```
# pcs status --full
# journalctl -u pacemaker --since "1 hour ago"
```
+  `sosreport` collects system configuration and diagnostic information
+ For more information, see Red Hat Documentation - [What is sosreport and how to create and retrieve one](https://access.redhat.com/solutions/3592) 

# Alerting and monitoring
<a name="alerting-monitoring-nw-rhel"></a>

This section covers the following topics.

**Topics**
+ [Using Amazon CloudWatch Application Insights](#application-insights-nw-rhel)
+ [Using the cluster alert agents](#cluster-alert-nw-rhel)

## Using Amazon CloudWatch Application Insights
<a name="application-insights-nw-rhel"></a>

For monitoring and visibility of cluster state and actions, Application Insights includes metrics for monitoring enqueue replication state, cluster metrics, and SAP and high availability checks. Additional metrics, such as EFS and CPU monitoring can also help with root cause analysis.

For more information, see [Get started with Amazon CloudWatch Application Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/appinsights-getting-started.html) and [SAP NetWeaver High Availability on Amazon EC2](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/component-configuration-examples-netweaver-ha.html).

## Using the cluster alert agents
<a name="cluster-alert-nw-rhel"></a>

Within the cluster configuration, you can call an external program (an alert agent) to handle alerts. This is a *push* notification. It passes information about the event via environment variables.

The agents can then be configured to send emails, log to a file, update a monitoring system, etc. For example, the following script can be used to access Amazon SNS.

```
#!/bin/sh

# alert_sns.sh
# modified from /usr/share/pacemaker/alerts/alert_smtp.sh.sample

##############################################################################
# SETUP
# * Create an SNS Topic and subscribe email or chatbot
# * Note down the ARN for the SNS topic
# * Give the IAM Role attached to both Instances permission to publish to the SNS Topic
# * Ensure the aws cli is installed
# * Copy this file to /usr/share/pacemaker/alerts/alert_sns.sh or other location on BOTH nodes
# * Ensure the permissions allow for hacluster and root to execute the script
# * Run the following as root (modify file location if necessary and replace SNS ARN):
#
# SLES:
# crm configure alert aws_sns_alert /usr/share/pacemaker/alerts/alert_sns.sh meta timeout=30s timestamp-format="%Y-%m-%d_%H:%M:%S" to <{ arn:aws:sns:region:account-id:myPacemakerAlerts  }>
#
# RHEL:
# pcs alert create id=aws_sns_alert path=/usr/share/pacemaker/alerts/alert_sns.sh meta timeout=30s timestamp-format="%Y-%m-%d_%H:%M:%S"
# pcs alert recipient add aws_sns_alert value=arn:aws:sns:region:account-id:myPacemakerAlerts
##############################################################################

# Additional information to send with the alerts
node_name=`uname -n`
sns_body=`env | grep CRM_alert_`

# Required for SNS
TOKEN=$(/usr/bin/curl --noproxy '*' -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Get metadata
REGION=$(/usr/bin/curl --noproxy '*' -w "\n" -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | awk -F\" '{print $4}')

sns_subscription_arn=${CRM_alert_recipient}

# Format depending on alert type
case ${CRM_alert_kind} in
   node)
     sns_subject="${CRM_alert_timestamp} ${cluster_name}: Node '${CRM_alert_node}' is now '${CRM_alert_desc}'"
   ;;
   fencing)
     sns_subject="${CRM_alert_timestamp} ${cluster_name}: Fencing ${CRM_alert_desc}"
   ;;
   resource)
     if [ ${CRM_alert_interval} = "0" ]; then
         CRM_alert_interval=""
     else
         CRM_alert_interval=" (${CRM_alert_interval})"
     fi
     if [ ${CRM_alert_target_rc} = "0" ]; then
         CRM_alert_target_rc=""
     else
         CRM_alert_target_rc=" (target: ${CRM_alert_target_rc})"
     fi
     case ${CRM_alert_desc} in
         Cancelled)
           ;;
         *)
           sns_subject="${CRM_alert_timestamp}: Resource operation '${CRM_alert_task}${CRM_alert_interval}' for '${CRM_alert_rsc}' on '${CRM_alert_node}': ${CRM_alert_desc}${CRM_alert_target_rc}"
           ;;
     esac
     ;;
   attribute)
     sns_subject="${CRM_alert_timestamp}: The '${CRM_alert_attribute_name}' attribute of the '${CRM_alert_node}' node was updated in '${CRM_alert_attribute_value}'"
     ;;
   *)
     sns_subject="${CRM_alert_timestamp}: Unhandled $CRM_alert_kind alert"
     ;;
esac

# Use this information to send the email.
aws sns publish --topic-arn "${sns_subscription_arn}" --subject "${sns_subject}" --message "${sns_body}" --region ${REGION}
```