# Cluster configuration
This section covers the following topics.
**Topics**
+ [Cluster resources](rhel-ase-ha-cluster-resources.md)
+ [Sample configuration (pcs config show)](rhel-ase-sample-configuration.md)
# Cluster resources
This section covers the following topics.
**Topics**
+ [Enable and start the cluster](#start-cluster)
+ [Increase corosync totem timeout](#increase-timeout)
+ [Check cluster status](#cluster-status)
+ [Prepare for resource creation](#resource-creation)
+ [Cluster bootstrap](#cluster-bootstrap)
+ [Create `fence_aws` STONITH resource](#create-stonith)
+ [Create file system resources](#filesystem-resources)
+ [Create overlay IP resources](#overlay-ip-resources)
+ [Create SAP ASE database resource](#ase-database-resource)
+ [Activate cluster](#activate-cluster)
## Enable and start the cluster
This is applicable to both cluster nodes. Run the following command to enable and start the `pacemaker` cluster service on both nodes.
```
pcs cluster enable --all
rhxdbhost01: Cluster Enabled
rhxdbhost02: Cluster Enabled
pcs cluster start --all
rhxdbhost01: Starting Cluster...
rhxdbhost02: Starting Cluster...
```
By enabling the `pacemaker` service, the server automatically joins the cluster after a reboot. This ensures that your system is protected. Alternatively, you can start the `pacemaker` service manually on boot. You can then investigate the cause of failure. However, this is generally not required for SAP NetWeaver ASCS cluster.
## Increase corosync totem timeout
**RHEL 7.x**
1. Edit the `/etc/corosync/corosync.conf` file in all cluster nodes to increase the token value or to add a value if it is not present.
```
totem {
version: 2
secauth: off
cluster_name: my-rhel-sap-cluster
transport: udpu
rrp_mode: passive
token: 29000 <------ Value to be set
}
```
1. Reload the corosync with the following command, on any one of the cluster nodes. This does not cause any downtime.
```
pcs cluster reload corosync
```
1. Use the following command to confirm if the changes are active.
```
corosync-cmapctl | grep totem.token
Runtime.config.totem.token (u32) = 29000
```
**RHEL 8.x**
Use the following command to increase the token value or to add a value if it is not present.
```
pcs cluster config update totem token=29000
```
## Check cluster status
Once the cluster service `pacemaker` is started, check the cluster status with `pcs status` command, as shown in the following example. Both the primary (`rhxdbhost01`) and secondary (`rhxdbhost02`) servers should be seen as online.
```
pcs status
Cluster name: rhelha
WARNINGS:
No stonith devices and stonith-enabled is not false
Cluster Summary:
* Stack: corosync
* Current DC: rhxdbhost01 (version 2.0.3-5.el8_2.5-4b1f869f0f) - partition with quorum
* Last updated: Tue Jan 10 21:32:15 2023
* Last change: Tue Jan 10 19:46:50 2023 by hacluster via crmd on rhxdbhost01
* 2 nodes configured
* 0 resource instances configured
Node List:
* Online: [ rhxdbhost01 rhxdbhost02 ]
Full List of Resources:
* No resources
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
```
## Prepare for resource creation
To ensure that the cluster does not perform any unexpected actions during setup of resources and configuration, set the maintenance mode to true.
Run the following command to put the cluster in maintenance mode.
```
pcs property set maintenance-mode=true
```
## Cluster bootstrap
Configure the cluster bootstrap parameters by running the following commands.
```
pcs resource defaults update resource-stickiness=1
pcs resource defaults update migration-threshold=3
```
## Create `fence_aws` STONITH resource
Modify the commands in the following table to match your configuration values.
```
pcs stonith create rsc_aws_stonith_ fence_aws region=us-east-1 pcmk_host_map="rhxdbhost01:i-xxxxinstidforhost1;rhxdbhost02:i-xxxxinstidforhost2" power_timeout=240 pcmk_reboot_timeout=300 pcmk_reboot_retries=2 pcmk_delay_max=30 pcmk_reboot_action=reboot op start timeout=180 op stop timeout=180 op monitor interval=180 timeout=60
```
**Note**
The default `pcmk` action is reboot. If you want to have the instance remain in a stopped state until it has been investigated, and then manually started again, add `pcmk_reboot_action=off`. Any High Memory (`u-tb1. `) instance or metal instance running on a dedicated host won’t support reboot, and will require `pcmk_reboot_action=off`.
## Create file system resources
Mounting and unmounting file system resources to align with the location of SAP ASE database is done using cluster resources.
Modify and run the following commands to create these file system resources.
**/sybase**
```
pcs resource create rsc_fs__sybase ocf:heartbeat:Filesystem params device=":/sybase" directory="/sybase" fstype="nfs4" options=" rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2" op start timeout=60s interval=0 op stop timeout=60s interval=0 op monitor interval=20s timeout=40s
```
**/sybase//sapdata\$11**
```
pcs resource create rsc_fs__data ocf:heartbeat:Filesystem params device=":/asedata" directory="/sybase//sapdata_1" fstype="nfs4"
options="rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=8,timeo=600,retrans=2" op start timeout=60s interval=0 op stop timeout=60s interval=0 op monitor interval=20s timeout=40s
```
**/sybase//saplog\$11**
```
pcs resource create rsc_fs__log ocf:heartbeat:Filesystem params device=":/aselog" directory="/sybase//saplog_1" fstype="nfs4"
options="rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2" op start timeout=60s interval=0 op stop timeout=60s interval=0 op monitor interval=20s timeout=40s
```
**/sybase//sapdiag**
```
pcs resource create rsc_fs__diag ocf:heartbeat:Filesystem params device=":/sapdiag" directory="/sybase//sapdiag" fstype="nfs4"
options="rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2" op start timeout=60s interval=0 op stop timeout=60s interval=0 op monitor interval=20s timeout=40s
```
**/sybase//saptmp**
```
pcs resource create rsc_fs__tmp ocf:heartbeat:Filesystem params device=":/saptmp" directory="/sybase//saptmp" fstype="nfs4"
options="rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2" op start timeout=60s interval=0 op stop timeout=60s interval=0 op monitor interval=20s timeout=40s
```
**/sybasebackup**
```
pcs resource create rsc_fs__bkp ocf:heartbeat:Filesystem params device=":/sybasebackup" directory="/backup" fstype="nfs4"
options="rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2" op start timeout=60s interval=0 op stop timeout=60s interval=0 op monitor interval=20s timeout=40s
```
**/usr/sap**
```
pcs resource create rsc_fs__sap ocf:heartbeat:Filesystem params device=":/usrsap" directory="/usr/sap" fstype="nfs4"
options="rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2" op start timeout=60s interval=0 op stop timeout=60s interval=0 op monitor interval=20s timeout=40s
```
**Notes**
+ Review the mount options to ensure that they match with your operating system, NFS file system type, and the latest recommendations from SAP and AWS.
+ must be the alias of the FSx for ONTAP resource. For example, `fs-xxxxxx.efs.xxxxxx.amazonaws.com`.
+ It is important to create the resources in the proper mount order/sequence.
## Create overlay IP resources
The IP resource provides the details necessary to update the route table entry for overlay IP.
Use the following command to create an SAP ASE database IP resource.
```
pcs resource create rsc_ip__ASEDB ocf:heartbeat:aws-vpc-move-ip ip=172.16.0.23 interface=eth0 routing_table=rtb-xxxxxroutetable1 op monitor interval=20s timeout=40s --group rsc_asedb_group
```
**Notes**
+ If more than one route table is required for connectivity or because of subnet associations, the `routing_table` parameter can have multiple values separated by a comma. For example, `routing_table=rtb-xxxxxroutetable1, rtb-xxxxxroutetable2`.
+ Additional parameters – `lookup_type` and `routing_table_role` are required for shared VPC. For more information, see [Shared VPC – optional](https://docs.aws.amazon.com/sap/latest/sap-netweaver/rhel-netweaver-ha-settings.html#rhel-netweaver-ha-shared-vpc).
## Create SAP ASE database resource
SAP ASE database is started and stopped using cluster resources.
Modify and run the following command to create the `SAPDatabase` resource.
```
pcs resource create rsc_ase__ASEDB ocf:heartbeat:SAPDatabase SID= DBTYPE=SYB STRICT_MONITORING=TRUE op start timeout=300 op stop timeout=300 --group grp__ASEDB
```
Use the following command for more details on the resource.
```
pcs resource describe ocf:heartbeat:SAPDatabase
```
## Activate cluster
Use `pcs config show` to review that all the values have been entered correctly.
On confirmation of correct values, set the maintenance mode to false using the following command. This enables the cluster to take control of the resources.
```
pcs property set maintenance-mode=false
```
See the [Sample configuration](https://docs.aws.amazon.com/sap/latest/sap-netweaver/rhel-sample-configuration.html).
# Sample configuration (pcs config show)
The following sample configuration is based on ENSA2.
```
pcs config show
Cluster Name: rhelha
Corosync Nodes:
rhxdbhost01 rhxdbhost02
Pacemaker Nodes:
rhxdbhost01 rhxdbhost02
Resources:
Group: rsc_asedb_group
Meta Attrs: resource-stickiness=5000
Resource: rsc_vip_asedb (class=ocf provider=heartbeat type=aws-vpc-move-ip)
Attributes: interface=eth0 ip=172.16.0.23 routing_table=rtb-0b3f1d6196f45300d
Operations: monitor interval=60s timeout=30s (rsc_vip_asedb-monitor-interval-60s)
start interval=0s timeout=180s (rsc_vip_asedb-start-interval-0s)
stop interval=0s timeout=180s (rsc_vip_asedb-stop-interval-0s)
Resource: rsc_fs_sybase (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=svm-09794aeece44cc025.fs-04af26e8311974f41.fsx.us-east-1.amazonaws.com:/sybase directory=/sybase force_unmount=safe fstype=nfs4 options=rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2
Operations: monitor interval=20s timeout=40s (rsc_fs_sybase-monitor-interval-20s)
start interval=0s timeout=60s (rsc_fs_sybase-start-interval-0s)
stop interval=0s timeout=60s (rsc_fs_sybase-stop-interval-0s)
Resource: rsc_fs_data (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=svm-01c02d046ae5a24a2.fs-04af26e8311974f41.fsx.us-east-1.amazonaws.com:/asedata directory=/sybase/ARD/sapdata_1 force_unmount=safe fstype=nfs4 options=rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=8,timeo=600,retrans=2
Operations: monitor interval=20s timeout=40s (rsc_fs_data-monitor-interval-20s)
start interval=0s timeout=60s (rsc_fs_data-start-interval-0s)
stop interval=0s timeout=60s (rsc_fs_data-stop-interval-0s)
Resource: rsc_fs_log (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=svm-04cd525dbd0b354d2.fs-04af26e8311974f41.fsx.us-east-1.amazonaws.com:/aselog directory=/sybase/ARD/saplog_1 force_unmount=safe fstype=nfs4 options=rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2
Operations: monitor interval=20s timeout=40s (rsc_fs_log-monitor-interval-20s)
start interval=0s timeout=60s (rsc_fs_log-start-interval-0s)
stop interval=0s timeout=60s (rsc_fs_log-stop-interval-0s)
Resource: rsc_fs_sapdiag (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=svm-09794aeece44cc025.fs-04af26e8311974f41.fsx.us-east-1.amazonaws.com:/sapdiag directory=/sybase/ARD/sapdiag force_unmount=safe fstype=nfs4 options=rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2
Operations: monitor interval=20s timeout=40s (rsc_fs_sapdiag-monitor-interval-20s)
start interval=0s timeout=60s (rsc_fs_sapdiag-start-interval-0s)
stop interval=0s timeout=60s (rsc_fs_sapdiag-stop-interval-0s)
Resource: rsc_fs_saptmp (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=svm-09794aeece44cc025.fs-04af26e8311974f41.fsx.us-east-1.amazonaws.com:/saptmp directory=/sybase/ARD/saptmp force_unmount=safe fstype=nfs4 options=rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2
Operations: monitor interval=20s timeout=40s (rsc_fs_saptmp-monitor-interval-20s)
start interval=0s timeout=60s (rsc_fs_saptmp-start-interval-0s)
stop interval=0s timeout=60s (rsc_fs_saptmp-stop-interval-0s)
Resource: rsc_fs_backup (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=svm-09794aeece44cc025.fs-04af26e8311974f41.fsx.us-east-1.amazonaws.com:/backup directory=/sybasebackup force_unmount=safe fstype=nfs4 options=rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2
Operations: monitor interval=20s timeout=40s (rsc_fs_backup-monitor-interval-20s)
start interval=0s timeout=60s (rsc_fs_backup-start-interval-0s)
stop interval=0s timeout=60s (rsc_fs_backup-stop-interval-0s)
Resource: rsc_fs_usrsap (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=svm-09794aeece44cc025.fs-04af26e8311974f41.fsx.us-east-1.amazonaws.com:/usrsap directory=/usr/sap force_unmount=safe fstype=nfs4 options=rw,noatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2
Operations: monitor interval=20s timeout=40s (rsc_fs_usrsap-monitor-interval-20s)
start interval=0s timeout=60s (rsc_fs_usrsap-start-interval-0s)
stop interval=0s timeout=60s (rsc_fs_usrsap-stop-interval-0s)
Resource: sybaseARD (class=ocf provider=heartbeat type=SAPDatabase)
Attributes: DBTYPE=SYB SID=ARD STRICT_MONITORING=TRUE
Operations: methods interval=0s timeout=5s (sybaseARD-methods-interval-0s)
monitor interval=120s timeout=60s (sybaseARD-monitor-interval-120s)
start interval=0s timeout=300 (sybaseARD-start-interval-0s)
stop interval=0s timeout=300 (sybaseARD-stop-interval-0s)
Stonith Devices:
Resource: clusterfence (class=stonith type=fence_aws)
Attributes: pcmk_delay_max=45 pcmk_host_map=rhxdbhost01:i-03939ad3f07e14e3f;rhxdbhost02:i-09f138e3a1290bfde pcmk_reboot_action=off pcmk_reboot_retries=4 pcmk_reboot_timeout=600 power_timeout=240 region=us-east-1
Operations: monitor interval=300 timeout=60 (clusterfence-monitor-interval-300)
start interval=0s timeout=600 (clusterfence-start-interval-0s)
Fencing Levels:
Location Constraints:
Ordering Constraints:
Colocation Constraints:
Ticket Constraints:
Alerts:
No alerts defined
Resources Defaults:
Meta Attrs: rsc_defaults-meta_attributes
migration-threshold=1
Operations Defaults:
No defaults set
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: rhelha
dc-version: 2.1.2-4.el8_6.7-ada5c3b36e2
have-watchdog: false
last-lrm-refresh: 1693394303
maintenance-mode: false
Tags:
No tags defined
Quorum:
Options:
```