# Planning This section covers the following topics. **Topics** + [Setup Overview](sap-nw-pacemaker-sles-setup-overview.md) + [Deployment Guidance](sap-nw-pacemaker-sles-references.md) + [Concepts](sap-nw-pacemaker-sles-concepts.md) + [Parameter Reference](sap-nw-pacemaker-sles-parameters.md) + [Architecture diagrams](sles-netweaver-ha-diagrams.md) # Setup Overview You must meet the following prerequisites before commencing setup. **Topics** + [Deployed Cluster Infrastructure](#cluster-nw-sles) + [Supported Operating System](#supported-os-nw-sles) + [SAP and SUSE references](#references-nw-sles) + [Required Access for Setup](#access-nw-sles) + [Reliability Requirements Defined](#reliability-nw-sles) ## Deployed Cluster Infrastructure Ensure that your AWS networking requirements and Amazon EC2 instances where SAP workloads are installed, are correctly configured for SAP. For more information, see [SAP NetWeaver Environment Setup for Linux on AWS](https://docs.aws.amazon.com/sap/latest/sap-netweaver/std-sap-netweaver-environment-setup.html). See the following ASCS cluster specific requirements. + Two cluster nodes created in private subnets in separate Availability Zones within the same Amazon VPC and AWS Region + Access to the route table(s) that are associated with the chosen subnets For more information, see [AWS – Overlay IP](sap-nw-pacemaker-sles-concepts.md#overlay-ip-sles). + Amazon EC2 instances must have connectivity to the Amazon EC2 endpoint via either internet or an Amazon VPC endpoint. ## Supported Operating System Protecting the ABAP SAP Central Services (ASCS) with a pacemaker cluster requires packages from SUSE, including targeted cluster resource agents for SAP and AWS that may not be available in standard repositories. For deploying SAP applications on SUSE, SAP and SUSE recommend using SUSE Linux Enterprise Server for SAP applications (SLES for SAP). SLES for SAP provides additional benefits, including Extended Service Pack Overlap Support (ESPOS), configuration and tuning packages for SAP applications, and High Availability Extensions (HAE). For more details, see SUSE website at [SUSE Linux Enterprise Server for SAP Applications](https://www.suse.com/products/sles-for-sap/). SLES for SAP is available at [AWS Marketplace](https://aws.amazon.com/marketplace) with an hourly or annual subscription. You can also use the bring your own subscription (BYOS) model. ## SAP and SUSE references In addition to this guide, see the following references for more details. + [SUSE documentation – SAP S/4 HANA - Enqueue Replication 2 High Availability Cluster With Simple Mount](https://documentation.suse.com/sbp/sap-15/html/SAP-S4HA10-setupguide-simplemount-sle15/index.html) + [SUSE documentation – SAP S/4 HANA - Enqueue Replication 2 High Availability Cluster](https://documentation.suse.com/sbp/all/single-html/SAP-S4HA10-setupguide-sle15/#id-1) + [SAP Note: 1656099 - SAP Applications on AWS: Supported DB/OS and Amazon EC2 products](https://me.sap.com/notes/1656099) + [SAP Note: 1984787 - SUSE Linux Enterprise Server 12: Installation Notes](https://me.sap.com/notes/1984787) + [SAP Note: 2578899 - SUSE Linux Enterprise Server 15: Installation Notes](https://me.sap.com/notes/2578899) + [SAP Note: 1275776 - Linux: Preparing SLES for SAP environments](https://me.sap.com/notes/1275776) You must have SAP portal access for reading all SAP Notes. ## Required Access for Setup The following access is required for setting up the cluster. + An IAM user with the following privileges. + modify Amazon VPC route tables + modify Amazon EC2 instance properties + create IAM policies and roles + create Amazon EFS file systems + Root access to the operating system of both cluster nodes + SAP administrative user access – `adm` In case of a new install, this user is created by the install process. ## Reliability Requirements Defined The SAP Lens of the Well-Architected framework, in particular the Reliability pillar, can be used to understand the reliability requirements for your SAP workload. The ASCS is a single point of failure in a highly available SAP architecture. The impact of an outage of this component must be evaluated against factors, such as, recovery point objective (RPO), recovery time objective (RTO), cost and operation complexity. For more information, see [Reliability](https://docs.aws.amazon.com/wellarchitected/latest/sap-lens/reliability.html) in SAP Lens - AWS Well-Architected Framework. # Deployment Guidance The following section details the documentation and deployment guidance from SUSE. **Topics** + [Deployment Patterns](#deployments-sles) + [Automated Deployment](#automation-nw-sles) + [Pacemaker - simple-mount and classic architecture](#simple-classic-nw-sles) ## Deployment Patterns The following table outlines the supported SAP deployment types and their corresponding AWS configuration patterns for high availability clustering. | SAP Deployment Type | Support Status | AWS Configuration Patterns | Notes | | --- | --- | --- | --- | | SAP NetWeaver ASCS/ERS (ENSA1) | AWS Documented & Supported | SAPNetweaver-Classic, SAPNetweaver-Simple-mount | | | SAP NetWeaver ASCS/ERS (ENSA2) | AWS Documented & Supported | SAPNetweaver-Classic, SAPNetweaver-Simple-mount | | | SAP S/4HANA ASCS/ERS | AWS Documented & Supported | SAPNetweaver-Classic, SAPNetweaver-Simple-mount | S/4HANA only supports ERS2 | | SAP SCS (Java) | Vendor Documented & Supported | | Follows SAP Documentation | ## Automated Deployment You can set up a cluster manually using the instructions provided here. You can also automate parts of this process to ensure consistency and repeatability. Use AWS Launch Wizard for SAP for automated deployments of SAP NetWeaver, SAP S/4 HANA, SAP B/4HANA, and Solution Manager. Launch Wizard uses AWS CloudFormation scripts to quickly provision the resources needed to deploy SAP NetWeaver and S/4 HANA. The automation performs SAP enqueue replication and pacemaker setup so that only validation and testing are required. For more information, see [AWS Launch Wizard for SAP](https://docs.aws.amazon.com/launchwizard/latest/userguide/launch-wizard-sap.html). To ensure that the behavior and operation of your cluster is well understood regardless of how your system is set up, we recommend a thorough test cycle. See [Testing](https://docs.aws.amazon.com/sap/latest/sap-netweaver/testing.html) for more details. ## Pacemaker - simple-mount and classic architecture This guide covers two architectures for SAP cluster solutions on SLES for SAP – simple-mount and classic (previous standard). Simple-mount was certified as the SLES for SAP Applications cluster solution in late 2021. It is now the recommended architecture for both ENSA1 and ENSA2 deployments running on SLES for SAP 15 and above. For more details, see SUSE blog [Simple Mount Structure for SAP Application Platform](https://www.suse.com/c/simple-mount-structure-for-sap-application-platform/). If you are configuring a new SAP installation, we recommend the simple-mount architecture. If you already have the classic architecture, and wish to migrate to the simple-mount architecture, see [Switching architecture to simple-mount](#switching-architecture-sles). The following are the differences between the classic and simple-mount architectures. + Removing file system resources from cluster – a file system is required but it is not mounted and unmounted by the cluster. The executable directory for the ASCS and ERS can be permanently mounted on both nodes. + Addition of SAPStartSrv – SAPStartSrv controls the matching SAPStartSrv framework process. + Sapping and sappong services – these services manage the start of SAPStartSrv services with sapinit. See the [Architecture diagrams](https://docs.aws.amazon.com/sap/latest/sap-netweaver/sles-netweaver-ha-diagrams.html) for more details. ### Switching architecture to simple-mount Follow along these steps if you want to switch an existing cluster with classic architecture to use the recommended configuration of simple-mount architecture. These steps must be performed in an outage window, allowing stop/start of services and basic testing. 1. Put the cluster in maintenance mode. See [Maintenance mode](planned-maintenance-nw-sles.md#maintenance-mode-nw-sles) 1. Stop SAP services, including application servers connected to the cluster as well as ASCS and ERS. 1. Install any missing operating system packages. See [Install Missing Operating System Packages](sap-nw-pacemaker-sles-os-settings.md#packages-nw-sles). It might be necessary to install `sapstartsrv-resource-agents`. However, all operating system prerequisites must be checked and updated to ensure that versions are compatible. 1. Add entries for ASCS and ERS mount point on both nodes (if not already added). See [Update /etc/fstab](sap-shared-filesystems-nw-sles.md#update-fstab-nw-sles) 1. Enable `sapping`/`sappong` services. See [Enable sapping and sappong Services (Simple-Mount Only)](sap-ascs-service-control-nw-sles.md#sapping-sappong-services-nw-sles) 1. Align and disable `systemd` services. See [Ensure ASCS and ERS SAP Services can run on either node (systemd)](sap-ascs-service-control-nw-sles.md#modify-sapservices-nw-sles) 1. Backup the configuration with the following command. ``` # crm config show >> /tmp/classic_ha_setup.txt ``` See [Prepare for Resource Creation](cluster-config-nw-sles.md#prepare-resource-nw-sles) 1. *Optional* – delete the configuration. You can edit in place but we recommend starting with a blank configuration. This ensures that latest timeout and priority parameters are in place. See [Reset Configuration – Optional](cluster-config-nw-sles.md#reset-config-nw-sles) ``` # crm config erase # crm config show ``` 1. Configure cluster resources again. 1. Check the cluster and perform some tests. 1. Resume standard operations by starting any additional services, including application servers. # Concepts This section covers AWS, SAP, and SUSE concepts. **Topics** + [SAP – ABAP SAP Central Services (ASCS)](#ascs-nw-sles) + [SAP – Enqueue Replication Server (ERS)](#ers-nw-sles) + [AWS – Availability Zones](#availability-zones-nw-sles) + [AWS – Overlay IP](#overlay-ip-sles) + [AWS – Shared VPC](#shared-vpc) + [Pacemaker - STONITH fencing agent](#stonith-nw-sles) ## SAP – ABAP SAP Central Services (ASCS) The ABAP SAP Central Services (ASCS) is an SAP instance consisting of the following two services. It is considered a single point of failure (SPOF) in a resilient SAP architecture. + **Message server** – Responsible for application load distribution (GUI and RFC), communication between application servers, and centralised configuration information for web dispatchers and application servers. + **Enqueue server (standalone)** – Maintains a lock table in main memory (shared memory). Unlike a database lock, an enqueue lock can exist across multiple logical units of work (LUW), and is set by a SAP Dialog work process. The lock mechanism prevents two transactions from changing the same data in the database simultaneously. **Note** With ABAP Release 7.53 (ABAP Platform 1809), the new Standalone Enqueue Server 2 (ENSA2) is installed by default. It replaces the previous version (ENSA1) but can be configured for the previous versions. See [SAP Note 2630416 - Support for Standalone Enqueue Server 2](https://me.sap.com/notes/2630416) (SAP portal access required) for more information. This document includes modifications to align with the correct ENSA version. ## SAP – Enqueue Replication Server (ERS) The Enqueue Replication Server (ERS) is an SAP instance containing a replica of the lock table (replication table). In a resilient setup, if the standalone enqueue server (EN/ENQ) fails, it can be restarted either by restart parameters or by high availability software, such as Pacemaker. The enqueue server retrieves the replication table remotely or by failing over to the host where the ERS is running. ## AWS – Availability Zones Availability Zone is one or more discreet data centers with redundant power, networking, and connectivity in an AWS Region. For more information, see [Regions and Availability Zones](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/). For mission critical deployments of SAP on AWS where the goal is to minimise the recovery time objective (RTO), we suggest distributing single points of failure across Availability Zones. Compared with single instance or single Availability Zone deployments, this increases resilience and isolation against a broad range of failure scenarios and issues, including natural disasters. Each Availability Zone is physically separated by a meaningful distance (many kilometers) from another Availability Zone. All Availability Zones in an AWS Region are interconnected with high-bandwidth, low-latency network, over fully redundant, dedicated metro fiber. This enables synchronous replication. All traffic between Availability Zones is encrypted. ## AWS – Overlay IP Overlay IP enables a connection to the application, regardless of which Availability Zone (and subnet) contains the active primary node. When deploying an Amazon EC2 instance in AWS, IP addresses are allocated from the CIDR range of the allocated subnet. The subnet cannot span across multiple Availability Zones, and therefore the subnet IP addresses may be unavailable after faults, including network connectivity or hardware issues which require a failover to the replication target in a different Availability Zone. To address this, we suggest that you configure an overlay IP, and use this in the connection parameters for the application. This IP address is a non-overlapping RFC1918 private IP address from outside of VPC CIDR block and is configured as an entry in the route table or tables. The route directs the connection to the active node and is updated during a failover by the cluster software. You can select any one of the following RFC1918 private IP addresses for your overlay IP address. + 10.0.0.0 – 10.255.255.255 (10/8 prefix) + 172.16.0.0 – 172.31.255.255 (172.16/12 prefix) + 192.168.0.0 – 192.168.255.255 (192.168/16 prefix) If, for example, you use the 10/8 prefix in your SAP VPC, selecting a 172 or a 192 IP address may help to differentiate the overlay IP. Consider the use of an IP Address Management (IPAM) tool such as Amazon VPC IP Address Manager to plan, track, and monitor IP addresses for your AWS workloads. For more information, see [What is IPAM?](https://docs.aws.amazon.com/vpc/latest/ipam/what-it-is-ipam.html) The overlay IP agent in the cluster can also be configured to update multiple route tables which contain the Overlay IP entry if your subnet association or connectivity requires it. **Access to overlay IP** The overlay IP is outside of the range of the VPC, and therefore cannot be reached from locations that are not associated with the route table, including on-premises and other VPCs. Use [AWS Transit Gateway](https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html) as a central hub to facilitate the network connection to an overlay IP address from multiple locations, including Amazon VPCs, other AWS Regions, and on-premises using [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) or [AWS Client VPN](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/what-is.html). If you do not have AWS Transit Gateway set up as a network transit hub or if it is not available in your preferred AWS Region, you can use a [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) to enable network access to an overlay IP. For more information, see [SAP on AWS High Availability with Overlay IP Address Routing](https://docs.aws.amazon.com/sap/latest/sap-hana/sap-ha-overlay-ip.html). ## AWS – Shared VPC An enterprise landing zone setup or security requirements may require the use of a separate cluster account to restrict the route table access required for the Overlay IP to an isolated account. For more information, see [Share your VPC with other accounts](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-sharing.html). Evaluate the operational impact against your security posture before setting up shared VPC. To set up, see [Shared VPC – optional](https://docs.aws.amazon.com/sap/latest/sap-netweaver/sles-netweaver-ha-settings.html#sles-netweaver-ha-shared-vpc). ## Pacemaker - STONITH fencing agent In a two-node cluster setup for a primary resource and its replication pair, it is important that there is only one node in the primary role with the ability to modify your data. In the event of a failure scenario where a node is unresponsive or incommunicable, ensuring data consistency requires that the faulty node is isolated by powering it down before the cluster commences other actions, such as promoting a new primary. This arbitration is the role of the fencing agent. Since a two-node cluster introduces the possibility of a fence race in which a dual shoot out can occur with communication failures resulting in both nodes simultaneously claiming, "I can’t see you, so I am going to power you off". The fencing agent is designed to minimise this risk by providing an external witness. SLES supports several fencing agents, including the one recommended for use with Amazon EC2 Instances (external/ec2). This resource uses API commands to check its own instance status - "Is my instance state anything other than running?" before proceeding to power off its pair. If it is already in a stopping or stopped state it will admit defeat and leave the surviving node untouched. # Parameter Reference The cluster setup relies on the following parameters. Gather this information prior to configuring Pacemaker to ensure a smooth setup process. **Topics** + [Global AWS parameters](#global-aws-parameters-nw-sles) + [Amazon EC2 instance parameters](#ec2-parameters-nw-sles) + [SAP Instance Parameters](#sap-pacemaker-resource-parameters-nw-sles) + [Pacemaker Parameters](#sles-cluster-parameters) ## Global AWS parameters | Name | Parameter | Example | | --- | --- | --- | | AWS account ID | `` | `123456789100` | | AWS Region | `` | `us-east-1` | + AWS account – For more details, see [Your AWS account ID and its alias](https://docs.aws.amazon.com/IAM/latest/UserGuide/console-account-alias.html). + AWS Region – For more details, see [Describe your Regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#using-regions-availability-zones-describe). ## Amazon EC2 instance parameters | Name | Parameter | Host 1 | Host 2 | | --- | --- | --- | --- | | Amazon EC2 instance ID | `` | `i-xxxxinstidforhost1` | `i-xxxxinstidforhost2` | | Hostname | `` | `slxhost01` | `slxhost02` | | Host IP | `` | `10.1.10.1` | `10.1.20.1` | | Host additional IP | `` | `10.1.10.2` | `10.1.20.2` | | Configured subnet | `` | `subnet-xxxxxxxxxxsubnet1` | `subnet-xxxxxxxxxxsubnet2` | | Associated VPC Route Table(s) | `` | `rtb-xxxxxroutetable1 [,rtb-xxxxxroutetable2]` | | | Sapmnt NFS ID or CNAME | `` | `fs-xxxxxxxxxxxxxefs1` | | + **Hostname** – Hostnames must comply with SAP requirements outlined in [SAP Note 611361 - Hostnames of SAP ABAP Platform servers](https://me.sap.com/notes/611361) (requires SAP portal access). Run the following command on your instances to retrieve the hostname. ``` # hostname ``` + **Amazon EC2 instance ID** – run the following command (IMDSv2 compatible) on your instances to retrieve instance metadata. ``` # /usr/bin/curl --noproxy '*' -w "\n" -s -H "X-aws-ec2-metadata-token: $(curl --noproxy '*' -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")" http://169.254.169.254/latest/meta-data/instance-id ``` For more details, see [Retrieve instance metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html) and [Instance identity documents](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html). + **Amazon EC2 subnet ID** – run the following command to retrieve the subnet ID for each of your instances. ``` # INSTANCE_ID=i-xxxxinstidforhost1 # aws ec2 describe-instances --instance-ids $INSTANCE_ID --query 'Reservations[0].Instances[0].SubnetId' --output text ``` For more details, see [describe-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html) and [VPC subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html). + **Route table(s) for subnets** – run the following AWS CLI commands to retrieve the route table(s) associated with both cluster node subnets. ``` # SUBNET_ID_1=subnet-xxxxxxxxxxsubnet1 # SUBNET_ID_2=subnet-xxxxxxxxxxsubnet2 # aws ec2 describe-route-tables --filters "Name=association.subnet-id,Values=$SUBNET_ID_1,$SUBNET_ID_2" --query 'RouteTables[].RouteTableId' --output text ``` If both cluster nodes are in subnets associated with the same route table, only one route table ID will be returned. If they are associated with different route tables, both route table IDs will be returned. For more details, see [describe-route-tables](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-route-tables.html) and [Route tables](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html). ## SAP Instance Parameters | Name | Parameter | Example | | --- | --- | --- | | SID | `` or `` | `SLX` | | ASCS Alias | `` | `slxascs` | | ASCS System Number | `` | `00` | | ASCS Overlay IP | `` | `172.16.1.50` | | ASCS NFS Mount Point | `` | `/SLX_ASCS00` | | ERS Alias | `` | `slxers` | | ERS System Number | `` | `10` | | ERS Overlay IP | `` | `172.16.1.51` | | ERS NFS Mount Point | `` | `/SLX_ERS10` | | ENSA Type | `` | `ENSA2` | ## Pacemaker Parameters | Name | Parameter | Example | | --- | --- | --- | | Cluster user | `cluster_user` | `hacluster` | | Cluster password | `cluster_password` | | | Cluster tag | `cluster_tag` | `pacemaker` | | AWS CLI cluster profile | `aws_cli_cluster_profile` | `cluster` | | Cluster connector | `cluster_connector` | `sap-suse-cluster-connector` | # Architecture diagrams This guide covers two architectures for SAP cluster solutions on SLES for SAP – simple-mount and classic (previous standard). See the following images to learn more. **Topics** + [Pacemaker - simple-mount architecture](#simple-mount-diagram-nw-sles) + [Pacemaker - classic architecture](#classic-diagram-nw-sles) ## Pacemaker - simple-mount architecture See the following image for more details. ![\[Simple Mount Achitecture\]](http://docs.aws.amazon.com/sap/latest/sap-netweaver/images/image-pacemaker-nw-sles-simplemount.png) ## Pacemaker - classic architecture See the following image for more details. ![\[Classic Architecture.\]](http://docs.aws.amazon.com/sap/latest/sap-netweaver/images/image-pacemaker-nw-sles-classic.png)