# Planning
<a name="sap-hana-pacemaker-rhel-planning"></a>

Review the following prerequisites carefully before beginning your high availability cluster deployment, ensuring all infrastructure, operating system, and access requirements are met. Familiarize yourself with linked references, supported configurations and the core concepts which are used in this solution.

**Topics**
+ [Setup Overview](sap-hana-pacemaker-rhel-setup-overview.md)
+ [Vendor Support](sap-hana-pacemaker-rhel-references.md)
+ [Concepts](sap-hana-pacemaker-rhel-concepts.md)
+ [Automated Deployment](sap-hana-pacemaker-rhel-automation.md)
+ [Parameter Reference](sap-hana-pacemaker-rhel-parameters.md)
+ [Architecture Diagrams](sap-hana-pacemaker-rhel-arch-diagrams.md)

# Setup Overview
<a name="sap-hana-pacemaker-rhel-setup-overview"></a>

## Deployed Cluster Infrastructure
<a name="_deployed_cluster_infrastructure"></a>

Ensure that your AWS networking requirements and Amazon EC2 instances where SAP workloads are installed, are correctly configured for SAP.

The following SAP HANA cluster specific requirements must be met:
+ Two cluster nodes created in private subnets in separate Availability Zones within the same Amazon VPC and AWS Region.
+ Access to the route table(s) that are associated with the chosen subnets. For more information, see [Overlay IP](sap-hana-pacemaker-rhel-concepts.md#overlay-ip-rhel).
+ Targeted Amazon EC2 instances must have connectivity to the Amazon EC2 endpoint via internet or an Amazon VPC endpoint.

## Supported Operating System
<a name="_supported_operating_system"></a>

Protecting the SAP HANA Database with a pacemaker cluster requires packages from Red Hat, including cluster resource agents for SAP and AWS that are not available in standard repositories.

For deploying SAP HANA on Red Hat, either "RHEL for SAP Solutions" (BYOS) or "RHEL for SAP with High Availability and Update Services" (PAYG) are required.

## Required Access for Setup
<a name="_required_access_for_setup"></a>

The following access is required for setting up the cluster:

An IAM user with the following privileges:
+ Modify Amazon VPC route tables
+ Modify Amazon EC2 instance properties
+ Create IAM policies and roles
+ Create Amazon EFS file systems

Additional required access:
+ Root access to the operating system of both cluster nodes
+ SAP HANA administrative user access – <sid>adm
+ SAP HANA SystemDB Administrative access for changing configuration and backup administration.

**Example**  
These access requirements are specific to the cluster setup process and can be restricted for ongoing cluster operations and maintenance.

## Reliability Requirements Defined
<a name="_reliability_requirements_defined"></a>

The SAP Lens of the Well-Architected framework, in particular the Reliability pillar, can be used to understand the reliability requirements for your SAP workload.

The SAP HANA application is a single point of failure in a highly available SAP architecture. The impact of an outage of this component must be evaluated against factors, such as, recovery point objective (RPO), recovery time objective (RTO), cost and operation complexity. For more information, see [Reliability in SAP Lens - AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/sap-lens/reliability.html).

# Vendor Support
<a name="sap-hana-pacemaker-rhel-references"></a>

## SAP and Red Hat References
<a name="_sap_and_red_hat_references"></a>

In addition to this guide, see the following references for more details:
+ Red Hat Documentation: [Automating SAP HANA Scale-Up System Replication using the RHEL HA Add-On - Red Hat Enterprise Linux for SAP Solutions 9](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_for_sap_solutions/9/html/automating_sap_hana_scale-up_system_replication_using_the_rhel_ha_add-on/index) 
+ Red Hat Documentation: [Deploying SAP HANA Scale-Up System Replication High Availability - Advanced Next Generation Interface](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_for_sap_solutions/9/html/deploying_sap_hana_scale-up_system_replication_high_availability/index) 
+ Red Hat Documentation: [Automating SAP HANA Scale-Out System Replication using the RHEL HA Add-On - Red Hat Enterprise Linux for SAP Solutions 9](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_for_sap_solutions/9/html/automating_sap_hana_scale-out_system_replication_using_the_rhel_ha_add-on) 
+ SAP Note: [1656099 - SAP Applications on AWS: Supported DB/OS and Amazon EC2 products](https://me.sap.com/notes/1656099) 
+ SAP Note: [2777782 - SAP HANA DB: Recommended OS Settings for RHEL 8](https://me.sap.com/notes/2777782) 
+ SAP Note: [3108302 - SAP HANA DB: Recommended OS Settings for RHEL 9](https://me.sap.com/notes/3108302) 

**Note**  
SAP portal access is required to access SAP Notes.

## Deployment Guidance
<a name="deployments-rhel"></a>

 AWS works in collaboration with Red Hat to support SAP HANA deployments on AWS. AWS provides detailed guidance on configuring EC2 instances and AWS-specific resources to meet SAP HANA requirements. While we strive to consolidate documentation to simplify the user experience, the underlying software components and resources owned by Pacemaker remain under the purview of the software vendor for development and support.


| SAP HANA Deployment Type | Support Status | Notes |  AWS Configuration Patterns | 
| --- | --- | --- | --- | 
|  SAP HANA Scale-Up Standard  |   AWS Documented & Supported  |  Covered in AWS SAP HANA guides  |  SAPHANAScaleUp-Classic, SAPHANAScaleUp-ANGI  | 
|  SAP HANA Scale-Up Secondary Read-Enabled  |  Vendor Documented & Supported  |  Follows SAP documentation  |  | 
|  SAP HANA Scale-Up Multi-Tier Replication  |  Vendor Documented & Supported  |  Follows SAP documentation  |  | 
|  SAP HANA Scale-Up Multi-Target Replication  |  Vendor Documented & Supported  |  Follows SAP documentation  |  | 
|  SAP HANA Scale-Out Standard  |   AWS Documented & Supported  |  Covered in AWS SAP HANA guides  |  SAPHANAScaleOut-Classic, SAPHANAScaleUp-ANGI  | 
|  SAP HANA Scale-Out Secondary Read-Enabled  |  Vendor Documented & Supported  |  Follows SAP documentation  |  | 
|  SAP HANA Scale-Out Multi-Tier Replication  |  Vendor Documented & Supported  |  Follows SAP documentation  |  | 
|  SAP HANA Scale-Out Multi-Target Replication  |  Vendor Documented & Supported  |  Follows SAP documentation  |  | 

**Note**  
 AWS configuration patterns represent standardized deployment templates that have been validated for specific use cases. In the documentation we will highlight where instructions deviate according to the configuration pattern.

**What is Angi?**  
SAPHanaSR-angi (SAP HANA SR - Advanced Next Generation Interface) is the latest unified high availability solution for managing SAP HANA System Replication in Pacemaker clusters, supported on RHEL 9.6 and newer. The solution consolidates the management of both scale-up and scale-out deployments into a single package and introduces technical improvements such as faster takeover times during filesystem failures, unresponsive HANA instances, and node failures in scale-out configurations.

This document covers new implementations using SAPHanaSR-angi. For migrations from existing SAPHanaSR or SAPHanaSR-ScaleOut installations to SAPHanaSR-angi, refer to the Red Hat documentation.

# Concepts
<a name="sap-hana-pacemaker-rhel-concepts"></a>

## SAP – SAP HANA and Hana System Replication
<a name="_sap_sap_hana_and_hana_system_replication"></a>

SAP HANA is an in-memory, column-oriented, relational database management system developed by SAP. It uses HANA System Replication (HSR) to replicate data and changes from a primary system to one or more secondary systems. In scale-out deployments, this replication occurs between corresponding nodes across the primary and secondary systems, with each service having its counterpart in the secondary system. HSR ensures changes are continuously replicated to minimize the Recovery Point Objective (RPO). While takeovers can be manually triggered using HANA tooling, the addition of a Pacemaker cluster automates the failover process through monitoring, orchestration, and integration with resource agents for hardware connectivity and management.

## AWS – Availability Zones
<a name="shared_aws_availability_zones"></a>

An Availability Zone is one or more discreet data centers with redundant power, networking, and connectivity in an AWS Region. For more information, see Regions and Availability Zones.

For mission critical deployments of SAP on AWS where the goal is to minimise the recovery time objective (RTO), we suggest distributing single points of failure across Availability Zones. Compared with single instance or single Availability Zone deployments, this increases resilience and isolation against a broad range of failure scenarios and issues, including natural disasters.

Each Availability Zone is physically separated by a meaningful distance (many kilometers) from another Availability Zone. All Availability Zones in an AWS Region re interconnected with high-bandwidth, low-latency network, over fully redundant, dedicated metro fiber. This enables synchronous replication. All traffic between Availability Zones is encrypted.

## AWS – Overlay IP
<a name="overlay-ip-rhel"></a>

An Overlay IP enables a connection to the application, regardless of which Availability Zone (and subnet) contains the active primary node.

When deploying an Amazon EC2 instance in AWS, IP addresses are allocated from the CIDR range of the allocated subnet. The subnet cannot span across multiple Availability Zones, and therefore the subnet IP addresses may be unavailable after faults, including network connectivity or hardware issues which require a failover to the replication target in a different Availability Zone.

To address this, we suggest that you configure an overlay IP, and use this in the connection parameters for the application. This IP address is a non-overlapping RFC1918 private IP address from outside of VPC CIDR block and is configured as an entry in the route table or tables. The route directs the connection to the active node and is updated during a failover by the cluster software.

You can select any one of the following RFC1918 private IP addresses for your overlay IP address:
+ 10.0.0.0 – 10.255.255.255 (10/8 prefix)
+ 172.16.0.0 – 172.31.255.255 (172.16/12 prefix)
+ 192.168.0.0 – 192.168.255.255 (192.168/16 prefix)

If, for example, you use the 10/8 prefix in your SAP VPC, selecting a 172 or a 192 IP address may help to differentiate the overlay IP. Consider the use of an IP Address Management (IPAM) tool such as Amazon VPC IP Address Manager to plan, track, and monitor IP addresses for your AWS workloads. For more information, see [What is IPAM?](https://docs.aws.amazon.com/vpc/latest/ipam/what-it-is-ipam.html) 

The overlay IP agent in the cluster can also be configured to update multiple route tables which contain the Overlay IP entry if your subnet association or connectivity requires it.

### Access to the Overlay IP
<a name="_access_to_the_overlay_ip"></a>

The overlay IP is outside of the range of the VPC, and therefore cannot be reached from locations that are not associated with the route table, including on-premises and other VPCs.

Use AWS Transit Gateway as a central hub to facilitate the network connection to an overlay IP address from multiple locations, including Amazon VPCs, other AWS Regions, and on-premises using AWS Direct Connect or AWS Client VPN.

If you do not have AWS Transit Gateway set up as a network transit hub or if it is not available in your preferred AWS Region, you can use a Network Load Balancer to enable network access to an overlay IP.

For more information, see [SAP on AWS High Availability Setup](sap-oip-sap-on-aws-high-availability-setup.md).

## AWS – Shared VPC
<a name="shared_aws_shared_vpc"></a>

An enterprise landing zone setup or security requirements may require the use of a separate cluster account to restrict the route table access required for the Overlay IP to an isolated account. For more information, see [Share your VPC with other accounts](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-sharing.html).

Evaluate the operational impact against your security posture before setting up shared VPC.

## Pacemaker - STONITH Fencing Agent
<a name="fencing-rhel"></a>

In SAP HANA deployments, whether in a scale-up configuration (two-node) or a scale-out configuration (two or more nodes per site), it is crucial that data consistency is maintained by ensuring only the designated primary node or nodes can process write operations at any given time. When a node becomes unresponsive or incommunicable, maintaining data consistency may require that the faulty node is isolated by powering it down before the cluster commences other actions, such as promoting a new primary. This arbitration is the role of the fencing agent.

In a two-node scale-up scenario, fence racing is a critical concern. This occurs when a communication failure causes both nodes to simultaneously attempt to fence (power off) each other, believing the other node has failed. The fencing agent addresses this risk by providing an external witness. In scale-out deployments, while fence racing is less likely due to the presence of multiple nodes that can participate in quorum decisions, proper fencing remains critical for maintaining data consistency across the larger node set.

Red Hat supports several fencing agents, including the one recommended for use with Amazon EC2 Instances (fence\$1aws).

# Automated Deployment
<a name="sap-hana-pacemaker-rhel-automation"></a>

You can set up a cluster manually using the instructions provided here. You can also automate parts of this process to ensure consistent and repeatable deployments.

Use AWS Launch Wizard for SAP to automated deployments of SAP Hana Platform, SAP NetWeaver, SAP S/4 HANA, SAP BW/4HANA, and Solution Manager. Launch Wizard uses AWS CloudFormation templates and advanced scripts to quickly provision the required resources. The automation handles SAP HANA Installation, HANA System Replication and Pacemaker setup, requiring only post-deployment validation and testing. For more information, see [AWS Launch Wizard for SAP](https://docs.aws.amazon.com/launchwizard/latest/userguide/launch-wizard-sap.html).

**Important**  
For reliable cluster operations, thoroughly test your system regardless of setup method. Testing helps identify system anomalies, validate changing requirements, and build operational understanding. See [Testing](sap-hana-pacemaker-rhel-testing.md) for more details.

# Parameter Reference
<a name="sap-hana-pacemaker-rhel-parameters"></a>

The cluster setup uses parameters, including SID and System Number that are unique to your setup. It is useful to predetermine the values with the following examples and guidance.

**Topics**
+ [Global AWS Parameters](#global_shared_aws_parameters)
+ [Amazon EC2 Instance Parameters](#_amazon_ec2_instance_parameters)
+ [SAP and Pacemaker Resource Parameters](#_sap_and_pacemaker_resource_parameters)
+ [Red Hat Cluster Parameters](#_red_hat_cluster_parameters)

## Global AWS Parameters
<a name="global_shared_aws_parameters"></a>


| Name | Parameter | Example | 
| --- | --- | --- | 
|   AWS account ID  |   `<account_id>`   |   `123456789100`   | 
|   AWS Region  |   `<region>`   |   `us-east-1`   | 
+  AWS account – For more details, see [Your AWS account ID and its alias](https://docs.aws.amazon.com/IAM/latest/UserGuide/console-account-alias.html).
+  AWS Region – For more details, see [Describe your Regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones).

## Amazon EC2 Instance Parameters
<a name="_amazon_ec2_instance_parameters"></a>


| Name | Parameter | Primary example | Secondary example | 
| --- | --- | --- | --- | 
|  Amazon EC2 instance ID  |   `<instance_id_x>`   |   `i-xxxxinstidforhost1`   |   `i-xxxxinstidforhost2`   | 
|  Hostname  |   `<hostname_x>`   |   `hanahost01`   |   `hanahost02`   | 
|  Host IP  |   `<host_ip_x>`   |   `10.1.20.1`   |   `10.2.20.1`   | 
|  Host additional IP  |   `<host_additional_ip_x>`   |   `10.1.20.2`   |   `10.2.20.2`   | 
|  Configured subnet  |   `<subnet_id>`   |   `subnet-xxxxxxxxxxsubnet1`   |   `subnet-xxxxxxxxxxsubnet2`   | 
+ Hostnames must comply with SAP requirements outlined in [SAP Note 611361 - Hostnames of SAP ABAP Platform servers](https://me.sap.com/notes/611361) (requires SAP portal access).
+ Run the following command on your instances to retrieve the hostname:

  ```
  $ hostname
  ```
+ Amazon EC2 instance ID – run the following command (IMDSv2 compatible) on your instances to retrieve instance metadata:

  ```
  $ /usr/bin/curl --noproxy '*' -w "\n" -s -H "X-aws-ec2-metadata-token: $(curl --noproxy '*' -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")" http://169.254.169.254/latest/meta-data/instance-id
  ```

  For more details, see [Retrieve instance metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html) and [Instance identity documents](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html).

 **For Scale-out Deployments** 


| Role | Primary Coordinator | Primary Worker | Primary Worker | Secondary Coordinator | Secondary Worker | Secondary Worker | Majority Maker | 
| --- | --- | --- | --- | --- | --- | --- | --- | 
|  Hostname  |   `hanahost01`   |   `hanahostworker01a`   |   `hanahostworker01b`   |   `hanahost02`   |   `hanahostworker02a`   |   `hanahostworker02b`   |   `hanamm`   | 
|  Subnet  |   `subnet-xxxxxxxxxxsubnet1`   |   `subnet-xxxxxxxxxxsubnet1`   |   `subnet-xxxxxxxxxxsubnet1`   |   `subnet-xxxxxxxxxxsubnet2`   |   `subnet-xxxxxxxxxxsubnet2`   |   `subnet-xxxxxxxxxxsubnet2`   |   `subnet-xxxxxxxxxxsubnet3`   | 
+ Example for a 6 node cluster with a majority maker
+ The majority maker can use minimal resources as it only provides cluster quorum functionality

## SAP and Pacemaker Resource Parameters
<a name="_sap_and_pacemaker_resource_parameters"></a>


| Name | Parameter | Example | 
| --- | --- | --- | 
|  SAP HANA SID  |   `<SID>` or `<sid>`   |   `HDB`   | 
|  SAP HANA System Number  |   `<hana_sys_nr>`   |   `00`   | 
|  SAP HANA Virtual Hostname  |   `<hana_virt_hostname>`   |   `hanahdb`   | 
|  SAP HANA Overlay IP  |   `<hana_overlayip>`   |   `172.16.52.1`   | 
|  SAP HANA Read Enabled Overlay IP (optional)  |   `<readenabled_overlayip>`   |   `172.16.52.2`   | 
|  VPC Route Tables  |   `<routetable_id>`   |   `rtb-xxxxxroutetable1`   | 
+ SAP details – SAP parameters, including SID and instance number must follow the guidance and limitations of SAP and Software Provisioning Manager. Refer to [SAP Note 1979280 - Reserved SAP System Identifiers (SAPSID) with Software Provisioning Manager](https://me.sap.com/notes/1979280) for more details.
+ Post-installation, use the following command to find the details of the instances running on a host:

  ```
  $ sudo /usr/sap/hostctrl/exe/saphostctrl -function ListInstances
  ```
+ Overlay IP – This value is defined by you. For more information, see [Overlay IP](sap-hana-pacemaker-rhel-concepts.md#overlay-ip-rhel).

## Red Hat Cluster Parameters
<a name="_red_hat_cluster_parameters"></a>


| Name | Parameter | Example | 
| --- | --- | --- | 
|  Cluster user  |   `<cluster_user>`   |   `hacluster`   | 
|  Cluster password  |   `<cluster_password>`   |  | 
|  Cluster name  |   `<cluster_name>`   |   `myCluster`   | 
|   AWS CLI cluster profile  |   `<cli_cluster_profile>`   |   `cluster`   | 
+ Cluster user – Installing cluster packages will create the user hacluster, set a password to this account to ensure that the cluster can perform the tasks which do not require root access.

# Architecture Diagrams
<a name="sap-hana-pacemaker-rhel-arch-diagrams"></a>

## Pacemaker - Scale-Up Architecture
<a name="_pacemaker_scale_up_architecture"></a>

![\[SAP Hana Pacemaker Red Hat Enterprise Linux Scale-Up\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/sap-hana-pacemaker-rhel-scaleup.png)