# SAP on AWS High Availability with Overlay IP Address Routing
This guide provides SAP customers and partners instructions to set up a highly available SAP architecture that uses overlay IP addresses on Amazon Web Services. This guide includes two configuration approaches:
+ AWS Transit Gateway serves as central hub to facilitate network connection to an overlay IP address.
+ Elastic Load Balancing where a Network Load Balancer enables network access to an overlay IP address.
This guide is intended for users who have previous experience installing and operating highly available SAP environments and systems.
**Topics**
+ [SAP on AWS High Availability Setup](sap-oip-sap-on-aws-high-availability-setup.md)
+ [Overlay IP Routing using AWS Transit Gateway](sap-oip-overlay-ip-routing-using-aws-transit-gateway.md)
+ [Overlay IP Routing with Network Load Balancer](sap-oip-overlay-ip-routing-with-network-load-balancer.md)
# SAP on AWS High Availability Setup
SAP customers can fully realize the benefit of running mission-critical SAP workloads by building reliable, fault-tolerant, and highly available systems in the AWS Cloud depending on the operating system and database. AWS offers the use of multiple Availability Zones within an AWS Region to provide resiliency for the SAP applications.
As part of your SAP implementation, you create an Amazon Virtual Private Cloud (Amazon VPC) to logically isolate the network from other virtual networks in the AWS Cloud. Then, you use AWS network routing features to direct the traffic to any instance in the VPCs or between different subnets in a VPC. Amazon VPC setup includes assigning [subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html) to your SAP ASCS/ERS for NetWeaver and primary/secondary nodes for the SAP HANA database. Each of these configured subnets has a classless inter-domain routing (CIDR) IP assignment from the VPC which resides entirely within one Availability Zone. This CIDR IP assignment cannot span multiple zones or be reassigned to the secondary instance in a different AZ during a failover scenario.
For this reason, AWS allows you to configure Overlay IP (OIP) outside of your VPC CIDR block to access the active SAP instance. With IP overlay routing, you can allow the AWS network to use a non-overlapping [RFC1918](https://tools.ietf.org/html/rfc1918) private IP address that resides outside an VPC CIDR range and direct the SAP traffic to any instance setup across the Availability Zone within the VPC by changing the routing entry in AWS.
A SAP HANA database or SAP NetWeaver application that is protected by a cluster solution such as [SUSE Linux Enterprise Server High Availability Extension](https://documentation.suse.com/sbp/all/html/SAP_HA740_SetupGuide_AWS/index.html) (SLES HAE), [RedHat Enterprise Linux HA Add-On](https://access.redhat.com/articles/4175371)(RHEL HA) or [SIOS](https://us.sios.com/solutions/cloud-high-availability/aws/sap/) uses the overlay IP address assigned to ensure that the HA cluster is still accessible during the failover scenarios. Since the overlay IP address uses the IP address range outside the VPC CIDR range and [Virtual Private Gateway](https://docs.aws.amazon.com/vpc/latest/adminguide/Introduction.html) connection, you can use [AWS Transit Gateway](https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html) as a central hub to facilitate the network connection to an overlay IP address from multiple locations including Amazon VPCs, other AWS Regions, and on-premises using AWS Direct Connect or AWS Client VPN.
If you do not have AWS Transit Gateway set up as a network transit hub or if AWS Transit Gateway is not available in your [preferred AWS Region](https://aws.amazon.com/transit-gateway/faqs/), you can use a [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) to enable network access to an OIP.
# Overlay IP Routing using AWS Transit Gateway
With Transit Gateway, you use route table rules which allow the overlay IP address to communicate to the SAP instance without having to configure any additional components, like a Network Load Balancer or Amazon Route 53. You can connect to the overlay IP from another VPC, another subnet (not sharing the same route table where overlay IP address is maintained), over a VPN connection, or via an AWS Direct Connect connection from a corporate network.
**Note:** If you do not use Amazon Route 53 or AWS Transit Gateway, see the [Overlay IP Routing with Network Load Balancer](sap-oip-overlay-ip-routing-with-network-load-balancer.md) section.
## Architecture
AWS Transit Gateway acts as a hub that controls how traffic is routed among all the connected networks which act like spokes. Your Transit Gateway routes packets between source and destination attachments using [Transit Gateway route tables](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-route-tables.html). You can configure these route tables to propagate routes from the route tables for the attached VPCs and VPN connections. You can also add static routes to the Transit Gateway route tables. You can add the overlay IP address or address CIDR range as a static route in the transit gateway route table with a target as the VPC where the EC2 instances of SAP cluster are running. This way, all the network traffic directed towards overlay IP addresses is routed to this VPC. The following figure shows this scenario with connectivity from different VPC and corporate network.
**Figure 1: Overlay IP address setup with AWS Transit Gateway**
![\[Overlay IP address setup with Transit Gateway\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image1.png)
*Pricing for the AWS Transit Gateway*:
AWS Transit Gateway [pricing](https://aws.amazon.com/transit-gateway/pricing/) is based on the number of connections made to the Transit Gateway per hour and the amount of traffic that flows through AWS Transit Gateway. For more information, see [AWS Transit Gateway Service Level Agreement](https://aws.amazon.com/transit-gateway/sla/).
# Configuration Steps for AWS Transit Gateway
This section includes high-level steps necessary to understand overlay IP address configuration for this scenario. See the [AWS Transit Gateway documentation](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-getting-started.html) for detailed steps regarding AWS Transit Gateway configuration.
## Step 1. Set up the Transit Gateway architecture
1. Create a Transit Gateway in your AWS account in the AWS Region where the SAP instance is deployed. For detailed steps, see [Getting Started with Transit Gateways](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-getting-started.html).
1. Attach VPCs where SAP instances are deployed (and any other VPCs as required) to the Transit Gateway. For detailed steps, see [Transit Gateway Attachments to a VPC](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-vpc-attachments.html).
**Note:** For attachment, select only the subnet where the SAP instances are running with cluster and overlay IP configured. In the following figure, the private subnet of the SAP instance is selected for the Transit Gateway attachment.
**Figure 2: Attaching Transit Gateway to private subnet**
![\[Attaching Transit Gateway to private subnet\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image2.png)
1. Do one of the following, depending on your connection:
+ **VPN connection**. Attach a VPN to this Transit Gateway. For detailed steps, see [Transit Gateway VPN Attachments](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-vpn-attachments.html).
When you create a site-to-site VPN connection, you specify the static routes for the overlay IP address. For detailed steps, see [VPN routing options](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNRoutingTypes.html).
+ ** AWS Direct Connect**. Attach a Direct Connect Gateway to this Transit Gateway. First, associate a Direct Connect Gateway with the Transit Gateway. Then, create a transit virtual interface for your AWS Direct Connect connection to the Direct Connect gateway. Here, you can advertise prefixes from on-premises to AWS and from AWS to on-premises. For detailed steps, see [Transit Gateway Attachments to a Direct Connect Gateway](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-dcg-attachments.html).
When you associate a Transit Gateway with a Direct Connect gateway, you specify the prefix lists to advertise the overlay IP address to the on-premises environment. For detailed steps, see [Allowed prefixes interactions](https://docs.aws.amazon.com/directconnect/latest/UserGuide/allowed-to-prefixes.html).
**Note:** AWS Direct Connect is recommended for business critical workloads. See [Resilience in AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/disaster-recovery-resiliency.html) to learn about resiliency at the network level.
## Step 2. Configure routing for AWS and corporate networks
The following table lists the IP addresses used in the example configuration. Make sure to use your valid private IP addresses for your implementation.
| Description | IP Range/IP Address |
| --- | --- |
| VPC CIDR of production SAP systems (with HA cluster running with Overlay IP) | 10.0.0.0/16 |
| VPC CIDR of non-production SAP systems (Instances in this VPC access the Production cluster overlay IP using AWS Transit Gateway) | 192.168.1.0/24 |
| Corporate network CIDR (Site-to-Site VPN is configured between corporate networks to AWS Transit Gateway) | 192.168.2.0/24 |
| Overlay IP address CIDR | 172.16.1.0/26 |
| Customer gateway IP address | 34.216.94.150/32 |
**Note**
If you are using [AWS Client VPN](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/what-is.html), you do not need to configure Transit Gateway. You can create additional entries in the routing table for overlay IP addresses. Route traffic to the subnets of the VPC of production SAP system where overlay IP addresses are configured.
When you create a Transit Gateway attachment to a VPC, the propagation route is created in the default Transit Gateway route table. In Figure 3, the first and second entry shows the propagated route created automatically for VPCs where SAP production and non-production systems are running through VPC attachment.
1. To route traffic from AWS Transit Gateway to the overlay IP address, create static routes in the **Transit Gateway route tables** to route overlay IP addresses to the VPC of production SAP system where the overlay IP addresses are configured. In Figure 3, the third entry shows that the static route created for the overlay IP range is attached. The target for this route is the SAP Production VPC.
**Figure 3: Transit Gateway route table: Overlay IP static route with VPC of production SAP system target**
![\[Transit Gateway route table: Overlay IP static route with VPC of production SAP system target\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image3.png)
1. To route the outgoing traffic from VPCs where SAP instances are running to private IP addresses of another VPC where SAP instances are running attached to same Transit Gateway, create entries in the **route tables associated with these VPC subnets**. The target of these routes is AWS Transit Gateway. In the following VPC of production SAP system route table example, the non-production SAP VPC (third entry) and corporate network (fourth entry) are routed to the Transit Gateway.
**Figure 4: VPC of production SAP system route table: VPC of production SAP system and corporate network routed to AWS Transit Gateway**
![\[VPC of production SAP system route table: VPC of production SAP system and corporate network routed to Transit Gateway\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image4.png)
1. In the VPC of the non-production SAP system, to route the outgoing traffic from the overlay IP address, create entries in the route tables with Transit Gateway as the target. In the following VPC of non-production SAP system route table example, the destination is the overlay IP range and the target is Transit Gateway.
**Figure 5: VPC of non-production SAP system route table: Outgoing traffic from overlay IP address routed to Transit Gateway**
![\[VPC of non-production SAP system route table: Outgoing traffic from overlay IP address routed to Transit Gateway\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image5.png)
1. Configure routing from corporate devices to Amazon VPC IP addresses.
## Step 3. Disable the source/destination check
Each Amazon EC2 performs source/destination checks by default. This means that the instance must be the source or destination of any traffic it sends or receives. For cluster instances, source/destination check must be disabled on both Amazon EC2 instances which are supposed to receive traffic from the Overlay IP address. You can use the [AWS CLI](https://aws.amazon.com/cli/) or the [AWS Management Console](https://aws.amazon.com/console/) to disable source/destination check. For details, see [ec2 modify-instance-attribute](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/modify-instance-attribute.html).
## Step 4. Test the configuration
Once the setup is complete, perform connectivity testing by making sure you can reach the SAP systems through overlay IP address. With this configuration, you can reach the overlay IP addresses from other VPCs and your corporate network just like any private IP address of the VPC. With the AWS Transit Gateway approach, no additional components are required for communication, such as Amazon Route 53 agent or Network Load Balancer.
## Step 5. Update overlay IP address
Step 4: Once the network connectivity is tested successfully, update the overlay IP address of the production or non-production SAP system in the message server parameter of your SAP Graphical User Interface (GUI) System Entry Properties along with other SAP connectivity properties for connection. You can use the corporate DNS or Amazon Route 53 to create a user friendly CNAME for the Overlay IP.
# Overlay IP Routing with Network Load Balancer
If you do not use Amazon Route 53 or AWS Transit Gateway, you can use [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) for accessing the overlay IP address externally. The Network Load Balancer functions at the fourth layer of the Open Systems Interconnection (OSI) model. It can handle millions of requests per second. After the load balancer receives a connection request, it selects a target from the Network Load Balancer target group to route network connection request to a destination address which can be an overlay IP address.
## Architecture
The following figure shows the network access flow of ASCS or SAP HANA overlay IP from outside the VPC.
**Figure 6: SAP High Availability with Overlay IP and Elastic Load Balancer**
![\[SAP High Availability with Overlay IP and Elastic Load Balancer\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image6.png)
*Pricing for Network Load Balancers*:
With Network Load Balancers, you only pay for what you use. See [Elastic Load Balancing pricing](https://aws.amazon.com/elasticloadbalancing/pricing/), for more information.
# Configuration Steps for Network Load Balancer
Use the following instructions to set up the Network Load Balancer to access the overlay IP address. The following values are used for the example configuration.
**Table 1: System Settings**
| System Setting | Value |
| --- | --- |
| Instance number for ASCS and SAP HANA | 00 |
| OIP for ASCS | 192.168.0.20 |
| OIP for HANA | 192.168.1.99 |
**Table 2: Listener Port Values**
| Listener Ports | Value |
| --- | --- |
| ASCS Message server port | 36 (3600) |
| SAP HANA | SAP HANA Studio service connection (login required) [SAP Note 1592925](https://me.sap.com/notes/1592925) |
| SAPStartSrv/HTTP Port | 513 (50013) |
| JDBC/SQL Port | 315 (30015) |
## Step 1. Create the target group
1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/
1. On the navigation pane, under **LOAD BALANCING**, choose **Target Groups**.
1. Choose Create target group.
1. For **Name**, type an easily identified target group name for the sap-ascs instance. (For example, type sap-ascs for your ASCS overlay IP address).
1. For **Target type**, select **IP**.
1. For **Protocol**, choose **TCP**.
1. For **Port**, type 36. For example: 3600, where 00 is the instance number.
1. For **Health checks**, keep the default health check settings, or change settings based on your requirements.
1. Choose **Create**.
1. Repeat steps 1 to 9 to create target group for JDBC/SQL port 315 and SAP HANA HTTP port 513 to access your SAP HANA instance with the respective overlay IP address.
1. Choose the **Targets** tab, then choose **Edit**.
1. Choose **Add** to register your targets.
1. Choose the **Network** drop-down and select **Other private IP address**. Then, enter the ASCS overlay IP address and choose **Add to list**.
1. Repeat steps 11 to 13 to register JDBC/SQL and HTTP ports with the respective overlay IP address.
## Step 2. Create the Network Load Balancer for ASCS
1. On the EC2 navigation pane, under **LOAD BALANCING**, choose **Load Balancers**.
1. Choose Create Load Balancer.
1. For Network Load Balancer, choose Create.
1. For **Name**, type a name for your load balancer. For example, sap-ha-nlb.
1. For **Scheme**, choose **internal**. An internal load balancer routes requests to targets using private IP addresses.
1. For **Listeners**, under Protocol, choose **TCP**. For **Port**, specify the ASCS port (36< SAP Instance number>. For example, use 3600 if your SAP instance number is 00.
1. For **Availability Zones**, select the VPC and subnets where the SAP instances with HA setup are deployed.
1. For **Tags**, choose **Add Tags** and for Key, type Name. For Value, type the name of the network load balancer, such as sap-ha-nlb.
1. Choose Next: Configure Security Settings.
1. Ignore the warning that appears and choose **Next: Configure Routing**. (In this scenario, the network load balancer is used as pass through without any SSL termination. For end-to-end encryption, use SNC from SAP GUI to SAP Instance.)
1. For **Target group**, choose **Existing target group** and select the **sap-ascs** target group created earlier.
1. Choose **Next: Register Targets**.
1. Choose **Next: Review**.
1. Choose **Create**.
1. Repeat the steps 1 to 14 to create another Network Load Balancer for SAP HANA setup with Network Load Balancer TCP protocol listener to JDBC/SQL port 315. Choose VPC and the subnets where the primary and secondary SAP HANA database is deployed and register the target JDBC/SQL target group.
1. Add an additional listener to the Network Load Balancer created in step 14 with SAP StartSrv/HTTP port 513 listener port and register the target StartSrv/HTTP port target group.
## Step 3. Set up VPC routing table
This step enables the connection to your SAP instance.
1. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/
1. In the navigation pane, choose **Route Tables**, and select the Amazon VPC routing table where your SAP instance is deployed.
1. Choose **Actions**, **Edit routes**.
1. For **Destination**, specify your overlay IP address. For **Target**, specify the SAP instance Elastic Network Interface.
1. Choose Save routes.
This setup allows the static Network Load Balancer DNS to forward the traffic to your SAP instance network interface through the static overlay IP address. During failover scenarios, you can point to the elastic network interface of the active SAP instance using manual steps or automatically using cluster management software.
## Step 4. Connect using SAP GUI
1. In the **Load Balancers** section of the EC2 console, make a note of the Network Load Balancer DNS name for the sap-ha-nlb.
**Figure 7: sap-ha-nlb DNS name**
![\[sap-ha-nlb DNS name\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image7.png)
1. Start SAP Logon.
1. Choose **New**, then **Next**.
1. In the System Entry Properties box, for Connection Type, choose Group/Server Selection.
1. For **Message Server**, type the Network Load Balancer DNS name, and choose **OK**.
**Figure 8: Configuring System Connection Parameters for SAP GUI**
![\[Configuring System Connection Parameters for SAP GUI\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image8.png)
## Step 5. Connect using SAP HANA Studio
1. In the **Load Balancers** section of the EC2 console, make a note of the Network Load Balancer DNS name for the JBDC/SQL and SAPStartSrv/HTTP ports.
**Figure 9: DNS name of ports**
![\[DNS name of ports\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image9.png)
1. In the Host Name parameter of SAP HANA Studio, use the Network Load Balancer DNS name and provide additional credentials to connect to the SAP HANA system.
**Figure 10: Updated Host Name in SAP HANA Studio**
![\[Updated Host Name in SAP HANA Studio\]](http://docs.aws.amazon.com/sap/latest/sap-hana/images/ha-overlay-ip-image10.png)
# Additional Implementation Notes
+ If other applications outside the VPC need to connect to the SAP system via the ASCS, create additional listeners with the ports on which these applications communicate.
+ For customers using SAP Gateway Service (GW) and have designed HA for this service, create a target group for the GW service as well (33). Point the health check port for the GW target group to the message server port (36).
+ You can use the corporate DNS or Amazon Route 53 Public Data Plane to create a user friendly CNAME for the Network Load Balancer DNS name. If you use an alias for connecting to the SAP GUI on-premises, the alias can be created as the CNAME for the Network Load Balancer DNS name. With this approach, there are no changes required on your SAP GUI configuration post migration to AWS. If other systems, such as SAP Landscape Management that requires a reverse lookup to function, are connecting to the highly available system, use A and PTR records instead of CNAME.