# Networking in Amazon EC2 Networking Amazon VPC enables you to launch AWS resources, such as Amazon EC2 instances, into a virtual network dedicated to your AWS account, known as a virtual private cloud (VPC). When you launch an instance, you can select a subnet from the VPC. The instance is configured with a primary network interface, which is a logical virtual network card. The instance receives a primary private IP address from the IPv4 address of the subnet, and it is assigned to the primary network interface. You can control whether the instance receives a public IP address from Amazon's pool of public IP addresses. The public IP address of an instance is associated with your instance only until it is stopped or terminated. If you require a persistent public IP address, you can allocate an Elastic IP address for your AWS account and associate it with an instance or a network interface. An Elastic IP address remains associated with your AWS account until you release it, and you can move it from one instance to another as needed. You can bring your own IP address range to your AWS account, where it appears as an address pool, and then allocate Elastic IP addresses from your address pool. To increase network performance and reduce latency, you can launch instances in a placement group. You can get significantly higher packet per second (PPS) performance using enhanced networking. You can accelerate high performance computing and machine learning applications using an Elastic Fabric Adapter (EFA), which is a network device that you can attach to a supported instance type. **Topics** + [ # Regions and Zones ](using-regions-availability-zones.md) + [Instance IP addressing](using-instance-addressing.md) + [ # EC2 instance hostnames and domains ](ec2-instance-naming.md) + [Bring your own IP addresses](ec2-byoip.md) + [Elastic IP addresses](elastic-ip-addresses-eip.md) + [Network interfaces](using-eni.md) + [Network bandwidth](ec2-instance-network-bandwidth.md) + [Enhanced networking](enhanced-networking.md) + [Elastic Fabric Adapter](efa.md) + [EC2 topology](ec2-instance-topology.md) + [Placement groups](placement-groups.md) + [Network MTU](network_mtu.md) + [Virtual private clouds](using-vpc.md) + [Secondary Networks](secondary-networks.md) # Regions and Zones Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of AWS Regions, Availability Zones, Local Zones, AWS Outposts, and Wavelength Zones. + **Regions** are separate geographic areas. + **Availability Zones** are multiple, isolated locations within each Region. + **Local Zones** provide you with the ability to place resources, such as compute and storage, in multiple locations closer to your end users. + **Wavelength Zones** provide you with the ability to build applications that deliver ultra-low latencies to 5G devices and end users. Wavelength deploys standard AWS compute and storage services to the edge of telecommunication carriers' 5G networks. + **AWS Outposts** brings native AWS services, infrastructure, and operating models to virtually any data center, colocation space, or on-premises facility. AWS operates state-of-the-art, highly available data centers. Although rare, failures can occur that affect the availability of instances that are in the same location. If you host all of your instances in a single location that is affected by a failure, none of your instances would be available. For more information, see [AWS Global Infrastructure](https://aws.amazon.com/about-aws/global-infrastructure/). **Topics** + [ ## Regions ](#concepts-regions) + [ ## Availability Zones ](#concepts-availability-zones) + [ ## Local Zones ](#concepts-local-zones) + [ ## Wavelength Zones ](#concepts-wavelength-zones) + [ ## AWS Outposts ](#concepts-outposts) ## Regions Each Region is designed to be isolated from the other Regions. This achieves the greatest possible fault tolerance and stability. When you launch an instance, select a Region that puts your instances close to specific customers, or that meets the legal or other requirements that you have. You can launch instances in multiple Regions. When you view your resources, you see only the resources that are tied to the Region that you specified. This is because Regions are isolated from each other, and we don't automatically replicate resources across Regions. ### Available Regions For the list of available Regions, see [AWS Regions](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html). ### Regional endpoints for Amazon EC2 When you work with an instance using the command line interface or API actions, you must specify its Regional endpoint. For more information about the Regions and endpoints for Amazon EC2, see [Amazon EC2 service endpoints](https://docs.aws.amazon.com/ec2/latest/devguide/ec2-endpoints.html) in the *Amazon EC2 Developer Guide*. For more information, see [AWS Regions](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html) in the *AWS Regions and Availability Zones User Guide*. ## Availability Zones Each Region has multiple, isolated locations known as *Availability Zones*. The code for an Availability Zone is its Region code followed by a letter identifier. For example, `us-east-1a`. By launching EC2 instances in multiple Availability Zones, you can protect your applications from the failure of a single location in the Region. The following diagram illustrates multiple Availability Zones in an AWS Region. Availability Zone A and Availability Zone B each have one subnet, and each subnet has EC2 instances. Availability Zone C has no subnets, therefore you can't launch instances into this Availability Zone. ![\[A Region with instances in one Availability Zone.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/region-with-azs.png) For more information, see [Virtual private clouds for your EC2 instances](using-vpc.md). ### Availability Zones by Region For the list of Availability Zones by Region, see [AWS Availability Zones](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-availability-zones.html). ### Instances in Availability Zones When you launch an instance, you select a Region and a virtual private cloud (VPC). Then, you can either select a subnet from one of the Availability Zones or let us choose a subnet for you. When you launch your initial instances, we recommend that you let us select an Availability Zone for you based on system health and available capacity. If you launch additional instances, specify an Availability Zone only if your new instances must be close to, or separated from, your existing instances. If you distribute instances across multiple Availability Zones and an instance fails, you can design your application so that an instance in another Availability Zone handles requests instead. For more information, see [AWS Availability Zones](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-availability-zones.html) in the *AWS Regions and Availability Zones User Guide*. ## Local Zones A Local Zone is an extension of an AWS Region in geographic proximity to your users. Local Zones have their own connections to the internet and support Direct Connect, so that resources created in a Local Zone can serve local users with low-latency communications. For more information, see [What is AWS Local Zones?](https://docs.aws.amazon.com/local-zones/latest/ug/what-is-aws-local-zones.html) in the *AWS Local Zones User Guide*. The code for a Local Zone is its Region code followed by an identifier that indicates its physical location. For example, `us-west-2-lax-1` in Los Angeles. The following diagram illustrates the AWS Region `us-west-2`, two of its Availability Zones, and two of its Local Zones. The VPC spans the Availability Zones and one of the Local Zones. Each zone in the VPC has one subnet, and each subnet has an instance. ![\[A VPC with Availability Zones and Local Zones.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/region-with-lzs.png) ### Available Local Zones For the list of available Local Zones, see [Available Local Zones](https://docs.aws.amazon.com/local-zones/latest/ug/available-local-zones.html) in the *AWS Local Zones User Guide*. For the list of announced Local Zones, see [AWS Local Zones locations](https://aws.amazon.com/about-aws/global-infrastructure/localzones/locations/). ### Instances in Local Zones To use a Local Zone, you must first enable it. Then, create a subnet in the Local Zone. You can specify the Local Zone subnet when you launch instances, which places it in the Local Zone subnet in the Local Zone. When you launch an instance in a Local Zone, you also allocate an IP address from a network border group. A network border group is a unique set of Availability Zones, Local Zones, or Wavelength Zones from which AWS advertises IP addresses, for example, `us-west-2-lax-1a`. You can allocate the following IP addresses from a network border group: + Amazon-provided Elastic IPv4 addresses + Amazon-provided IPv6 VPC addresses (available only in the Los Angeles zones) For more information about how to launch an instance in a Local Zone, see [Getting started with AWS Local Zones](https://docs.aws.amazon.com/local-zones/latest/ug/getting-started.html) in the *AWS Local Zones User Guide*. ## Wavelength Zones AWS Wavelength enables developers to build applications that deliver ultra-low latencies to mobile devices and end users. Wavelength deploys standard AWS compute and storage services to the edge of telecommunication carriers' 5G networks. Developers can extend a virtual private cloud (VPC) to one or more Wavelength Zones, and then use AWS resources like Amazon EC2 instances to run applications that require ultra-low latency and a connection to AWS services in the Region. A Wavelength Zone is an isolated zone in the carrier location where the Wavelength infrastructure is deployed. Wavelength Zones are tied to a Region. A Wavelength Zone is a logical extension of a Region, and is managed by the control plane in the Region. The code for a Wavelength Zone is its Region code followed by an identifier that indicates the physical location. For example, `us-east-1-wl1-bos-wlz-1` in Boston. The following diagram illustrates the AWS Region `us-west-2`, two of its Availability Zones, and a Wavelength Zone. The VPC spans the Availability Zones and the Wavelength Zone. Each zone in the VPC has one subnet, and each subnet has an instance. ![\[A VPC with Availability Zones and a Wavelength Zone.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/region-with-wavelength-zones.png) Wavelength Zones are not available in every Region. For information about the Regions that support Wavelength Zones, see [Available Wavelength Zones](https://docs.aws.amazon.com/wavelength/latest/developerguide/wavelength-quotas.html#concepts-available-zones) in the *AWS Wavelength Developer Guide*. ### Available Wavelength Zones For the list of available Wavelength Zones, see [Available Wavelength Zones](https://docs.aws.amazon.com/wavelength/latest/developerguide/available-wavelength-zones.html) in the *AWS Wavelength Guide*. ### Instances in Wavelength Zones To use a Wavelength Zone, you must first opt in to the Zone. Then, create a subnet in the Wavelength Zone. You can specify the Wavelength subnet when you launch instances. You also allocate a carrier IP address from a network border group, which is a unique set of Availability Zones, Local Zones, or Wavelength Zones from which AWS advertises IP addresses, for example, `us-east-1-wl1-bos-wlz-1`. For step-by-step directions to launch an instance in a Wavelength Zone, see [Get started with AWS Wavelength](https://docs.aws.amazon.com/wavelength/latest/developerguide/get-started-wavelength.html) in the *AWS Wavelength Developer Guide*. ## AWS Outposts AWS Outposts is a fully managed service that extends AWS infrastructure, services, APIs, and tools to customer premises. By providing local access to AWS managed infrastructure, AWS Outposts enables customers to build and run applications on premises using the same programming interfaces as in AWS Regions, while using local compute and storage resources for lower latency and local data processing needs. An Outpost is a pool of AWS compute and storage capacity deployed at a customer site. AWS operates, monitors, and manages this capacity as part of an AWS Region. You can create subnets on your Outpost and specify them when you create AWS resources. Instances in Outpost subnets communicate with other instances in the AWS Region using private IP addresses, all within the same VPC. The following diagram illustrates the AWS Region `us-west-2`, two of its Availability Zones, and an Outpost. The VPC spans the Availability Zones and the Outpost. The Outpost is in an on-premises customer data center. Each zone in the VPC has one subnet, and each subnet has an instance. ![\[A VPC with Availability Zones and an Outpost.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/region-with-outpost.png) ### Instances on an Outpost To begin using AWS Outposts, you must create an Outpost and order Outpost capacity. AWS Outposts offers two form factors, Outposts racks and Outposts servers. For more information about Outposts configurations, see [AWS Outposts Family](https://aws.amazon.com/outposts/). After your Outpost equipment is installed, the compute and storage capacity is available for you when you launch EC2 instances on your Outpost. To launch EC2 instances, you must create an Outpost subnet. Security groups control inbound and outbound traffic for instances in an Outpost subnet, just as they do for instances in an Availability Zone subnet. To connect to instances in Outpost subnets using SSH, specify a key pair when you launch them, just as you do for instances in Availability Zone subnets. For more information, see [Get started with Outposts racks](https://docs.aws.amazon.com/outposts/latest/userguide/get-started-outposts.html) or [Get started with Outposts servers](https://docs.aws.amazon.com/outposts/latest/server-userguide/get-started-outposts.html). ### Volumes on an Outposts rack If your Outposts compute capacity is on an Outpost rack, you can create EBS volumes in the Outpost subnet that you created. When you create the volume, specify the Amazon Resource Name (ARN) of the Outpost. The following [create-volume](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-volume.html) command creates an empty 50 GB volume on the specified Outpost. ``` aws ec2 create-volume --availability-zone us-east-2a --outpost-arn arn:aws:outposts:us-east-2:123456789012:outpost/op-03e6fecad652a6138 --size 50 ``` You can dynamically modify the size of your Amazon EBS gp2 volumes without detaching them. For more information about modifying a volume without detaching it, see [Request modifications to your EBS volumes](https://docs.aws.amazon.com/ebs/latest/userguide/requesting-ebs-volume-modifications.html) in the *Amazon EBS User Guide*. We recommend that you limit the root volume for an instance on an Outpost rack to 30 GiB or smaller. You can specify data volumes in the block device mapping of the AMI or the instance to provide additional storage. To trim unused blocks from the boot volume, see [How to Build Sparse EBS Volumes](https://aws.amazon.com/blogs/apn/how-to-build-sparse-ebs-volumes-for-fun-and-easy-snapshotting/) in the *AWS Partner Network Blog*. We recommend that you increase the NVMe timeout for the root volume. For more information, see [I/O operation timeout](https://docs.aws.amazon.com/ebs/latest/userguide/nvme-ebs-volumes.html#timeout-nvme-ebs-volumes) in the *Amazon EBS User Guide*. ### Volumes on an Outposts server Instances on Outposts servers provide instance store volumes but do not support EBS volumes. Choose an Amazon EBS-backed AMI with only a single EBS snapshot. Choose an instance size with enough instance storage to meet the needs of your application. For more information, see [Instance store volume limits](instance-store-volumes.md). # Amazon EC2 instance IP addressing Instance IP addressing Amazon EC2 and Amazon VPC support both the IPv4 and IPv6 addressing protocols. By default, Amazon VPC uses the IPv4 addressing protocol; you can't disable this behavior. When you create a VPC, you must specify an IPv4 CIDR block (a range of private IPv4 addresses). You can optionally assign an IPv6 CIDR block to your VPC and assign IPv6 addresses from that block to instances in your subnets. When you launch an EC2 instance, you specify a VPC and a subnet. The instance receives a private IPv4 address from the CIDR range of the subnet. You can optionally configure your instances with public IPv4 addresses and IPv6 addresses. If EC2 instances in different VPCs communicate using public IP addresses, the traffic stays in the AWS private global network and does not traverse the public internet. **Topics** + [ ## Private IPv4 addresses ](#concepts-private-addresses) + [ ## Public IPv4 addresses ](#concepts-public-addresses) + [ ## Public IPv4 address optimization ](#concepts-public-ip-address-opt) + [ ## IPv6 addresses ](#ipv6-addressing) + [ ## Multiple IP addresses ](#multiple-ip-addresses) + [ ## EC2 instance hostnames ](#amazon-dns) + [ ## Link-local addresses ](#link-local-addresses) + [ # Manage the IPv4 addresses for your EC2 instances ](working-with-ip-addresses.md) + [ # Manage the IPv6 addresses for your EC2 instances ](working-with-ipv6-addresses.md) + [ # Secondary IP addresses for your EC2 instances ](instance-secondary-ip-addresses.md) + [ # Configure secondary private IPv4 addresses for Windows instances ](config-windows-multiple-ip.md) ## Private IPv4 addresses A private IPv4 address is an IP address that's not reachable over the Internet. You can use private IPv4 addresses for communication between instances in the same VPC. For more information about the standards and specifications of private IPv4 addresses, see [RFC 1918](http://www.faqs.org/rfcs/rfc1918.html). We allocate private IPv4 addresses to instances using DHCP. **Note** You can create a VPC with a publicly routable CIDR block that falls outside of the private IPv4 address ranges specified in RFC 1918. However, for the purposes of this documentation, we refer to private IPv4 addresses (or 'private IP addresses') as the IP addresses that are within the IPv4 CIDR range of your VPC. VPC subnets can be one of the following types: + IPv4-only subnets – You can only create resources in these subnets with IPv4 addresses assigned to them. + IPv6-only subnets – You can only create resources in these subnets with IPv6 addresses assigned to them. + IPv4 and IPv6 subnets – You can create resources in these subnets with either IPv4 or IPv6 addresses assigned to them. When you launch an EC2 instance into an IPv4-only or dual stack (IPv4 and IPv6) subnet, the instance receives a primary private IP address from the IPv4 address range of the subnet. For more information, see [IP addressing](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html) in the *Amazon VPC User Guide*. If you don't specify a primary private IP address when you launch the instance, we select an available IP address in the subnet's IPv4 range for you. Each instance has a default network interface (index 0) that is assigned the primary private IPv4 address. You can also specify additional private IPv4 addresses, known as *secondary private IPv4 addresses*. Unlike primary private IP addresses, secondary private IP addresses can be reassigned from one instance to another. For more information, see [Multiple IP addresses](#multiple-ip-addresses). A private IPv4 address, regardless of whether it is a primary or secondary address, remains associated with the network interface when the instance is stopped and started, or hibernated and started, and is released when the instance is terminated. ## Public IPv4 addresses A public IP address is an IPv4 address that's reachable from the Internet. You can use public addresses for communication between your instances and the Internet. When you launch an instance in a default VPC, we assign it a public IP address by default. When you launch an instance into a nondefault VPC, the subnet has an attribute that determines whether instances launched into that subnet receive a public IP address from the public IPv4 address pool. By default, we don't assign a public IP address to instances launched in a nondefault subnet. You can control whether your instance receives a public IP address as follows: + **Modify the public IP addressing attribute of your subnet.** For more information, see [Modify the public IPv4 addressing attribute for your subnet](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-public-ip.html) in the *Amazon VPC User Guide*. + **Enable or disable the public IP addressing feature during launch.** This overrides the subnet's public IP addressing attribute. For more information, see [Assign a public IPv4 address at launch](working-with-ip-addresses.md#public-ip-addresses). + **Unassign a public IP address from your instance after launch.** For more information, see [Manage the IP addresses for your network interface](managing-network-interface-ip-addresses.md). A public IP address is assigned to your instance from Amazon's pool of public IPv4 addresses, and is not associated with your AWS account. When a public IP address is disassociated from your instance, it is released back into the public IPv4 address pool, and you cannot reuse it. We release the public IP address from your instance and assign a new one in the following cases: + We release the public IP address when the instance is stopped, hibernated, or terminated. We assign a new public IP address when you start your stopped or hibernated instance. + We release the public IP address when you associate an Elastic IP address with the instance. We assign a new public IP address when you disassociate the Elastic IP address from your instance. + If we release the public IP address of your instance and it has a secondary network interface, we do not assign a new public IP address. + If we release the public IP address of your instance and it has a secondary private IP address that is associated with an Elastic IP address, we do not assign a new public IP address. If you require a persistent public IP address that can be associated to and from instances as you require, use an Elastic IP address instead. If you use dynamic DNS to map an existing DNS name to a new instance's public IP address, it might take up to 24 hours for the IP address to propagate through the Internet. As a result, new instances might not receive traffic while terminated instances continue to receive requests. To solve this problem, use an Elastic IP address. You can allocate your own Elastic IP address, and associate it with your instance. For more information, see [Elastic IP addresses](elastic-ip-addresses-eip.md). If you are using Amazon VPC IP Address Manager (IPAM), you can get a contiguous block of public IPv4 addresses from AWS and use it to allocate Elastic IP addresses to AWS resources. Using contiguous IPv4 address blocks can significantly reduce management overhead for security access control lists and simplify IP address allocation and tracking for enterprises scaling on AWS. For more information, see [Allocate sequential Elastic IP addresses from an IPAM pool](https://docs.aws.amazon.com/vpc/latest/ipam/tutorials-eip-pool.html) in the *Amazon VPC IPAM User Guide*. **Considerations** + AWS charges for all public IPv4 addresses, including public IPv4 addresses associated with running instances and Elastic IP addresses. For more information, see the **Public IPv4 Address** tab on the [Amazon VPC pricing page](https://aws.amazon.com/vpc/pricing/). + Instances that access other instances through their public NAT IP address are charged for regional or Internet data transfer, depending on whether the instances are in the same Region. ## Public IPv4 address optimization AWS charges for all public IPv4 addresses, including public IPv4 addresses associated with running instances and Elastic IP addresses. For more information, see the **Public IPv4 Address** tab on the [Amazon VPC pricing page](https://aws.amazon.com/vpc/pricing/). The following list contains actions you can take to optimize the number of public IPv4 addresses you use: + Use an [elastic load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/load-balancer-getting-started.html) to load balance traffic to your EC2 instances and [disable **Auto-assign public IP** on the primary ENI assigned to the instances](managing-network-interface-ip-addresses.md). Load balancers use a single public IPv4 address, so this reduces your public IPv4 address count. You may also want consolidate existing load balancers to further reduce the public IPv4 address count. + If the only reason for using a NAT gateway is to SSH into an EC2 instance in a private subnet for maintenance or emergencies, consider using [EC2 Instance Connect Endpoint](connect-using-eice.md) instead. With EC2 Instance Connect Endpoint, you can connect to an instance from the internet without requiring the instance to have a public IPv4 address. + If your EC2 instances are in a public subnet with public IP addresses allocated to them, consider moving the instances to a private subnet, removing the public IP addresses, and using a [public NAT gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) to allow access to and from your EC2 instances. There are cost considerations for using NAT gateways. Use this calculation method to decide if NAT gateways are cost effective. You can get the `Number of public IPv4 addresses` required for this calculation by [creating an AWS Billing Cost and Usage Report](https://aws.amazon.com/blogs/networking-and-content-delivery/identify-and-optimize-public-ipv4-address-usage-on-aws/). ``` NAT gateway per hour + NAT gateway public IPs + NAT gateway transfer / Existing public IP cost ``` Where: + `NAT gateway per hour = $0.045 * 730 hours in a month * Number of Availability Zones the NAT gateways are in` + `NAT gateway public IPs = $0.005 * 730 hours in a month * Number of IPs associated with your NAT gateways` + `NAT gateway transfer = $0.045 * Number of GBs that will go through the NAT gateway in a month` + `Existing public IP cost = $0.005 * 730 hours in a month * Number of public IPv4 addresses` If the total is less than 1, NAT gateways are cheaper than public IPv4 addresses. + Use [AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-services-overview.html) to connect privately to AWS services or services hosted by other AWS accounts rather than using public IPv4 addresses and internet gateways. + [Bring your own IP address range (BYOIP) to AWS](ec2-byoip.md) and use the range for public IPv4 addresses rather than using Amazon-owned public IPv4 addresses. + Turn off [auto-assign public IPv4 address for instances launched into subnets](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-public-ip.html). This option is generally disabled by default for VPCs when you create a subnet, but you should check your existing subnets to ensure it’s disabled. + If you have EC2 instances that do not need public IPv4 addresses, [ check that the network interfaces attached to your instances have **Auto-assign public IP** disabled](managing-network-interface-ip-addresses.md). + [Configure accelerator endpoints in AWS Global Accelerator](https://docs.aws.amazon.com/global-accelerator/latest/dg/about-endpoints.html) for EC2 instances in private subnets to enable internet traffic to flow directly to the endpoints in your VPCs without requiring public IP addresses. You can also [bring your own addresses to AWS Global Accelerator](https://docs.aws.amazon.com/global-accelerator/latest/dg/using-byoip.html) and use your own IPv4 addresses for your accelerator’s static IP addresses. ## IPv6 addresses IPv6 addresses are globally unique and can be configured to remain private or reachable over the Internet. Both public and private IPv6 addressing is available in AWS: + **Private IPv6**: AWS considers private IPv6 addresses those that are not advertised and cannot be advertised on the Internet from AWS. + **Public IPv6**: AWS considers public IPv6 addresses those that are advertised on the Internet from AWS. For more information about public and private IPv6 addresses, see [IPv6 addresses](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html#vpc-ipv6-addresses) in the *Amazon VPC User Guide*. All instance types support IPv6 addresses except for the following: C1, M1, M2, M3, and T1. Your EC2 instances receive an IPv6 address if an IPv6 CIDR block is associated with your VPC and subnet, and if one of the following is true: + Your subnet is configured to automatically assign an IPv6 address to an instance during launch. For more information, see [Modify the IP addressing attributes of your subnet](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-public-ip.html). + You assign an IPv6 address to your instance during launch. + You assign an IPv6 address to the primary network interface of your instance after launch. + You assign an IPv6 address to a network interface in the same subnet, and attach the network interface to your instance after launch. When your instance receives an IPv6 address during launch, the address is associated with the primary network interface (index 0) of the instance. You can manage the IPv6 addresses for your instances primary network interface as follows: + Assign and unassign IPv6 addresses from the network interface. The number of IPv6 addresses you can assign to a network interface and the number of network interfaces you can attach to an instance varies per instance type. For more information, see [Maximum IP addresses per network interface](AvailableIpPerENI.md). + Enable a primary IPv6 address. A primary IPv6 address enables you to avoid disrupting traffic to instances or ENIs. For more information, see [Create a network interface for your EC2 instance](create-network-interface.md) or [Manage the IP addresses for your network interface](managing-network-interface-ip-addresses.md). An IPv6 address persists when you stop and start, or hibernate and start, your instance, and is released when you terminate your instance. You cannot reassign an IPv6 address while it's assigned to another network interface—you must first unassign it. You can control whether instances are reachable via their IPv6 addresses by controlling the routing for your subnet or by using security group and network ACL rules. For more information, see [Internetwork traffic privacy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access.html) in the *Amazon VPC User Guide*. For more information about reserved IPv6 address ranges, see [IANA IPv6 Special-Purpose Address Registry](http://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml) and [RFC4291](https://tools.ietf.org/html/rfc4291). ## Multiple IP addresses You can specify multiple private IPv4 and IPv6 addresses for your instances. The number of network interfaces and private IPv4 and IPv6 addresses that you can specify for an instance depends on the instance type. For more information, see [Maximum IP addresses per network interface](AvailableIpPerENI.md). **Use cases** + Host multiple websites on a single server by using multiple SSL certificates on a single server and associating each certificate with a specific IP address. + Operate network appliances, such as firewalls or load balancers, that have multiple IP addresses for each network interface. + Redirect internal traffic to a standby instance in case your instance fails, by reassigning the secondary IP address to the standby instance. **How multiple IP addresses work** + You can assign a secondary private IPv4 address to any network interface. + You can assign multiple IPv6 addresses to a network interface that's in a subnet that has an associated IPv6 CIDR block. + You must choose a secondary IPv4 address from the IPv4 CIDR block range of the subnet for the network interface. + You must choose IPv6 addresses from the IPv6 CIDR block range of the subnet for the network interface. + You associate security groups with network interfaces, not individual IP addresses. Therefore, each IP address you specify in a network interface is subject to the security group of its network interface. + Multiple IP addresses can be assigned and unassigned to network interfaces attached to running or stopped instances. + Secondary private IPv4 addresses that are assigned to a network interface can be reassigned to another one if you explicitly allow it. + An IPv6 address cannot be reassigned to another network interface; you must first unassign the IPv6 address from the existing network interface. + When assigning multiple IP addresses to a network interface using the command line tools or API, the entire operation fails if one of the IP addresses can't be assigned. + Primary private IPv4 addresses, secondary private IPv4 addresses, Elastic IP addresses, and IPv6 addresses remain with a secondary network interface when it is detached from an instance or attached to an instance. + Although you can't detach the primary network interface from an instance, you can reassign the secondary private IPv4 address of the primary network interface to another network interface. For more information, see [Secondary IP addresses for your EC2 instances](instance-secondary-ip-addresses.md). ## EC2 instance hostnames When you create an EC2 instance, AWS creates a hostname for that instance. For more information on the types of hostnames and how they're provisioned by AWS, see [EC2 instance hostnames and domains](ec2-instance-naming.md). Amazon provides a DNS server that resolves Amazon-provided hostnames to IPv4 and IPv6 addresses. The Amazon DNS server is located at the base of your VPC network range plus two. For more information, see [DNS attributes for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html) in the *Amazon VPC User Guide*. ## Link-local addresses Link-local addresses are well-known, non-routable IP addresses. Amazon EC2 uses addresses from the link-local address space to provide services that are accessible only from an EC2 instance. These services do not run on the instance, they run on the underlying host. When you access the link-local addresses for these services, you're communicating with either the Xen hypervisor or the Nitro controller. **Link-local address ranges** + IPv4 – 169.254.0.0/16 (169.254.0.0 to 169.254.255.255) + IPv6 – fe80::/10 **Services that you access using link-local addresses** + [Instance Metadata Service](instancedata-data-retrieval.md) + [Amazon Route 53 Resolver](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#AmazonDNS) (also known as the Amazon DNS server) + [Amazon Time Sync Service](set-time.md) + [AWS KMS servers](common-messages.md#activate-windows) # Manage the IPv4 addresses for your EC2 instances IPv4 addresses You can assign a public IPv4 address to your instance when you launch it. You can view the IPv4 addresses for your instance in the console through either the **Instances** page or the **Network Interfaces** page. **Topics** + [ ## Assign a public IPv4 address at launch ](#public-ip-addresses) + [ ## Assign a private IPv4 address at launch ](#assign-private-ipv4-address) + [ ## View the primary IPv4 address ](#view-instance-ipv4-addresses) + [ ## View the IPv4 addresses using instance metadata ](#view-instance-ipv4-addresses-imds) ## Assign a public IPv4 address at launch Each subnet has an attribute that determines whether instances launched into that subnet are assigned a public IP address. By default, nondefault subnets have this attribute set to false, and default subnets have this attribute set to true. When you launch an instance, a public IPv4 addressing feature is also available for you to control whether your instance is assigned a public IPv4 address; you can override the default behavior of the subnet's IP addressing attribute. The public IPv4 address is assigned from Amazon's pool of public IPv4 addresses, and is assigned to the network interface with the device index of 0. This feature depends on certain conditions at the time you launch your instance. **Considerations** + You can unassign the public IP address from your instance after launch by [managing the IP addresses associated with a network interface](managing-network-interface-ip-addresses.md). For more information about public IPv4 addresses, see [Public IPv4 addresses](using-instance-addressing.md#concepts-public-addresses). + You can't auto-assign a public IP address if you specify more than one network interface. Additionally, you cannot override the subnet setting using the auto-assign public IP feature if you specify an existing network interface for device index 0. + Whether you assign a public IP address to your instance during launch or not, you can associate an Elastic IP address with your instance after it's launched. For more information, see [Elastic IP addresses](elastic-ip-addresses-eip.md). You can also modify your subnet's public IPv4 addressing behavior. For more information, see [Modify the public IPv4 addressing attribute for your subnet](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-public-ip.html). ------ #### [ Console ] **To assign a public IPv4 address at launch** Follow the procedure to [launch an instance](ec2-launch-instance-wizard.md), and when you configure [Network Settings](ec2-instance-launch-parameters.md#liw-network-settings), choose the option to **Auto-assign Public IP**. ------ #### [ AWS CLI ] **To assign a public IPv4 address at launch** Use the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command with the `--associate-public-ip-address` option. ``` --associate-public-ip-address ``` ------ #### [ PowerShell ] **To assign a public IPv4 address at launch** Use the [New-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2Instance.html) cmdlet with the `-AssociatePublicIp` parameter. ``` -AssociatePublicIp $true ``` ------ ## Assign a private IPv4 address at launch You can specify a private IPv4 address from the IPv4 address range of the subnet, or let Amazon EC2 chose one for you. This address is assigned to the primary network interface. To assign IPv4 addresses after launch, see [Assign secondary IP addresses to an instance](instance-secondary-ip-addresses.md#assign-secondary-ip-address). ------ #### [ Console ] **To assign a private IPv4 address at launch** Follow the procedure to [launch an instance](ec2-launch-instance-wizard.md). When you configure [Network Settings](ec2-instance-launch-parameters.md#liw-network-settings), expand **Advanced network configuration** and enter a value for **Primary IP**. ------ #### [ AWS CLI ] **To assign a private IPv4 address at launch** Use the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command with the `--private-ip-address` option. ``` --private-ip-addresses 10.251.50.12 ``` To let Amazon EC2 choose the IP address, omit this option. ------ #### [ PowerShell ] **To assign a private IPv4 address at launch** Use the [New-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2Instance.html) cmdlet with the `-PrivateIpAddress` parameter. ``` -PrivateIpAddress 10.251.50.12 ``` To let Amazon EC2 choose the IP address, omit this parameter. ------ ## View the primary IPv4 address The public IPv4 address is displayed as a property of the network interface in the console, but it's mapped to the primary private IPv4 address through NAT. Therefore, if you inspect the properties of your network interface on your instance, for example, through `ifconfig` (Linux) or `ipconfig` (Windows), the public IPv4 address is not displayed. ------ #### [ Console ] **To view the IPv4 addresses for an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the instance. 1. On the **Networking** tab, find **Public IPv4 address** and **Private IPv4 addresses**. 1. (Optional) The **Networking** tab also contains the network interfaces and Elastic IP addresses for the instance. ------ #### [ AWS CLI ] **To view the primary IPv4 address for an instance** Use the [describe-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html) command. ``` aws ec2 describe-instances \ --instance-ids i-1234567890abcdef0 \ --query "Reservations[].Instances[].PrivateIpAddress" \ --output text ``` The following is example output. ``` 10.251.50.12 ``` ------ #### [ PowerShell ] **To view the primary IPv4 address for an instance** Use the [Get-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2Instance.html) cmdlet. ``` (Get-EC2Instance ` -InstanceId i-1234567890abcdef0).Instances.PrivateIpAddress ``` The following is example output. ``` 10.251.50.12 ``` ------ ## View the IPv4 addresses using instance metadata You can get the IPv4 addresses for your instance by retrieving instance metadata. For more information, see [Use instance metadata to manage your EC2 instance](ec2-instance-metadata.md). **To view the IPv4 addresses using instance metadata** 1. Connect to your instance. For more information, see [Connect to your EC2 instance](connect.md). 1. Run one of the following commands. ------ #### [ IMDSv2 ] **Linux** Run the following command from your Linux instance. ``` TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \ && curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-ipv4 ``` **Windows** Run the following command from your Windows instance. ``` [string]$token = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} ` -Method PUT -Uri http://169.254.169.254/latest/api/token ``` ``` Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} ` -Method GET -Uri http://169.254.169.254/latest/meta-data/local-ipv4 ``` ------ #### [ IMDSv1 ] **Linux** Run the following command from your Linux instance. ``` curl http://169.254.169.254/latest/meta-data/local-ipv4 ``` **Windows** Run the following command from your Windows instance. ``` Invoke-RestMethod http://169.254.169.254/latest/meta-data/local-ipv4 ``` ------ 1. Use one of the following commands to access the public IP address. If there is an Elastic IP address associated with the instance, the command returns the Elastic IP address. ------ #### [ IMDSv2 ] **Linux** Run the following command from your Linux instance. ``` [ec2-user ~]$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \ && curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/public-ipv4 ``` **Windows** Run the following command from your Windows instance. ``` [string]$token = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} ` -Method PUT -Uri http://169.254.169.254/latest/api/token ``` ``` Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} ` -Method GET -Uri http://169.254.169.254/latest/meta-data/public-ipv4 ``` ------ #### [ IMDSv1 ] **Linux** Run the following command from your Linux instance. ``` curl http://169.254.169.254/latest/meta-data/public-ipv4 ``` **Windows** Run the following command from your Windows instance. ``` Invoke-RestMethod http://169.254.169.254/latest/meta-data/public-ipv4 ``` ------ # Manage the IPv6 addresses for your EC2 instances IPv6 addresses If your VPC and subnet have IPv6 CIDR blocks associated with them, you can assign an IPv6 address to your instance during or after launch. You can access the IPv6 addresses for your instances in the console on either the **Instances** page or the **Network Interfaces** page. The following tasks configure IP addresses for your instances. To configure IP addresses for your network interfaces instead, see [Manage the IP addresses for your network interface](managing-network-interface-ip-addresses.md). **Topics** + [ ## Assign an IPv6 address to an instance ](#assign-ipv6-address) + [ ## View the IPv6 addresses for an instance ](#view-ipv6-addresses) + [ ## View IPv6 addresses using instance metadata ](#view-ipv6-addresses-imds) + [ ## Unassign an IPv6 address from an instance ](#unassign-ipv6-address) ## Assign an IPv6 address to an instance You can specify an IPv6 address from the IPv6 address range of the subnet, or let Amazon EC2 choose one for you. This address is assigned to the primary network interface. Note that the following instance types do not support IPv6 addresses: C1, M1, M2, M3, and T1. ------ #### [ Console ] **To assign an IPv6 address at launch** Follow the procedure to [launch an instance](ec2-launch-instance-wizard.md). When you configure [Network Settings](ec2-instance-launch-parameters.md#liw-network-settings), choose the option to **Auto-assign IPv6 IP**. If you don't see this option, the selected subnet does not have an associated IPv6 CIDR block. **To assign an IPv6 address after launch** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select your instance, and choose **Actions**, **Networking**, **Manage IP addresses**. 1. Expand the network interface. Under **IPv6 addresses**, choose **Assign new IP address**. 1. Enter an IPv6 address from the range of the subnet, or leave the field blank to let Amazon EC2 choose the IPv6 address for you. If you don't see this option, the instance subnet does not have an associated IPv6 CIDR block. 1. Choose **Save**. ------ #### [ AWS CLI ] **To assign an IPv6 address at launch** Use the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command with the `--ipv6-addresses` option. The following example assigns two IPv6 addresses. ``` --ipv6-addresses Ipv6Address=2001:db8::1234:5678:1.2.3.4 Ipv6Address=2001:db8::1234:5678:5.6.7.8 ``` To let Amazon EC2 choose the IPv6 addresses, use the `--ipv6-address-count` option instead. The following example assigns two IPv6 addresses. ``` --ipv6-address-count 2 ``` **To assign an IPv6 address after launch** Use the [assign-ipv6-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-ipv6-addresses.html) command. The following example assigns two IPv6 addresses. ``` aws ec2 assign-ipv6-addresses \ --network-interface-id eni-1234567890abcdef0 \ --ipv6-addresses 2001:db8::1234:5678:1.2.3.4 2001:db8::1234:5678:5.6.7.8 ``` To let Amazon EC2 choose the IPv6 addresses, use the `--ipv6-address-count` option instead. The following example assigns two IPv6 addresses. ``` aws ec2 assign-ipv6-addresses \ --network-interface-id eni-1234567890abcdef0 \ --ipv6-address-count 2 ``` ------ #### [ PowerShell ] **To assign an IPv6 address at launch** Use the [New-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2Instance.html) cmdlet with the `-Ipv6Address` parameter. The following example assigns two IPv6 addresses. ``` -Ipv6Address $ipv6addr1,$ipv6addr2 ``` Define the IPv6 addresses as follows. ``` $ipv6addr1 = New-Object Amazon.EC2.Model.InstanceIpv6Address $ipv6addr1.Ipv6Address = "2001:db8::1234:5678:1.2.3.4" $ipv6addr2 = New-Object Amazon.EC2.Model.InstanceIpv6Address $ipv6addr2.Ipv6Address = "2001:db8::1234:5678:5.6.7.8" ``` To let Amazon EC2 choose the IPv6 addresses, use the `-Ipv6AddressCount` parameter instead. The following example assigns two IPv6 addresses. ``` -Ipv6AddressCount 2 ``` **To assign an IPv6 address after launch** Use the [Register-EC2Ipv6AddressList](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Ipv6AddressList.html) cmdlet. The following example assigns two IPv6 addresses. ``` Register-EC2Ipv6AddressList ` -NetworkInterfaceId eni-1234567890abcdef0 ` -Ipv6Address "2001:db8::1234:5678:1.2.3.4","2001:db8::1234:5678:5.6.7.8" ``` To let Amazon EC2 choose the IPv6 addresses, use the `-Ipv6AddressCount` parameter instead. The following example assigns two IPv6 addresses. ``` Register-EC2Ipv6AddressList ` -NetworkInterfaceId eni-1234567890abcdef0 ` -Ipv6AddressCount 2 ``` ------ ## View the IPv6 addresses for an instance You can view the IPv6 addresses for your instances. ------ #### [ Console ] **To view the IPv6 addresses for an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the instance. 1. On the **Networking** tab, locate **IPv6 addresses**. ------ #### [ AWS CLI ] **To view the IPv6 address for an instance** Use the [describe-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html) command. ``` aws ec2 describe-instances \ --instance-ids i-1234567890abcdef0 \ --query "Reservations[*].Instances[].Ipv6Address" \ --output text ``` The following is example output. ``` 2001:db8::1234:5678:1.2.3.4 ``` ------ #### [ PowerShell ] **To view the IPv6 address for an instance** Use the [Get-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2Instance.html) cmdlet. ``` (Get-EC2Instance ` -InstanceId i-1234567890abcdef0).Instances.Ipv6Address ``` The following is example output. ``` 2001:db8::1234:5678:1.2.3.4 ``` ------ ## View IPv6 addresses using instance metadata After you connect to your instance, you can retrieve the IPv6 addresses using instance metadata. First, you must get the MAC address of the instance from `http://169.254.169.254/latest/meta-data/network/interfaces/macs/`. ------ #### [ IMDSv2 ] **Linux** Run the following command from your Linux instance. ``` TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \ && curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/network/interfaces/macs/mac-address/ipv6s ``` **Windows** Run the following cmdlets from your Windows instance. ``` [string]$token = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} ` -Method PUT -Uri http://169.254.169.254/latest/api/token ``` ``` Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} ` -Method GET -Uri http://169.254.169.254/latest/meta-data/network/interfaces/macs/mac-address/ipv6s ``` ------ #### [ IMDSv1 ] **Linux** Run the following command from your Linux instance. ``` curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/mac-address/ipv6s ``` **Windows** Run the following cmdlet from your Windows instance. ``` Invoke-RestMethod -Uri http://169.254.169.254/latest/meta-data/network/interfaces/macs/mac-address/ipv6s ``` ------ ## Unassign an IPv6 address from an instance You can unassign an IPv6 address from an instance at any time. ------ #### [ Console ] **To unassign an IPv6 address from an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select your instance, and choose **Actions**, **Networking**, **Manage IP addresses**. 1. Expand the network interface. Under **IPv6 addresses**, choose **Unassign** next to the IPv6 address. 1. Choose **Save**. ------ #### [ AWS CLI ] **To unassign an IPv6 address from an instance** Use the [unassign-ipv6-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/unassign-ipv6-addresses.html) command. ``` aws ec2 unassign-ipv6-addresses \ --network-interface-id eni-1234567890abcdef0 \ --ipv6-addresses 2001:db8::1234:5678:1.2.3.4 ``` ------ #### [ PowerShell ] **To unassign an IPv6 address from an instance** Use the [Unregister-EC2Ipv6AddressList](https://docs.aws.amazon.com/powershell/latest/reference/items/Unregister-EC2Ipv6AddressList.html) cmdlet. ``` Unregister-EC2Ipv6AddressList ` -NetworkInterfaceId eni-1234567890abcdef0 ` -Ipv6Address 2001:db8::1234:5678:1.2.3.4 ``` ------ # Secondary IP addresses for your EC2 instances Secondary IP addresses The first IPv4 address assigned to a network interface is known as the primary IP address. Secondary IP addresses are additional IPv4 address assigned to a network interface. For more information, see [Multiple IP addresses](using-instance-addressing.md#multiple-ip-addresses). You can also assign multiple IPv6 addresses to an instance. For more information, see [Manage the IPv6 addresses for your EC2 instances](working-with-ipv6-addresses.md). **Topics** + [ ## Assign secondary IP addresses to an instance ](#assign-secondary-ip-address) + [ ## Configure the operating system to use secondary IP addresses ](#StepTwoConfigOS) + [ ## Unassign a secondary IP address from an instance ](#unassign-secondary-ip-address) ## Assign secondary IP addresses to an instance Assign secondary IP addresses to an instance You can assign secondary IP addresses to the network interface for an instance as you launch the instance, or after the instance is running. ------ #### [ Console ] **To assign a secondary IP address at launch** 1. Follow the procedure to [launch an instance](ec2-launch-instance-wizard.md). When you configure [Network Settings](ec2-instance-launch-parameters.md#liw-network-settings), expand **Advanced network configuration**. 1. For **Secondary IP**, choose **Automatically assign** and enter the number of IP addresses for Amazon EC2 to assign. Alternatively, choose **Manually assign** and enter the IPv4 addresses. 1. Complete the remaining steps to launch the instance. **To assign a secondary IP address after launch** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select your instance, and choose **Actions**, **Networking**, **Manage IP addresses**. 1. Expand the network interface. 1. To add an IPv4 address, under **IPv4 addresses**, choose **Assign new IP address**. Enter an IPv4 address from the range of the subnet, or leave the field blank to let Amazon EC2 choose one for you. 1. Choose **Save**. ------ #### [ AWS CLI ] **To assign a secondary IP address at launch** Use the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command with the `--secondary-private-ip-addresses` option. ``` --secondary-private-ip-addresses 10.251.50.12 ``` To let Amazon EC2 choose the IP address, use the `--secondary-private-ip-address-count` option instead. The following example assigns one secondary IP address. ``` --secondary-private-ip-address-count 1 ``` Alternatively, you can create a network interface. For more information, see [Create a network interface for your EC2 instance](create-network-interface.md). **To assign a secondary IP address after launch** Use the [assign-private-ip-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-private-ip-addresses.html) command with the `--private-ip-addresses` option. ``` aws ec2 assign-private-ip-addresses \ --network-interface-ids eni-1234567890abcdef0 \ --private-ip-addresses 10.251.50.12 ``` To let Amazon EC2 choose the IPv4 address, use the `--secondary-private-ip-address-count` parameter instead. The following example assigns one IPv4 address. ``` aws ec2 assign-private-ip-addresses \ --network-interface-ids eni-1234567890abcdef0 \ --secondary-private-ip-address-count 1 ``` ------ #### [ PowerShell ] **To assign a secondary IP address at launch** You must create a network interface. For more information, see [Create a network interface for your EC2 instance](create-network-interface.md). **To assign a secondary IP address after launch** Use the [Register-EC2PrivateIpAddress](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2PrivateIpAddress.html) cmdlet with the `-PrivateIpAddress` parameter. ``` Register-EC2PrivateIpAddress ` -NetworkInterfaceId eni-1234567890abcdef0 ` -PrivateIpAddress 10.251.50.12 ``` To let Amazon EC2 choose the IPv4 addresses, use the `-SecondaryPrivateIpAddressCount` parameter instead. The following example assigns one IPv4 address. ``` Register-EC2PrivateIpAddress ` -NetworkInterfaceId eni-1234567890abcdef0 ` -SecondaryPrivateIpAddressCount 1 ``` ------ ## Configure the operating system to use secondary IP addresses Configure the OS to use secondary IP addresses After you assign a secondary IP address to your instance, you must configure the operating system on your instance to recognize the additional private IPv4 address. **Linux instances** + If you are using Amazon Linux, the ec2-net-utils package can take care of this step for you. It configures additional network interfaces that you attach while the instance is running, refreshes secondary IPv4 addresses during DHCP lease renewal, and updates the related routing rules. You can immediately refresh the list of interfaces by using one of the following commands, depending on your system: `sudo systemctl restart systemd-networkd` (AL2023) or `sudo service network restart` (Amazon Linux 2). You can view the up-to-date list using the following command: `ip addr li`. If you require manual control over your network configuration, you can remove the ec2-net-utils package. For more information, see [Configure your network interface using ec2-net-utils](https://docs.aws.amazon.com/linux/al2/ug/ec2-net-utils.html). + If you are using another Linux distribution, see the documentation for your Linux distribution. Search for information about configuring additional network interfaces and secondary IPv4 addresses. If the instance has two or more interfaces on the same subnet, search for information about using routing rules to work around asymmetric routing. **Windows instances** For more information, see [Configure secondary private IPv4 addresses for Windows instances](config-windows-multiple-ip.md). ## Unassign a secondary IP address from an instance If you no longer require a secondary IP address, you can unassign it from the instance or the network interface. When a secondary private IPv4 address is unassigned from a network interface, the Elastic IP address (if it exists) is also disassociated. ------ #### [ Console ] **To unassign a secondary private IPv4 address from an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select an instance, choose **Actions**, **Networking**, **Manage IP addresses**. 1. Expand the network interface. For **IPv4 addresses**, choose **Unassign** for the IPv4 address to unassign. 1. Choose **Save**. ------ #### [ AWS CLI ] **To unassign a secondary private IP address** Use the [unassign-private-ip-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/unassign-private-ip-addresses.html) command. ``` aws ec2 unassign-private-ip-addresses \ --network-interface eni-1234567890abcdef0\ --private-ip-addresses 10.251.50.12 ``` ------ #### [ PowerShell ] **To unassign a secondary private IP address** Use the [Unregister-EC2PrivateIpAddress](https://docs.aws.amazon.com/powershell/latest/reference/items/Unregister-EC2PrivateIpAddress.html) cmdlet. ``` Unregister-EC2PrivateIpAddress ` -NetworkInterface eni-1234567890abcdef0 ` -PrivateIpAddress 10.251.50.12 ``` ------ # Configure secondary private IPv4 addresses for Windows instances IPv4 addresses on Windows You can specify multiple private IPv4 addresses for your instances. After you assign a secondary private IPv4 address to an instance, you must configure the operating system on the instance to recognize the secondary private IPv4 address. **Note** These instructions are based on Windows Server 2022. The implementation of these steps might vary based on the operating system of the Windows instance. **Topics** + [ ## Prerequisites ](#prereq-steps) + [ ## Step 1: Configure static IP addressing in your instance ](#step1) + [ ## Step 2: Configure a secondary private IP address for your instance ](#step2) + [ ## Step 3: Configure applications to Use the secondary private IP address ](#step3) ## Prerequisites + Assign the secondary private IPv4 address to the network interface for the instance. You can assign the secondary private IPv4 address when you launch the instance, or after the instance is running. For more information, see [Assign secondary IP addresses to an instance](instance-secondary-ip-addresses.md#assign-secondary-ip-address). ## Step 1: Configure static IP addressing in your instance To enable your Windows instance to use multiple IP addresses, you must configure your instance to use static IP addressing rather than a DHCP server. **Important** When you configure static IP addressing in your instance, the IP address must match exactly what is shown in the console, CLI, or API. If you enter these IP addresses incorrectly, the instance could become unreachable. **To configure static IP addressing on a Windows instance** 1. Connect to your instance. 1. Find the IP address, subnet mask, and default gateway addresses for the instance by performing the following steps: 1. Run the following command in PowerShell: ``` ipconfig /all ``` Review the output and note the **IPv4 Address**, **Subnet Mask**, **Default Gateway**, and **DNS Servers** values for the network interface. Your output should resemble the following example: ``` ... Ethernet adapter Ethernet 4: Connection-specific DNS Suffix . : us-west-2.compute.internal Description . . . . . . . . . . . : Amazon Elastic Network Adapter #2 Physical Address. . . . . . . . . : 02-9C-3B-FC-8E-67 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::f4d1:a773:5afa:cd1%7(Preferred) IPv4 Address. . . . . . . . . . . : 10.200.0.128(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Lease Obtained. . . . . . . . . . : Monday, April 8, 2024 12:19:29 PM Lease Expires . . . . . . . . . . : Monday, April 8, 2024 4:49:30 PM Default Gateway . . . . . . . . . : 10.200.0.1 DHCP Server . . . . . . . . . . . : 10.200.0.1 DHCPv6 IAID . . . . . . . . . . . : 151166011 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2D-67-AC-FC-12-34-9A-BE-A5-E7 DNS Servers . . . . . . . . . . . : 10.200.0.2 NetBIOS over Tcpip. . . . . . . . : Enabled ``` 1. Open the **Network and Sharing Center** by running the following command in PowerShell: ``` & $env:SystemRoot\system32\control.exe ncpa.cpl ``` 1. Open the context (right-click) menu for the network interface (Local Area Connection or Ethernet) and choose **Properties**. 1. Choose **Internet Protocol Version 4 (TCP/IPv4)**, **Properties**. 1. In the **Internet Protocol Version 4 (TCP/IPv4) Properties** dialog box, choose **Use the following IP address**, enter the following values, and then choose **OK**. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/config-windows-multiple-ip.html) **Important** If you set the IP address to any value other than the current IP address, you will lose connectivity to the instance. ![\[IP Addresses\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/windows-ip-static.png) You will lose RDP connectivity to the Windows instance for a few seconds while the instance converts from using DHCP to static addressing. The instance retains the same IP address information as before, but now this information is static and not managed by DHCP. ## Step 2: Configure a secondary private IP address for your instance After you have set up static IP addressing on your Windows instance, you are ready to prepare a second private IP address. **To configure a secondary IP address** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances** and select your instance. 1. On the **Networking**, note the secondary IP address. 1. Connect to your instance. 1. On your Windows instance, choose **Start**, **Control Panel**. 1. Choose **Network and Internet**, **Network and Sharing Center**. 1. Select the network interface (Local Area Connection or Ethernet) and choose **Properties**. 1. On the **Local Area Connection Properties** page, choose **Internet Protocol Version 4 (TCP/IPv4)**, **Properties**, **Advanced**. 1. Choose **Add**. 1. In the **TCP/IP Address** dialog box, type the secondary private IP address for **IP address**. For **Subnet mask**, type the same subnet mask that you entered for the primary private IP address in [Step 1: Configure static IP addressing in your instance](#step1), and then choose **Add**. ![\[TCP/IP Address dialog box\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/windows-ip-add.png) 1. Verify the IP address settings and choose **OK**. ![\[IP Settings tab\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/windows-ip-advanced-tcpip.png) 1. Choose **OK**, **Close**. 1. To confirm that the secondary IP address has been added to the operating system, run the `ipconfig /all` command in PowerShell. Your output should resemble the following: ``` Ethernet adapter Ethernet 4: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Amazon Elastic Network Adapter #2 Physical Address. . . . . . . . . : 02-9C-3B-FC-8E-67 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::f4d1:a773:5afa:cd1%7(Preferred) IPv4 Address. . . . . . . . . . . : 10.200.0.128(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 IPv4 Address. . . . . . . . . . . : 10.200.0.129(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : 10.200.0.1 DHCPv6 IAID . . . . . . . . . . . : 151166011 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2D-67-AC-FC-12-34-9A-BE-A5-E7 DNS Servers . . . . . . . . . . . : 10.200.0.2 NetBIOS over Tcpip. . . . . . . . : Enabled ``` ## Step 3: Configure applications to Use the secondary private IP address You can configure any applications to use the secondary private IP address. For example, if your instance is running a website on IIS, you can configure IIS to use the secondary private IP address. **To configure IIS to use the secondary private IP address** 1. Connect to your instance. 1. Open Internet Information Services (IIS) Manager. 1. In the **Connections** pane, expand **Sites**. 1. Open the context (right-click) menu for your website and choose **Edit Bindings**. 1. In the **Site Bindings** dialog box, for **Type**, choose **http**, **Edit**. 1. In the **Edit Site Binding** dialog box, for **IP address**, select the secondary private IP address. (By default, each website accepts HTTP requests from all IP addresses.) ![\[IP Addresses\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/windows-ip-iis-site-binding.png) 1. Choose **OK**, **Close**. # EC2 instance hostnames and domains Understanding EC2 instance hostnames and domains is important for effectively managing and accessing your Amazon EC2 instances. Each EC2 instance can have different types of hostnames - private and public - that serve different purposes and follow specific naming conventions. This topic explains the structure of EC2 instance hostnames, how they're constructed, and the different hostname types available. You'll learn how to view and modify hostname settings, understand when to use each type, and learn best practices for hostname management in your AWS environment. **Topics** + [ # Understanding EC2 instance hostnames and domains ](understanding-ec2-instance-hostnames-domains.md) + [ # Hostname types ](hostname-types.md) # Understanding EC2 instance hostnames and domains A EC2 instance address is made up of different components. The following is an example of an EC2 instance address that uses the private IPv4 address of the instance: ``` IP address Domain name ↓--------↓ ↓------------------------↓ ip-10-24-34-0.us-west-2.compute.internal ↑-----------↑ Hostname ↑--------------------------------------↑ Fully qualified domain name (FQDN) ``` Where: + **IP address**: The primary IPv4 address of the primary network interface associated with an instance. + **Hostname**: The local name of a specific EC2 instance (used by the operating system and for local network identification) + **Domain name**: The part of the FQDN that AWS provides + **Fully qualified domain name (FQDN)**: The complete address that includes both the hostname and the domain name. This is the full, globally unique identifier used to reach your instance across networks. Depending on the hostname type you choose for the instance or primary network interface attached to the instance, the hostname and domain name formats will be different from the example above. This section explains the hostname type options. # Hostname types AWS provides two types of hostnames: **private** and **public**. The following table compares the key differences between private and public hostnames, including how they resolve, how they're configured, and when to use each type. | ​ | Private hostnames | Public hostnames | | --- | --- | --- | | DNS resolution | Private hostnames enable private FQDNs that are not accessible from the public internet. Private hostnames only allow requests to resolve to private IPv4 and IPv6 GUA addresses within the VPC. | Public hostnames enable public FQDNs that are accessible from the public internet. Public hostnames enable requests to resolve to private IPv4 and IPv6 GUA within the VPC and public IPs from the internet (split-horizon DNS). | | Configuration | Private hostnames are configured at the instance level. | Public hostnames are configured at the network interface level. | | When to use | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/hostname-types.html) | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/hostname-types.html) | **Topics** + [ ## Public hostnames ](#public-hostnames) + [ ## Private hostnames ](#ec2-instance-private-hostnames) ## Public hostnames You can use *public hostnames* to access EC2 instances using hostnames that resolve to the public IPv4 or IPv6 address of the instance and ease your transition to IPv6. Public hostnames allow you to: + Access your EC2 instances over both IPv4 and IPv6, giving you more flexibility in how you connect to your resources. + Migrate from IPv4 to IPv6 environments at your own pace. You can, for example, decouple your database and application migrations, reducing complexity and risk. + Use multiple hostname options (IPv4-only, IPv6-only, and dual-stack) that automatically resolve to the appropriate IP addresses. + Benefit from improved security with [split-horizon DNS](https://en.wikipedia.org/wiki/Split-horizon_DNS), which resolves hostnames to private IP addresses when queried from within your VPC. **Topics** + [ ### Public hostname types and when to use them ](#public-hostname-types) + [ ### View public hostnames ](#view-public-hostnames) + [ ### Modify public hostnames types ](#modify-public-hostnames) ### Public hostname types and when to use them To use public hostnames, you have to modify an existing network interface. This section describes the three public hostname type options and helps you decide which to use: + **Dual-stack - IP based name** + This is the best option if you are migrating or planning to migrate from IPv4 to IPv6. This option allows for connectivity over both IPv4 and IPv6, giving flexibility to clients that may be using either protocol and enables clients to keep the same hostname throughout the migration to IPv6. + Requests from within the VPC resolve to both the private IPv4 address and the IPv6 Global Unicast Address (GUA) of the network interface. Requests from the internet resolve to both the public IPv4 and the IPv6 GUA address of the network interface. + **Example** + When you choose this option, a dual-stack FQDN will be generated for this network interface. This is an example of the FQDN that will be generated: + f5lnz-0khrm-nt2u3-gyqqt-nbdl5-q3cdpO.ap-southeast-2.ip.aws + Where: + f5lnz-0khrm-nt2u3-gyqqt-nbdl5-q3cdpO is the hostname that is a [base36](https://en.wikipedia.org/wiki/Base36) representation of the primary public IPv6 address (f5lnz-0khrm-nt2u3-gyqqt-nbdl5) on the network interface along with a base36 representation of the primary public IPv4 address (q3cdpO) on the network interface. + f5lnz-0khrm-nt2u3-gyqqt-nbdl5 is resolved by the Amazon DNS resolver to the IPv6 GUA address FFFF:1407:4:f000:81d:2689:1066:4489. This is the first IPv6 GUA assigned to the network interface. + q3cdpO is resolved to the IPv4 address of 52.54.55.56. This is the public IPv4 address attached to the primary network interface. + ap-southeast-2 is the Region of the subnet that the network interface is in. + ip.aws is the domain provided by AWS. + **IPv6 - IP based name** + This is a good option if you have already migrated to IPv6 and require connectivity only over IPv6. + Requests from within the VPC or from the internet resolve to the IPv6 GUA of the network interface. + **Example** + When you choose this option, a FQDN will be generated for this network interface. This is an example of the FQDN that will be generated: + f5lnz-0khrm-nt2u3-gyqqt-nbdl5.ap-southeast-2.ip.aws + Where: + f5lnz-0khrm-nt2u3-gyqqt-nbdl5 is a hostname that is a base36 representation of the primary public IPv6 address on the network interface. + f5lnz-0khrm-nt2u3-gyqqt-nbdl5 is resolved by the Amazon DNS resolver to the IPv6 GUA address FFFF:1407:4:f000:81d:2689:1066:4489. This is the first IPv6 GUA assigned to the network interface. + ap-southeast-2 is the Region of the subnet that the network interface is in. + ip.aws is the domain provided by AWS. + **IPv4 - IP based name** + This is a good option if the instance using this network interface need to maintain IPv4 access during the transition to IPv6 or if applications or systems running on the instance only support IPv4. This is the best option if you only need to maintain IPv4 connectivity and your workloads don't require IPv6 support. For example, if you are migrating to IPv6, you may decide to keep some applications on IPv4 while others move to IPv6. + Requests from within the VPC resolve to the private primary IPv4 address of the network interface. Requests from the internet resolve to the public IPv4 address of the network interface. + **Example** + If you choose this option, an IPv4-enabled public hostname will be generated for this network interface. This is an example of the DNS name that will be generated: + ec2-52-54-55-66.ap-southeast-2.compute.amazonaws.com + Where: + ec2-52-54-55-66 is a hostname that is a base36 representation of the primary public IPv4 address of a network interface. + ec2-52-54-55-66 resolves to the IPv4 address of 52.54.55.56. This is the public IPv4 address attached to the primary network interface. + ap-southeast-2 is the Region of the subnet that the network interface is in. + ip.aws is the domain provided by AWS. **Important** In the examples above, you can see that IP addresses are used to generate the hostname. If you change the primary private IPv4 address or the first IPv6 GUA assigned to the network interface, the portion of the hostname that translates to the IP address will change and **the previously-generated public hostname will no longer be valid**. In addition, changing the primary IPv4 public address forces a downstream refresh of [Instance Metadata Service (IMDS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html) in the *Amazon EC2 User Guide*, so the EC2 instance IMDS metadata is automatically updated. ### View public hostnames If the VPC that the network interface is in does not have both EnableDnsHostnames and EnableDnsSupport enabled, there is no hostname type defined or generated. ------ #### [ Console ] You can view the public hostnames for an instance or primary network interface. **To view the hostname type and DNS names of an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the checkbox for the instance. 1. On the **Network** tab, under **Hostname and DNS**, find the following: + **Public hostname type** + **Public DNS** + **IPv4-only IP based name** + **IPv6-only - IP based name** + **Dualstack - IP based name** **To view the hostname type and DNS names of a network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. In the search field, enter the ID of the instance. Select the ID of the network interface to open its details page. 1. Under **Hostname and DNS**, find the following: + **Public hostname type** + **Public DNS name** + **Public IPv4 DNS name** + **Public IPv6 DNS name** + **Public Dualstack DNS name** ------ #### [ AWS CLI ] **To view the hostname type and DNS names of a network interface** Use the [describe-network-interfaces](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-network-interfaces.html) command. ``` aws ec2 describe-network-interfaces \ --network-interface-id eni-1234567890abcdef0 \ --query NetworkInterfaces[].PublicIpDnsNameOptions ``` The following is example output. Because the hostname type is `public-dual-stack-dns-name`, the DNS hostname is the same as `PublicDualStackDnsName`. ``` [ { "DnsHostnameType": "public-dual-stack-dns-name", "PublicIpv4DnsName": "ec2-52-54-55-66.ap-southeast-2.compute.amazonaws.com", "PublicIpv6DnsName": "f5lnz-0khrm-nt2u3-gyqqt-nbdl5.ap-southeast-2.ip.aws", "PublicDualStackDnsName": "f5lnz-0khrm-nt2u3-gyqqt-nbdl5-q3cdpO.ap-southeast-2.ip.aws" } ] ``` ------ #### [ PowerShell ] **To view the hostname type and DNS names of a network interface** Use the [Get-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2NetworkInterface.html) cmdlet. ``` (Get-EC2NetworkInterface ` -NetworkInterfaceId eni-1234567890abcdef0).PublicIpDnsNameOptions ``` The following is example output. Because the hostname type is `public-dual-stack-dns-name`, the DNS hostname is the same as `PublicDualStackDnsName`. ``` DnsHostnameType : public-dual-stack-dns-name PublicDualStackDnsName : f5lnz-0khrm-nt2u3-gyqqt-nbdl5-q3cdpO.ap-southeast-2.ip.aws PublicIpv4DnsName : ec2-52-54-55-66.ap-southeast-2.compute.amazonaws.com PublicIpv6DnsName : f5lnz-0khrm-nt2u3-gyqqt-nbdl5.ap-southeast-2.ip.aws ``` ------ ### Modify public hostnames types The public hostname type options depend on the IP addresses associated with the network interface: + If the network interface has only a public IPv4 address, the hostname type must be **IPv4 - IP based name**. + If the network interface has only an IPv6 address, the hostname type must be **IPv6 - IP based name**. + If the network interface has both a public IPv4 address and an IPv6 address, the hostname type can be **Dual-stack - IP based name**. **Prerequisites** + The network interface must have an associated public IPv4 address or an IPv6 address. + The VPC that the network interface is in must have EnableDnsHostnames and EnableDnsSupport enabled. See [View and update DNS attributes for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns-updating.html) in the *Amazon VPC User Guide*. ------ #### [ Console ] **To modify the public hostname type** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. In the search field, enter the ID of the instance. Select the checkbox for the network interface. Alternatively, from the instance detail page, choose the **Networking** tab and select the ID of the network interface for device index 0. 1. Choose **Actions**, **Modify public hostname type**. 1. Choose an option: + **Dual-stack - IP based name**: A dual-stack public hostname for a network interface. Requests from within the VPC resolve to both the private IPv4 address and the IPv6 Global Unicast Address of the network interface. Requests from the internet resolve to both the public IPv4 and the IPv6 GUA address of the network interface. + **IPv4 - IP based name**: An IPv4-enabled public hostname for a network interface. Requests from within the VPC resolve to the private primary IPv4 address of the network interface. Requests from the internet resolve to the public IPv4 address of the network interface. + **IPv6 - IP based name**: An IPv6-enabled public hostname for a network interface. Requests from within the VPC or from the internet resolve to the IPv6 GUA of the network interface. 1. Choose **Modify**. ------ #### [ AWS CLI ] **To modify the public hostname type** Use the [ modify-public-ip-dns-name-options](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-public-ip-dns-name-options.html) command. ``` aws ec2 modify-public-ip-dns-name-options \ --network-interface-id eni-1234567890abcdef0 \ --hostname-type public-dual-stack-dns-name ``` The following is example output. ``` { "Successful": true } ``` ------ #### [ PowerShell ] **To modify the public hostname type** Use the [Edit-EC2PublicIpDnsNameOption](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2PublicIpDnsNameOption.html) cmdlet. ``` Edit-EC2PublicIpDnsNameOption ` -NetworkInterfaceId eni-1234567890abcdef0 ` -HostNameType public-dual-stack-dns-name ``` ------ ## Private hostnames Private hostnames This section describes the Amazon EC2 instance private hostnames available when you launch instances into your VPC subnets. The private hostname distinguishes the EC2 instances on your network. You may use the private hostname of an instance if, for example, you want to run scripts to communicate with some or all of the instances on your network. **Topics** + [ ### Private hostname types ](#instance-naming-types) + [ ### Where to find resource names and IP names ](#instance-naming-presence) + [ ### Choosing between resource names and IP names ](#instance-naming-choose) + [ ### Change resource based naming options for Amazon EC2 ](#instance-naming-modify) ### Private hostname types There are two private hostname types for the guest OS hostname when EC2 instances are launched in a VPC: + **IP name**: The legacy naming scheme where, when you launch an instance, the *private IPv4 address* of the instance is included in the hostname of the instance. The IP name exists for the life of the EC2 instance. When used as the Private DNS hostname, it will only return the private IPv4 address (A record). + **Resource name**: When you launch an instance, the *EC2 instance ID* is included in the hostname of the instance. The resource name exists for the life of the EC2 instance. When used as the Private DNS hostname, it can return both the private IPv4 address (A record) and/or the IPv6 Global Unicast Address (AAAA record). The EC2 instance guest OS hostname type depends on the subnet settings: + If the instance is launched into an IPv4-only subnet, you can select either IP name or resource name. + If the instance is launched into a dual-stack (IPv4\$1IPv6) subnet, you can select either IP name or resource name. + If the instance is launched into an IPv6-only subnet, resource name is used automatically. **Topics** + [ #### IP name ](#instance-naming-ipbn) + [ #### Resource name ](#instance-naming-rbn) + [ #### The difference between IP name and Resource name ](#instance-naming-diff) #### IP name When you launch an EC2 instance with the **Hostname type** of **IP name**, the guest OS hostname is configured to use the private IPv4 address. + Format for an instance in us-east-1: `private-ipv4-address.ec2.internal` + Example: `ip-10-24-34-0.ec2.internal` + Format for an instance in any other AWS Region: `private-ipv4-address.region.compute.internal` + Example: `ip-10-24-34-0.us-west-2.compute.internal` #### Resource name When you launch EC2 instances in IPv6-only subnets, the **Hostname type** of **Resource name** is selected by default. When you launch an instance in IPv4-only or dual-stack (IPv4\$1IPv6) subnets, **Resource name** is an option that you can select. After you launch an instance, you can manage the hostname configuration. For more information, see [Change resource based naming options for Amazon EC2](#instance-naming-modify). When you launch an EC2 instance with a **Hostname type** of **Resource name**, the guest OS hostname is configured to use the EC2 instance ID. + Format for an instance in us-east-1: `ec2-instance-id.ec2.internal` + Example: `i-0123456789abcdef.ec2.internal` + Format for an instance in any other AWS Region: `ec2-instance-id.region.compute.internal` + Example: `i-0123456789abcdef.us-west-2.compute.internal` #### The difference between IP name and Resource name DNS queries for both IP names and resource names coexist to ensure backward compatibility and to allow you to migrate from IP based-naming for hostnames to resource-based naming. For private DNS hostnames based on IP names, you cannot configure whether a DNS A record query for the instance is responded to or not. DNS A record queries are always responded to irrespective of the guest OS hostname settings. In contrast, for private DNS hostnames based on resource name, you can configure whether DNS A and/or DNS AAAA queries for the instance are responded to or not. You configure the response behavior when you launch an instance or modify a subnet. For more information, see [Change resource based naming options for Amazon EC2](#instance-naming-modify). ### Where to find resource names and IP names You can see the hostname types, resource name and IP name, in the Amazon EC2 console. **Topics** + [ #### When creating an EC2 instance ](#instance-naming-presence-create) + [ #### When viewing the details of an existing EC2 instance ](#instance-naming-presence-view) #### When creating an EC2 instance When you create an EC2 instance, depending on which type of subnet you select, **Hostname type** of **Resource name** might be available or it might be selected and not be modifiable. This section explains the scenarios in which you see the hostname types resource name and IP name. ##### Scenario 1 You create an EC2 instance in the wizard (see [Launch an EC2 instance using the launch instance wizard in the console](ec2-launch-instance-wizard.md)) and, when you configure the details, you choose a subnet that you configured to be IPv6-only. In this case, the **Hostname type** of **Resource name** is selected automatically and is not modifiable. **DNS Hostname** options of **Enable IP name IPv4 (A record) DNS requests** and **Enable resource-based IPv4 (A record) DNS requests** are deselected automatically and are not modifiable. **Enable resource-based IPv6 (AAAA record) DNS requests** is selected by default but is modifiable. If selected, DNS requests to the resource name will resolve to the IPv6 address (AAAA record) of this EC2 instance. ##### Scenario 2 You create an EC2 instance in the wizard (see [Launch an EC2 instance using the launch instance wizard in the console](ec2-launch-instance-wizard.md)) and, when you configure the details, you choose a subnet configured with an IPv4 CIDR block or both an IPv4 and IPv6 CIDR block ("dual stack"). In this case, **Enable IP name IPv4 (A record) DNS requests** is selected automatically and can't be changed. This means that requests to the IP name will resolve to the IPv4 address (A record) of this EC2 instance. The options default to the configurations of the subnet, but you can modify the options for this instance depending on the subnet settings: + **Hostname type**: Determines whether you want the guest OS hostname of the EC2 instance to be the resource name or IP name. The default value is **IP name**. + **Enable resource-based IPv4 (A record) DNS requests**: Determines whether requests to your resource name resolve to the private IPv4 address (A record) of this EC2 instance. This option is not selected by default. + **Enable resource-based IPv6 (AAAA record) DNS requests**: Determines whether requests to your resource name resolve to the IPv6 GUA address (AAAA record) of this EC2 instance. This option is not selected by default. #### When viewing the details of an existing EC2 instance You can see the hostname values for an existing EC2 instance in the **Details** tab for the EC2 instance: + **Hostname type**: The hostname in IP name or resource name format. + **Private IP DNS name (IPv4 only)**: The IP name that will always resolve to the private IPv4 address of the instance. + **Private resource DNS name**: The resource name that resolves to the DNS records selected for this instance. + **Answer private resource DNS name**: The resource name resolves to IPv4 (A), IPv6 (AAAA) or IPv4 and IPv6 (A and AAAA) DNS records. In addition, if you connect to your EC2 instance directly over SSH and enter the `hostname` command, you'll see the hostname in either the IP name or resource name format. ### Choosing between resource names and IP names When you launch an EC2 instance (see [Launch an EC2 instance using the launch instance wizard in the console](ec2-launch-instance-wizard.md)), if you choose a **Hostname type** of **Resource name**, the EC2 instance launches with a hostname in the resource name format. In such cases, the DNS record for this EC2 instance can also point to the resource name. This gives you the flexibility to choose whether that hostname resolves to the IPv4 address, the IPv6 address, or both the IPv4 and IPv6 address of the instance. If you plan to use IPv6 in the future or if you are using dual-stack subnets today, it’s best to use a **Hostname type** of **Resource name** so that you change DNS resolution for the hostnames of your instances without making any changes to the DNS records themselves. The resource name allows you to add and remove IPv4 and IPv6 DNS resolution on an EC2 instance. If instead you choose a **Hostname type** of **IP name**, and use it as the DNS hostname, it can only resolve to the IPv4 address of the instance. It will not resolve to the IPv6 address of the instance even if the instance has both an IPv4 address and an IPv6 address associated with it. ### Change resource based naming options for Amazon EC2 Change resource based naming options You can change the hostname type and DNS hostname configurations for subnets, which affects all subsequent instance launches in that subject, or you can change them for an EC2 instances after you launch it. **Resource based naming options** + **Hostname type**: Determines the default setting for the guest OS hostname of EC2 instances launched in the subnet. This is either the resource name or IP name. + **Enable DNS hostname IPv4 (A record) requests**: Determines whether DNS requests/queries to the resource name resolve to the private IPv4 address (A record) of the EC2 instance. + **Enable DNS hostname IPv6 (AAAA record) requests**: Determines whether DNS requests/queries to the resource name resolve to the IPv6 address (AAAA record) of the EC2 instance. #### Subnets Changing the subnet settings doesn't change the configuration of EC2 instances that are already launched in the subnet. ------ #### [ Console ] **To modify the options for a subnet** Open the Amazon VPC console and select the subnet. Choose **Actions**, **Edit subnet settings**. Modify the settings as needed and then save your changes. ------ #### [ AWS CLI ] **To modify the options for a subnet** Use the [modify-subnet-attribute](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-subnet-attribute.html) command. ``` aws ec2 modify-subnet-attribute \ --subnet-id subnet-0abcdef1234567890 \ --private-dns-hostname-type-on-launch resource-name \ --enable-resource-name-dns-a-record-on-launch \ --enable-resource-name-dns-aaaa-record-on-launch ``` ------ #### [ PowerShell ] **To modify the options for a subnet** Use the [Edit-EC2SubnetAttribute](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2SubnetAttribute.html) cmdlet. ``` Edit-EC2SubnetAttribute ` -SubnetId subnet-0abcdef1234567890 ` -PrivateDnsHostnameTypeOnLaunch ResourceName ` -EnableResourceNameDnsAAAARecordOnLaunch $true ` -EnableResourceNameDnsARecordOnLaunch $true ``` ------ #### EC2 instances **Considerations** + To change the hostname type, you must first stop the instance. There is no need to stop an instance to change the other two options. + Because you can't stop an instance with an instance store root volume, you can only configure the hostname type and DNS hostname options at instance launch. Only the following instance types support instance store root volumes: C1, C3, D2, I2, M1, M2, M3, R3, and X1. ------ #### [ Console ] **To modify the hostname type and DNS hostname options for an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. If you're going to change the **Use resource based naming as guest OS hostname** setting, first stop the EC2 instance. Otherwise, skip this step. To stop the instance, select the instance and choose **Instance state**, **Stop instance**. 1. Select the instance and choose **Actions**, **Instance settings**, **Change resource based naming options**. + **Use resource based naming as guest OS hostname**: Determines whether you want the guest OS hostname of the EC2 instance to be the resource name or IP name. + **Answer DNS hostname IPv4 (A record) requests**: Determines whether DNS requests/queries to your resource name resolve to the private IPv4 address of this EC2 instance. + **Answer DNS hostname IPv6 (AAAA record) requests**: Determines whether DNS requests/queries to your resource name resolve to the IPv6 address (AAAA record) of this EC2 instance. 1. Choose **Save**. 1. If you stopped the instance, start it again. ------ #### [ AWS CLI ] **To modify the hostname type and DNS hostname options for an instance** Use the [modify-private-dns-name-options](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-private-dns-name-options.html) command. ``` aws ec2 modify-private-dns-name-options \ --instance-id i-1234567890abcdef0 \ --private-dns-hostname-type resource-name \ --enable-resource-name-dns-a-record \ --enable-resource-name-dns-aaaa-record ``` ------ #### [ PowerShell ] **To modify the hostname type and DNS hostname options for an instance** Use the [Edit-EC2PrivateDnsNameOption](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2PrivateDnsNameOption.html) cmdlet. ``` Edit-EC2PrivateDnsNameOption ` -InstanceId i-1234567890abcdef0 ` -PrivateDnsHostnameType ResourceName ` -EnableResourceNameDnsAAAARecord $true` -EnableResourceNameDnsARecord $true ``` ------ # Bring your own IP addresses (BYOIP) to Amazon EC2 Bring your own IP addresses You can bring part or all of your publicly routable IPv4 or IPv6 address range from your on-premises network to your AWS account. You continue to control the address range and you can advertise the address range on the internet through AWS. After you bring the address range to Amazon EC2, it appears in your AWS account as an address pool. **Note** This documentation describes how to bring your own IP address range for use in Amazon EC2 only. To bring your own IP address range for use in AWS Global Accelerator, see [Bring your own IP addresses (BYOIP)](https://docs.aws.amazon.com/global-accelerator/latest/dg/using-byoip.html) in the *AWS Global Accelerator Developer Guide*. To bring your own IP address range for use with Amazon VPC IP Address Manager, see [Tutorial: Bring your IP addresses to IPAM](https://docs.aws.amazon.com/vpc/latest/ipam/tutorials-byoip-ipam.html) in the *Amazon VPC IPAM User Guide*. When you bring an IP address range to AWS, AWS validates that you control the IP address range. There are two methods that you can use to show that you control the range: + If your IP address range is registered with an Internet Registry that supports RDAP (such as ARIN, RIPE and APNIC), you can verify control of your domain with an X.509 certificate by using the process on this page. The certificate must only be valid for the duration of the provisioning process. You can remove the certificate from your RIR's record after provisioning is complete. + Regardless of whether your Internet Registry supports RDAP, you can use Amazon VPC IPAM to verify control of your domain with a DNS TXT record. That process is documented in [Tutorial: Bring your IP addresses to IPAM](https://docs.aws.amazon.com/vpc/latest/ipam/tutorials-byoip-ipam.html) in the *Amazon VPC IPAM User guide*. For more information, see the AWS Online Tech talk [Deep Dive on Bring Your Own IP](https://pages.awscloud.com/Deep-Dive-on-Bring-Your-Own-IP_1024-NET_OD.html). **Topics** + [ ## BYOIP definitions ](#byoip-definitions) + [ ## Requirements and quotas ](#byoip-requirements) + [ ## Regional availability ](#byoip-reg-avail) + [ ## Local Zone availability ](#byoip-zone-avail) + [Prerequisites](prepare-for-byoip.md) + [Onboard your address range](byoip-onboard.md) + [Use your address range](byoip-working-with.md) ## BYOIP definitions + **X.509 Self-sign certificate** — A certificate standard most commonly used to encrypt and authenticate data within a network. It is a certificate used by AWS to validate control over IP space from an RDAP record. For more information about X.509 certificates, see [RFC 3280](https://datatracker.ietf.org/doc/html/rfc3280). + **Autonomous System Number (ASN)** – A globally unique identifier that defines a group of IP prefixes run by one or more network operators that maintain a single, clearly-defined routing policy. + **Regional Internet Registry (RIR)** – An organization that manages allocation and registration of IP addresses and ASNs within a region of the world. + **Registry Data Access Protocol (RDAP)** — A read-only protocol to query current registration data within a RIR. Entries within the queried RIR database are referred to as "RDAP records". Certain record types need to be updated by customers via a RIR-provided mechanism. These records are queried by AWS to verify control of an address space in the RIR. + **Route Origin Authorization (ROA)** — An object created by RIRs for customers to authenticate IP advertisement in particular autonomous systems. For an overview, see [Route Origin Authorizations (ROAs)](https://www.arin.net/resources/manage/rpki/roa_request/) on the ARIN website. + **Local Internet Registry (LIR)** — Organizations such as internet service providers that allocate a block of IP addresses from an RIR for their customers. ## Requirements and quotas + The address range must be registered with your Regional Internet Registry (RIR). See your RIR for any policies regarding geographic regions. BYOIP currently supports registration in the American Registry for Internet Numbers (ARIN), Réseaux IP Européens Network Coordination Centre (RIPE), or Asia-Pacific Network Information Centre (APNIC). It must be registered to a business or institutional entity and cannot be registered to an individual person. + The most specific IPv4 address range that you can bring is /24. + The most specific IPv6 address range that you can bring is /48 for CIDRs that are publicly advertisable and /60 for CIDRs that are [not publicly advertisable](byoip-onboard.md#byoip-provision-non-public). + ROAs are not required for CIDR ranges that are not publicly advertisable, but the RDAP records still need to be updated. + You can bring each address range to one AWS Region at a time. + You can bring a total of five BYOIP IPv4 and IPv6 address ranges per AWS Region to your AWS account. You cannot adjust the quotas for BYOIP CIDRs using the Service Quotas console, but you can request a quota increase by contacting the AWS Support Center as described in [AWS service quotas](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) in the *AWS General Reference*. + You cannot share your IP address range with other accounts using AWS RAM unless you use Amazon VPC IP Address Manager (IPAM) and integrate IPAM with AWS Organizations. For more information, see [Integrate IPAM with AWS Organizations](https://docs.aws.amazon.com/vpc/latest/ipam/enable-integ-ipam.html) in the *Amazon VPC IPAM User Guide*. + The addresses in the IP address range must have a clean history. We might investigate the reputation of the IP address range and reserve the right to reject an IP address range if it contains an IP address that has a poor reputation or is associated with malicious behavior. + Legacy address space, the IPv4 address space that was distributed by the Internet Assigned Numbers Authority's (IANA) central registry prior to the formation of the Regional Internet Registry (RIR) system, still requires a corresponding ROA object. + For LIRs, it is common that they use a manual process to update their records. This can take days to deploy depending on the LIR. + A single ROA object and RDAP record are needed for a large CIDR block. You can bring multiple smaller CIDR blocks from that range to AWS, even across multiple AWS Regions, using the single object and record. + BYOIP is not supported for Wavelength Zones or on AWS Outposts. + Do not make any manual changes for BYOIP in RADb or any other IRR. BYOIP will automatically update RADb. Any manual changes that include the BYOIP ASN will cause the BYOIP provision operation to fail. + Once you bring an IPv4 address range to AWS, you can use all of the IP addresses in the range, including the first address (the network address) and the last address (the broadcast address). ## Regional availability The BYOIP feature is currently available in all commercial [AWS Regions](https://aws.amazon.com//about-aws/global-infrastructure/regions_az/) except for China Regions. ## Local Zone availability A [Local Zone](https://docs.aws.amazon.com/local-zones/latest/ug/how-local-zones-work.html) is an extension of an AWS Region in geographic proximity to your users. Local Zones are grouped into "network border groups". In AWS, a network border group is a collection of Availability Zones (AZs), Local Zones, or Wavelength Zones from which AWS advertises a public IP address. Local Zones may have different network border groups than the AZs in an AWS Region to ensure minimum latency or physical distance between the AWS network and the customers accessing the resources in these Zones. You can provision BYOIPv4 address ranges to and advertise them in the following Local Zone network border groups using the `--network-border-group` option: + af-south-1-los-1 + ap-northeast-1-tpe-1 + ap-south-1-ccu-1 + ap-south-1-del-1 + ap-southeast-1-bkk-1 + ap-southeast-1-mnl-1 + ap-southeast-2-akl-1 + ap-southeast-2-per-1 + eu-central-1-ham-1 + eu-central-1-waw-1 + eu-north-1-cph-1 + eu-north-1-hel-1 + me-south-1-mct-1 + us-east-1-atl-2 + us-east-1-bos-1 + us-east-1-bue-1 + us-east-1-chi-2 + us-east-1-dfw-2 + us-east-1-iah-2 + us-east-1-lim-1 + us-east-1-mci-1 + us-east-1-mia-2 + us-east-1-msp-1 + us-east-1-nyc-1 + us-east-1-nyc-2 + us-east-1-phl-1 + us-east-1-qro-1 + us-east-1-scl-1 + us-west-2-den-1 + us-west-2-hnl-1 + us-west-2-las-1 + us-west-2-lax-1 + us-west-2-pdx-1 + us-west-2-phx-2 + us-west-2-sea-1 If you have Local Zones enabled (see [Enable a Local Zone](https://docs.aws.amazon.com/local-zones/latest/ug/getting-started.html#getting-started-find-local-zone)), you can choose a network border group for Local Zones when you provision and advertise a BYOIPv4 CIDR. Choose the network border group carefully as the EIP and the AWS resource it is associated with must reside in the same network border group. **Note** You cannot provision or advertise BYOIPv6 address ranges in Local Zones at this time. # Prerequisites for BYOIP in Amazon EC2 Prerequisites The onboarding process for BYOIP has two phases, for which you must perform three steps. These steps correspond to the steps depicted in the following diagram. We include manual steps in this documentation, but your RIR might offer managed services to help you with these steps. **Tip** The tasks in this section require a Linux terminal and may be performed using Linux, the [AWS CloudShell](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html), or the [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/about). **Topics** + [ ## Overview ](#byoip-onboarding-overview) + [ ## Create a private key and generate an X.509 certificate ](#byoip-certificate) + [ ## Upload the X.509 certificate to the RDAP record in your RIR ](#byoip-add-certificate) + [ ## Create a ROA object in your RIR ](#byoip-create-roa-object) ## Overview **Preparation phase** [1] [Create a private key](#byoip-certificate) and use it to generate a self-signed X.509 certificate for authentication purposes. This certificate is only used during the provisioning phase. You can remove the certificate from your RIR's record after provisioning is complete **RIR configuration phase** [2] [Upload the self-signed certificate](#byoip-add-certificate) to your RDAP record comments. [3] [Create a ROA object](#byoip-create-roa-object) in your RIR. The ROA defines the desired address range, the Autonomous System Numbers (ASNs) allowed to advertise the address range, and an expiration date to register with the Resource Public Key Infrastructure (RPKI) of your RIR. **Note** A ROA is not required for non-publicly advertisable IPv6 address space. ![\[The 3-step onboarding process for BYOIP.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/byoip-preonboarding.png) To bring on multiple non-contiguous address ranges, you must repeat this process with each address range. However, the preparation and RIR configuration steps don't need to be repeated if splitting a contiguous block across several different AWS Regions. Bringing on an address range has no effect on any address ranges that you brought on previously. ## Create a private key and generate an X.509 certificate Use the following procedure to create a self-signed X.509 certificate and add it to the RDAP record for your RIR. This key pair is used to authenticate the address range with the RIR. The **openssl** commands require OpenSSL version 1.0.2 or later. Copy the following commands and replace only the placeholder values (in colored italic text). This procedure follows the best practice of encrypting your private RSA key and requiring a passphrase to access it. 1. Generate an RSA 2048-bit private key as shown in the following. ``` $ openssl genpkey -aes256 -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out private-key.pem ``` The `-aes256` parameter specifies the algorithm used to encrypt the private key. The command returns the following output, including prompts to set a passphrase: ``` ......+++ .+++ Enter PEM pass phrase: xxxxxxx Verifying - Enter PEM pass phrase: xxxxxxx ``` You can inspect the key using the following command: ``` $ openssl pkey -in private-key.pem -text ``` This returns a passphrase prompt and the contents of the key, which should be similar to the following: ``` Enter pass phrase for private-key.pem: xxxxxxx -----BEGIN PRIVATE KEY----- MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDFBXHRI4HVKAhh 3seiciooizCRTbJe1+YsxNTja4XyKypVGIFWDGhZs44FCHlPOOSVJ+NqP74w96oM 7DPS3xo9kaQyZBFn2YEp2EBq5vf307KHNRmZZUmkn0zHOSEpNmY2fMxISBxewlxR FAniwmSd/8TDvHJMY9FvAIvWuTsv5l0tJKk+a91K4+tO3UdDR7Sno5WXExfsBrW3 g1ydo3TBsx8i5/YiVOcNApy7ge2/FiwY3aCXJB6r6nuF6H8mRgI4r4vkMRsOlAhJ DnZPNeweboo+K3Q3lwbgbmOKD/z9svk8N/+hUTBtIX0fRtbG+PLIw3xWRHGrMSn2 BzsPVuDLAgMBAAECggEACiJUj2hfJkKv47Dc3es3Zex67A5uDVjXmxfox2Xhdupn fAcNqAptV6fXt0SPUNbhUxbBKNbshoJGufFwXPli1SXnpzvkdU4Hyco4zgbhXFsE RNYjYfOGzTPwdBLpNMB6k3Tp4RHse6dNrlH0jDhpioL8cQEBdBJyVF5X0wymEbmV mC0jgH/MxsBAPWW6ZKicg9ULMlWiAZ3MRAZPjHHgpYkAAsUWKAbCBwVQcVjGO59W jfZjzTX5pQtVVH68ruciH88DTZCwjCkjBhxg+OIkJBLE5wkh82jIHSivZ63flwLw z+E0+HhELSZJrn2MY6Jxmik3qNNUOF/Z+3msdj2luQKBgQDjwlC/3jxp8zJy6P8o JQKv7TdvMwUj4VSWOHZBHLv4evJaaia0uQjIo1UDa8AYitqhX1NmCCehGH8yuXj/ v6V3CzMKDkmRr1NrONnSz5QsndQ04Z6ihAQlPmJ96g4wKtgoC7AYpyP0g1a+4/sj b1+o3YQI4pD/F71c+qaztH7PRwKBgQDdc23yNmT3+Jyptf0fKjEvONK+xwUKzi9c L/OzBq5yOIC1Pz2T85gOe1i8kwZws+xlpG6uBT6lmIJELd0k59FyupNu4dPvX5SD 6GGqdx4jk9KvI74usGeOBohmF0phTHkrWKBxXiyT0oS8zjnJlEn8ysIpGgO28jjr LpaHNZ/MXQKBgQDfLNcnS0LzpsS2aK0tzyZU8SMyqVHOGMxj7quhneBq2T6FbiLD T9TVlYaGNZ0j71vQaLI19qOubWymbautH0Op5KV8owdf4+bf1/NJaPIOzhDUSIjD Qo01WW31Z9XDSRhKFTnWzmCjBdeIcajyzf10YKsycaAW9lItu8aBrMndnQKBgQDb nNp/JyRwqjOrNljk7DHEs+SD39kHQzzCfqd+dnTPv2sc06+cpym3yulQcbokULpy fmRo3bin/pvJQ3aZX/Bdh9woTXqhXDdrrSwWInVYMQPyPk8f/D9mIOJp5FUWMwHD U+whIZSxsEeE+jtixlWtheKRYkQmzQZXbWdIhYyI3QKBgD+F/6wcZ85QW8nAUykA 3WrSIx/3cwDGdm4NRGct8ZOZjTHjiy9ojMOD1L7iMhRQ/3k3hUsin5LDMp/ryWGG x4uIaLat40kiC7T4I66DM7P59euqdz3w0PD+VU+h7GSivvsFDdySUt7bNK0AUVLh dMJfWxDN8QV0b5p3WuWH1U8B -----END PRIVATE KEY----- Private-Key: (2048 bit) modulus: 00:c5:05:71:d1:23:81:d5:28:08:61:de:c7:a2:72: 2a:28:8b:30:91:4d:b2:5e:d7:e6:2c:c4:d4:e3:6b: 85:f2:2b:2a:55:18:81:56:0c:68:59:b3:8e:05:08: 79:4f:38:e4:95:27:e3:6a:3f:be:30:f7:aa:0c:ec: 33:d2:df:1a:3d:91:a4:32:64:11:67:d9:81:29:d8: 40:6a:e6:f7:f7:d3:b2:87:35:19:99:65:49:a4:9f: 4c:c7:39:21:29:36:66:36:7c:cc:48:48:1c:5e:c2: 5c:51:14:09:e2:c2:64:9d:ff:c4:c3:bc:72:4c:63: d1:6f:00:8b:d6:b9:3b:2f:e6:5d:2d:24:a9:3e:6b: dd:4a:e3:eb:4e:dd:47:43:47:b4:a7:a3:95:97:13: 17:ec:06:b5:b7:83:5c:9d:a3:74:c1:b3:1f:22:e7: f6:22:54:e7:0d:02:9c:bb:81:ed:bf:16:2c:18:dd: a0:97:24:1e:ab:ea:7b:85:e8:7f:26:46:02:38:af: 8b:e4:31:1b:0e:94:08:49:0e:76:4f:35:ec:1e:6e: 8a:3e:2b:74:37:97:06:e0:6e:63:8a:0f:fc:fd:b2: f9:3c:37:ff:a1:51:30:6d:21:7d:1f:46:d6:c6:f8: f2:c8:c3:7c:56:44:71:ab:31:29:f6:07:3b:0f:56: e0:cb publicExponent: 65537 (0x10001) privateExponent: 0a:22:54:8f:68:5f:26:42:af:e3:b0:dc:dd:eb:37: 65:ec:7a:ec:0e:6e:0d:58:d7:9b:17:e8:c7:65:e1: 76:ea:67:7c:07:0d:a8:0a:6d:57:a7:d7:b7:44:8f: 50:d6:e1:53:16:c1:28:d6:ec:86:82:46:b9:f1:70: 5c:f9:62:d5:25:e7:a7:3b:e4:75:4e:07:c9:ca:38: ce:06:e1:5c:5b:04:44:d6:23:61:f3:86:cd:33:f0: 74:12:e9:34:c0:7a:93:74:e9:e1:11:ec:7b:a7:4d: ae:51:f4:8c:38:69:8a:82:fc:71:01:01:74:12:72: 54:5e:57:d3:0c:a6:11:b9:95:98:2d:23:80:7f:cc: c6:c0:40:3d:65:ba:64:a8:9c:83:d5:0b:32:55:a2: 01:9d:cc:44:06:4f:8c:71:e0:a5:89:00:02:c5:16: 28:06:c2:07:05:50:71:58:c6:3b:9f:56:8d:f6:63: cd:35:f9:a5:0b:55:54:7e:bc:ae:e7:22:1f:cf:03: 4d:90:b0:8c:29:23:06:1c:60:f8:e2:24:24:12:c4: e7:09:21:f3:68:c8:1d:28:af:67:ad:df:97:02:f0: cf:e1:34:f8:78:44:2d:26:49:ae:7d:8c:63:a2:71: 9a:29:37:a8:d3:54:38:5f:d9:fb:79:ac:76:3d:a5: b9 prime1: 00:e3:c2:50:bf:de:3c:69:f3:32:72:e8:ff:28:25: 02:af:ed:37:6f:33:05:23:e1:54:96:38:76:41:1c: bb:f8:7a:f2:5a:6a:26:b4:b9:08:c8:a3:55:03:6b: c0:18:8a:da:a1:5f:53:66:08:27:a1:18:7f:32:b9: 78:ff:bf:a5:77:0b:33:0a:0e:49:91:af:53:6b:38: d9:d2:cf:94:2c:9d:d4:34:e1:9e:a2:84:04:25:3e: 62:7d:ea:0e:30:2a:d8:28:0b:b0:18:a7:23:f4:83: 56:be:e3:fb:23:6f:5f:a8:dd:84:08:e2:90:ff:17: bd:5c:fa:a6:b3:b4:7e:cf:47 prime2: 00:dd:73:6d:f2:36:64:f7:f8:9c:a9:b5:fd:1f:2a: 31:2f:38:d2:be:c7:05:0a:ce:2f:5c:2f:f3:b3:06: ae:72:38:80:b5:3f:3d:93:f3:98:0e:7b:58:bc:93: 06:70:b3:ec:65:a4:6e:ae:05:3e:a5:98:82:44:2d: dd:24:e7:d1:72:ba:93:6e:e1:d3:ef:5f:94:83:e8: 61:aa:77:1e:23:93:d2:af:23:be:2e:b0:67:8e:06: 88:66:17:4a:61:4c:79:2b:58:a0:71:5e:2c:93:d2: 84:bc:ce:39:c9:94:49:fc:ca:c2:29:1a:03:b6:f2: 38:eb:2e:96:87:35:9f:cc:5d exponent1: 00:df:2c:d7:27:4b:42:f3:a6:c4:b6:68:ad:2d:cf: 26:54:f1:23:32:a9:51:ce:18:cc:63:ee:ab:a1:9d: e0:6a:d9:3e:85:6e:22:c3:4f:d4:d5:95:86:86:35: 9d:23:ef:5b:d0:68:b2:35:f6:a3:ae:6d:6c:a6:6d: ab:ad:1f:43:a9:e4:a5:7c:a3:07:5f:e3:e6:df:d7: f3:49:68:f2:0e:ce:10:d4:48:88:c3:42:8d:35:59: 6d:f5:67:d5:c3:49:18:4a:15:39:d6:ce:60:a3:05: d7:88:71:a8:f2:cd:fd:74:60:ab:32:71:a0:16:f6: 52:2d:bb:c6:81:ac:c9:dd:9d exponent2: 00:db:9c:da:7f:27:24:70:aa:33:ab:36:58:e4:ec: 31:c4:b3:e4:83:df:d9:07:43:3c:c2:7e:a7:7e:76: 74:cf:bf:6b:1c:d3:af:9c:a7:29:b7:ca:e9:50:71: ba:24:50:ba:72:7e:64:68:dd:b8:a7:fe:9b:c9:43: 76:99:5f:f0:5d:87:dc:28:4d:7a:a1:5c:37:6b:ad: 2c:16:22:75:58:31:03:f2:3e:4f:1f:fc:3f:66:20: e2:69:e4:55:16:33:01:c3:53:ec:21:21:94:b1:b0: 47:84:fa:3b:62:c6:55:ad:85:e2:91:62:44:26:cd: 06:57:6d:67:48:85:8c:88:dd coefficient: 3f:85:ff:ac:1c:67:ce:50:5b:c9:c0:53:29:00:dd: 6a:d2:23:1f:f7:73:00:c6:76:6e:0d:44:67:2d:f1: 93:99:8d:31:e3:8b:2f:68:8c:c3:83:d4:be:e2:32: 14:50:ff:79:37:85:4b:22:9f:92:c3:32:9f:eb:c9: 61:86:c7:8b:88:68:b6:ad:e3:49:22:0b:b4:f8:23: ae:83:33:b3:f9:f5:eb:aa:77:3d:f0:d0:f0:fe:55: 4f:a1:ec:64:a2:be:fb:05:0d:dc:92:52:de:db:34: ad:00:51:52:e1:74:c2:5f:5b:10:cd:f1:05:74:6f: 9a:77:5a:e5:87:d5:4f:01 ``` Keep your private key in a secure location when it is not in use. 1. Generate an X.509 certificate using the private key created in the previous step. In this example, the certificate expires in 365 days, after which time it cannot be trusted. Be sure to set the expiration appropriately. The certificate must only be valid for the duration of the provisioning process. You can remove the certificate from your RIR's record after provisioning is complete. The `tr -d "\n"` command strips newline characters (line breaks) from the output. You need to provide a Common Name when prompted, but the other fields can be left blank. ``` $ openssl req -new -x509 -key private-key.pem -days 365 | tr -d "\n" > certificate.pem ``` This results in output similar to the following: ``` Enter pass phrase for private-key.pem: xxxxxxx You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) []: State or Province Name (full name) []: Locality Name (eg, city) []: Organization Name (eg, company) []: Organizational Unit Name (eg, section) []: Common Name (eg, fully qualified host name) []:example.com Email Address []: ``` **Note** The Common Name is not needed for AWS provisioning. It can be any internal or public domain name. You can inspect the certificate with the following command: ``` $ cat certificate.pem ``` The output should be a long, PEM-encoded string without line breaks, prefaced by `-----BEGIN CERTIFICATE-----` and followed by `-----END CERTIFICATE-----`. ## Upload the X.509 certificate to the RDAP record in your RIR Add the certificate that you previously created to the RDAP record for your RIR. Be sure to include the `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` strings before and after the encoded portion. All of this content must be on a single, long line. The procedure for updating RDAP depends on your RIR: + For ARIN, use the [Account Manager portal](https://account.arin.net/public/secure/dashboard) to add the certificate in the "Public Comments" section for the "Network Information" object representing your address range. Do not add it to the comments section for your organization. + For RIPE, add the certificate as a new "descr" field to the "inetnum" or "inet6num" object representing your address range. These can usually be found in the "My Resources" section of the [RIPE Database portal](https://apps.db.ripe.net/db-web-ui/myresources/overview). Do not add it to the comments section for your organization or the "remarks" field of the above objects. + For APNIC, add the certificate by editing remarks on the "inetnum" or "inet6num" record. You can remove the certificate from your RIR's record after the provisioning stage below has been completed. ## Create a ROA object in your RIR Create a ROA object to authorize the Amazon ASNs 16509 and 14618 to advertise your address range, as well as the ASNs that are currently authorized to advertise the address range. For the AWS GovCloud (US) Regions, authorize ASN 8987 instead of 16509 and 14618. You must set the maximum length to the size of the CIDR that you are bringing in. The most specific IPv4 prefix you can bring is /24. The most specific IPv6 address range that you can bring is /48 for CIDRs that are publicly advertisable and /60 for CIDRs that are not publicly advertisable. **Important** If you are creating a ROA object for Amazon VPC IP Address Manager (IPAM), when you create the ROAs, for IPv4 CIDRs you must set the maximum length of an IP address prefix to `/24`. For IPv6 CIDRs, if you are adding them to an advertisable pool, the maximum length of an IP address prefix must be `/48`. This ensures that you have full flexibility to divide your public IP address across AWS Regions. IPAM enforces the maximum length you set. For more information about BYOIP addresses to IPAM, see [Tutorial: BYOIP address CIDRs to IPAM](https://docs.aws.amazon.com/vpc/latest/ipam/tutorials-byoip-ipam.html) in the *Amazon VPC IPAM User Guide*. It might take up to 24 hours for the ROA to become available to Amazon. For more information, consult your RIR: + ARIN — [ROA Requests](https://www.arin.net/resources/rpki/roarequest.html) + RIPE — [Managing ROAs](https://www.ripe.net/manage-ips-and-asns/resource-management/rpki/resource-certification-roa-management/) + APNIC — [Route Management](https://www.apnic.net/wp-content/uploads/2017/01/route-roa-management-guide.pdf) When you migrate advertisements from an on-premises workload to AWS, you must create a ROA for your existing ASN before creating the ROAs for Amazon's ASNs. Otherwise, you might see an impact to your existing routing and advertisements. **Important** For Amazon to advertise and continue advertising your IP address range, your ROAs with Amazon ASNs must be compliant with the above guidelines. If your ROAs are invalid or not compliant with the above guidelines, Amazon reserves the right to stop advertising your IP address range. **Note** This step is not required for non-publicly advertisable IPv6 address space. # Onboard your address range for use in Amazon EC2 Onboard your address range The onboarding process for BYOIP includes the following tasks, depending on your needs. **Topics** + [ ## Provision a publicly advertisable address range in AWS ](#byoip-provision) + [ ## Provision an IPv6 address range that's not publicly advertisable ](#byoip-provision-non-public) + [ ## Advertise the address range through AWS ](#byoip-advertise) + [ ## Deprovision the address range ](#byoip-deprovision) + [ ## Validate your BYOIP ](#byoip-validation) ## Provision a publicly advertisable address range in AWS When you provision an address range for use with AWS, you are confirming that you control the address range and are authorizing Amazon to advertise it. We also verify that you control the address range through a signed authorization message. This message is signed with the self-signed X.509 key pair that you used when updating the RDAP record with the X.509 certificate. AWS requires a cryptographically signed authorization message that it presents to the RIR. The RIR authenticates the signature against the certificate that you added to RDAP, and checks the authorization details against the ROA. **To provision the address range** 1. **Compose message** Compose the plaintext authorization message. The format of the message is as follows, where the date is the expiry date of the message: ``` 1|aws|account|cidr|YYYYMMDD|SHA256|RSAPSS ``` Replace the account number, address range, and expiry date with your own values to create a message resembling the following: ``` text_message="1|aws|0123456789AB|198.51.100.0/24|20211231|SHA256|RSAPSS" ``` This is not to be confused with a ROA message, which has a similar appearance. 1. **Sign message** Sign the plaintext message using the private key that you created previously. The signature returned by this command is a long string that you need to use in the next step. **Important** We recommend that you copy and paste this command. Except for the message content, do not modify or replace any of the values. ``` signed_message=$( echo -n $text_message | openssl dgst -sha256 -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-1 -sign private-key.pem -keyform PEM | openssl base64 | tr -- '+=/' '-_~' | tr -d "\n") ``` 1. **Provision address** Use the AWS CLI [provision-byoip-cidr](https://docs.aws.amazon.com/cli/latest/reference/ec2/provision-byoip-cidr.html) command to provision the address range. The `--cidr-authorization-context` option uses the message and signature strings that you created previously. **Important** You must specify the AWS Region where the BYOIP range should be provisioned if it differs from your [Configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) `Default region name`. ``` aws ec2 provision-byoip-cidr --cidr address-range --cidr-authorization-context Message="$text_message",Signature="$signed_message" --region us-east-1 ``` Provisioning an address range is an asynchronous operation, so the call returns immediately, but the address range is not ready to use until its status changes from `pending-provision` to `provisioned`. 1. **Monitor progress** While most provisioning will be completed within two hours, it might take up to one week to complete the provisioning process for publicly advertisable ranges. Use the [describe-byoip-cidrs](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-byoip-cidrs.html) command to monitor progress, as in this example: ``` aws ec2 describe-byoip-cidrs --max-results 5 --region us-east-1 ``` If there are issues during provisioning and the status goes to `failed-provision`, you must run the `provision-byoip-cidr` command again after the issues have been resolved. ## Provision an IPv6 address range that's not publicly advertisable By default, an address range is provisioned to be publicly advertisable to the internet. You can provision an IPv6 address range that will not be publicly advertisable. For routes that are not publicly advertisable, the provisioning process generally completes within minutes. When you associate an IPv6 CIDR block from a non-public address range with a VPC, the IPv6 CIDR can only be accessed through hybrid connectivity options that support IPv6, such as [Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html), [AWS Site-to-Site VPN](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html), or [Amazon VPC Transit Gateways](https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html). A ROA is not required to provision a non-public address range. **Important** You can only specify whether an address range is publicly advertisable during provisioning. You cannot change the advertisable status later on. Amazon VPC doesn't support [unique local address](https://en.wikipedia.org/wiki/Unique_local_address) (ULA) CIDRs. All VPCs must have unique IPv6 CIDRs. Two VPCs can’t have the same IPv6 CIDR range. To provision an IPv6 address range that will not be publicly advertisable, use the following [provision-byoip-cidr](https://docs.aws.amazon.com/cli/latest/reference/ec2/provision-byoip-cidr.html) command. ``` aws ec2 provision-byoip-cidr --cidr address-range --cidr-authorization-context Message="$text_message",Signature="$signed_message" --no-publicly-advertisable --region us-east-1 ``` ## Advertise the address range through AWS After the address range is provisioned, it is ready to be advertised. You must advertise the exact address range that you provisioned. You can't advertise only a portion of the provisioned address range. If you provisioned an IPv6 address range that will not be publicly advertised, you do not need to complete this step. We recommend that you stop advertising the address range or any portion of the range from other locations before you advertise it through AWS. If you keep advertising your IP address range or any portion of it from other locations, we can't reliably support it or troubleshoot issues. Specifically, we can't guarantee that traffic to the address range or a portion of the range will enter our network. To minimize down time, you can configure your AWS resources to use an address from your address pool before it is advertised, and then simultaneously stop advertising it from the current location and start advertising it through AWS. For more information about allocating an Elastic IP address from your address pool, see [Allocate an Elastic IP address](working-with-eips.md#using-instance-addressing-eips-allocating). **Limitations** + You can run the **advertise-byoip-cidr** command at most once every 10 seconds, even if you specify different address ranges each time. + You can run the **withdraw-byoip-cidr** command at most once every 10 seconds, even if you specify different address ranges each time. To advertise the address range, use the following [advertise-byoip-cidr](https://docs.aws.amazon.com/cli/latest/reference/ec2/advertise-byoip-cidr.html) command. ``` aws ec2 advertise-byoip-cidr --cidr address-range --region us-east-1 ``` To stop advertising the address range, use the following [withdraw-byoip-cidr](https://docs.aws.amazon.com/cli/latest/reference/ec2/withdraw-byoip-cidr.html) command. ``` aws ec2 withdraw-byoip-cidr --cidr address-range --region us-east-1 ``` ## Deprovision the address range To stop using your address range with AWS, first release any Elastic IP addresses and disassociate any IPv6 CIDR blocks that are still allocated from the address pool. Then stop advertising the address range, and finally, deprovision the address range. You cannot deprovision a portion of the address range. If you want to use a more specific address range with AWS, deprovision the entire address range and provision a more specific address range. (IPv4) To release each Elastic IP address, use the following [release-address](https://docs.aws.amazon.com/cli/latest/reference/ec2/release-address.html) command. ``` aws ec2 release-address --allocation-id eipalloc-12345678abcabcabc --region us-east-1 ``` (IPv6) To disassociate an IPv6 CIDR block, use the following [disassociate-vpc-cidr-block](https://docs.aws.amazon.com/cli/latest/reference/ec2/disassociate-vpc-cidr-block.html) command. ``` aws ec2 disassociate-vpc-cidr-block --association-id vpc-cidr-assoc-12345abcd1234abc1 --region us-east-1 ``` To stop advertising the address range, use the following [withdraw-byoip-cidr](https://docs.aws.amazon.com/cli/latest/reference/ec2/withdraw-byoip-cidr.html) command. ``` aws ec2 withdraw-byoip-cidr --cidr address-range --region us-east-1 ``` To deprovision the address range, use the following [deprovision-byoip-cidr](https://docs.aws.amazon.com/cli/latest/reference/ec2/deprovision-byoip-cidr.html) command. ``` aws ec2 deprovision-byoip-cidr --cidr address-range --region us-east-1 ``` It can take up to a day to deprovision an address range. ## Validate your BYOIP 1. Validate the self-signed x.509 key pair Validate that the certificate has been uploaded and is valid via the whois command. For ARIN, use `whois -h whois.arin.net r + 2001:0DB8:6172::/48` to look up the RDAP record for your address range. Check the `Public Comments` section for the `NetRange` (network range) in the command output. The certificate should be added in the `Public Comments` section for the address range. You can inspect the `Public Comments` containing the certificate using the following command: ``` whois -h whois.arin.net r + 2001:0DB8:6172::/48 | grep Comments | grep BEGIN ``` This returns output with the contents of the key, which should be similar to the following: ``` Public Comments: -----BEGIN CERTIFICATE----- MIID1zCCAr+gAwIBAgIUBkRPNSLrPqbRAFP8RDAHSP+I1TowDQYJKoZIhvcNAQE LBQAwezELMAkGA1UEBhMCTloxETAPBgNVBAgMCEF1Y2tsYW5kMREwDwYDVQQHDA hBdWNrbGFuZDEcMBoGA1UECgwTQW1hem9uIFdlYiBTZXJ2aWNlczETMBEGA1UEC wwKQllPSVAgRGVtbzETMBEGA1UEAwwKQllPSVAgRGVtbzAeFw0yMTEyMDcyMDI0 NTRaFw0yMjEyMDcyMDI0NTRaMHsxCzAJBgNVBAYTAk5aMREwDwYDVQQIDAhBdWN rbGFuZDERMA8GA1UEBwwIQXVja2xhbmQxHDAaBgNVBAoME0FtYXpvbiBXZWIgU2 VydmljZXMxEzARBgNVBAsMCkJZT0lQIERlbW8xEzARBgNVBAMMCkJZT0lQIERlb W8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCfmacvDp0wZ0ceiXXc R/q27mHI/U5HKt7SST4X2eAqufR9wXkfNanAEskgAseyFypwEEQr4CJijI/5hp9 prh+jsWHWwkFRoBRR9FBtwcU/45XDXLga7D3stsI5QesHVRwOaXUdprAnndaTug mDPkD0vrl475JWDSIm+PUxGWLy+60aBqiaZq35wU/x+wXlAqBXg4MZK2KoUu27k Yt2zhmy0S7Ky+oRfRJ9QbAiSu/RwhQbh5Mkp1ZnVIc7NqnhdeIW48QaYjhMlUEf xdaqYUinzz8KpjfADZ4Hvqj9jWZ/eXo/9b2rGlHWkJsbhr0VEUyAGu1bwkgcdww 3A7NjOxQbAgMBAAGjUzBRMB0GA1UdDgQWBBStFyujN6SYBr2glHpGt0XGF7GbGT AfBgNVHSMEGDAWgBStFyujN6SYBr2glHpGt0XGF7GbGTAPBgNVHRMBAf8EBTADA QH/MA0GCSqGSIb3DQEBCwUAA4IBAQBX6nn6YLhz521lfyVfxY0t6o3410bQAeAF 08ud+ICtmQ4IO4A4B7zV3zIVYr0clrOOaFyLxngwMYN0XY5tVhDQqk4/gmDNEKS Zy2QkX4Eg0YUWVzOyt6fPzjOvJLcsqc1hcF9wySL507XQz76Uk5cFypBOzbnk35 UkWrzA9KK97cXckfIESgK/k1N4ecwxwG6VQ8mBGqVpPpey+dXpzzzv1iBKN/VY4 ydjgH/LBfdTsVarmmy2vtWBxwrqkFvpdhSGCvRDl/qdO/GIDJi77dmZWkh/ic90 MNk1f38gs1jrCj8lThoar17Uo9y/Q5qJIsoNPyQrJRzqFU9F3FBjiPJF -----END CERTIFICATE----- ``` For RIPE, use `whois -r -h whois.ripe.net 2001:0DB8:7269::/48` to look up the RDAP record for your address range. Check the `descr` section for the `inetnum` object (network range) in the command output. The certificate should be added as a new `descr` field for the address range. You can inspect the `descr` containing the certificate using the following command: ``` whois -r -h whois.ripe.net 2001:0DB8:7269::/48 | grep descr | grep BEGIN ``` This returns output with the contents of the key, which should be similar to the following: ``` descr: -----BEGIN CERTIFICATE-----MIID1zCCAr+gAwIBAgIUBkRPNSLrPqbRAFP8 RDAHSP+I1TowDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCTloxETAPBgNVBAg MCEF1Y2tsYW5kMREwDwYDVQQHDAhBdWNrbGFuZDEcMBoGA1UECgwTQW1hem9uIF dlYiBTZXJ2aWNlczETMBEGA1UECwwKQllPSVAgRGVtbzETMBEGA1UEAwwKQllPS VAgRGVtbzAeFw0yMTEyMDcyMDI0NTRaFw0yMjEyMDcyMDI0NTRaMHsxCzAJBgNV BAYTAk5aMREwDwYDVQQIDAhBdWNrbGFuZDERMA8GA1UEBwwIQXVja2xhbmQxHDA aBgNVBAoME0FtYXpvbiBXZWIgU2VydmljZXMxEzARBgNVBAsMCkJZT0lQIERlbW 8xEzARBgNVBAMMCkJZT0lQIERlbW8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwg gEKAoIBAQCfmacvDp0wZ0ceiXXcR/q27mHI/U5HKt7SST4X2eAqufR9wXkfNanA EskgAseyFypwEEQr4CJijI/5hp9prh+jsWHWwkFRoBRR9FBtwcU/45XDXLga7D3 stsI5QesHVRwOaXUdprAnndaTugmDPkD0vrl475JWDSIm+PUxGWLy+60aBqiaZq 35wU/x+wXlAqBXg4MZK2KoUu27kYt2zhmy0S7Ky+oRfRJ9QbAiSu/RwhQbh5Mkp 1ZnVIc7NqnhdeIW48QaYjhMlUEfxdaqYUinzz8KpjfADZ4Hvqj9jWZ/eXo/9b2r GlHWkJsbhr0VEUyAGu1bwkgcdww3A7NjOxQbAgMBAAGjUzBRMB0GA1UdDgQWBBS tFyujN6SYBr2glHpGt0XGF7GbGTAfBgNVHSMEGDAWgBStFyujN6SYBr2glHpGt0 XGF7GbGTAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBX6nn6Y Lhz521lfyVfxY0t6o3410bQAeAF08ud+ICtmQ4IO4A4B7zV3zIVYr0clrOOaFyL xngwMYN0XY5tVhDQqk4/gmDNEKSZy2QkX4Eg0YUWVzOyt6fPzjOvJLcsqc1hcF9 wySL507XQz76Uk5cFypBOzbnk35UkWrzA9KK97cXckfIESgK/k1N4ecwxwG6VQ8 mBGqVpPpey+dXpzzzv1iBKN/VY4ydjgH/LBfdTsVarmmy2vtWBxwrqkFvpdhSGC vRDl/qdO/GIDJi77dmZWkh/ic90MNk1f38gs1jrCj8lThoar17Uo9y/Q5qJIsoN PyQrJRzqFU9F3FBjiPJF -----END CERTIFICATE----- ``` For APNIC, use `whois -h whois.apnic.net 2001:0DB8:6170::/48` to look up the RDAP record for your BYOIP address range. Check the `remarks` section for the `inetnum` object (network range) in the command output. The certificate should be added as a new `remarks` field for the address range. You can inspect the `remarks` containing the certificate using the following command: ``` whois -h whois.apnic.net 2001:0DB8:6170::/48 | grep remarks | grep BEGIN ``` This returns output with the contents of the key, which should be similar to the following: ``` remarks: -----BEGIN CERTIFICATE----- MIID1zCCAr+gAwIBAgIUBkRPNSLrPqbRAFP8RDAHSP+I1TowDQYJKoZIhvcNAQE LBQAwezELMAkGA1UEBhMCTloxETAPBgNVBAgMCEF1Y2tsYW5kMREwDwYDVQQHDA hBdWNrbGFuZDEcMBoGA1UECgwTQW1hem9uIFdlYiBTZXJ2aWNlczETMBEGA1UEC wwKQllPSVAgRGVtbzETMBEGA1UEAwwKQllPSVAgRGVtbzAeFw0yMTEyMDcyMDI0 NTRaFw0yMjEyMDcyMDI0NTRaMHsxCzAJBgNVBAYTAk5aMREwDwYDVQQIDAhBdWN rbGFuZDERMA8GA1UEBwwIQXVja2xhbmQxHDAaBgNVBAoME0FtYXpvbiBXZWIgU2 VydmljZXMxEzARBgNVBAsMCkJZT0lQIERlbW8xEzARBgNVBAMMCkJZT0lQIERlb W8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCfmacvDp0wZ0ceiXXc R/q27mHI/U5HKt7SST4X2eAqufR9wXkfNanAEskgAseyFypwEEQr4CJijI/5hp9 prh+jsWHWwkFRoBRR9FBtwcU/45XDXLga7D3stsI5QesHVRwOaXUdprAnndaTug mDPkD0vrl475JWDSIm+PUxGWLy+60aBqiaZq35wU/x+wXlAqBXg4MZK2KoUu27k Yt2zhmy0S7Ky+oRfRJ9QbAiSu/RwhQbh5Mkp1ZnVIc7NqnhdeIW48QaYjhMlUEf xdaqYUinzz8KpjfADZ4Hvqj9jWZ/eXo/9b2rGlHWkJsbhr0VEUyAGu1bwkgcdww 3A7NjOxQbAgMBAAGjUzBRMB0GA1UdDgQWBBStFyujN6SYBr2glHpGt0XGF7GbGT AfBgNVHSMEGDAWgBStFyujN6SYBr2glHpGt0XGF7GbGTAPBgNVHRMBAf8EBTADA QH/MA0GCSqGSIb3DQEBCwUAA4IBAQBX6nn6YLhz521lfyVfxY0t6o3410bQAeAF 08ud+ICtmQ4IO4A4B7zV3zIVYr0clrOOaFyLxngwMYN0XY5tVhDQqk4/gmDNEKS Zy2QkX4Eg0YUWVzOyt6fPzjOvJLcsqc1hcF9wySL507XQz76Uk5cFypBOzbnk35 UkWrzA9KK97cXckfIESgK/k1N4ecwxwG6VQ8mBGqVpPpey+dXpzzzv1iBKN/VY4 ydjgH/LBfdTsVarmmy2vtWBxwrqkFvpdhSGCvRDl/qdO/GIDJi77dmZWkh/ic90 MNk1f38gs1jrCj8lThoar17Uo9y/Q5qJIsoNPyQrJRzqFU9F3FBjiPJF -----END CERTIFICATE----- ``` 1. Validate the creation of a ROA object Validate the successful creation of the ROA objects using the RIPEstat Data API. Be sure to test your address range against the Amazon ASNs 16509 and 14618, plus the ASNs that are currently authorized to advertise the address range. You can inspect the ROA objects from different Amazon ASNs with your address range by using the following command: ``` curl --location --request GET "https://stat.ripe.net/data/rpki-validation/data.json?resource=ASN&prefix=CIDR ``` In this example output, the response has a result of `"status": "valid"` for the Amazon ASN 16509. This indicates the ROA object for the address range was created successfully: ``` { "messages": [], "see_also": [], "version": "0.3", "data_call_name": "rpki-validation", "data_call_status": "supported", "cached": false, "data": { "validating_roas": [ { "origin": "16509", "prefix": "2001:0DB8::/32", "max_length": 48, "validity": "valid" }, { "origin": "14618", "prefix": "2001:0DB8::/32", "max_length": 48, "validity": "invalid_asn" }, { "origin": "64496", "prefix": "2001:0DB8::/32", "max_length": 48, "validity": "invalid_asn" } ], "status": "valid", "validator": "routinator", "resource": "16509", "prefix": "2001:0DB8::/32" }, "query_id": "20230224152430-81e6384e-21ba-4a86-852a-31850787105f", "process_time": 58, "server_id": "app116", "build_version": "live.2023.2.1.142", "status": "ok", "status_code": 200, "time": "2023-02-24T15:24:30.773654" } ``` A status of `“unknown”` indicates the ROA object for the address range has not been created. A status of `“invalid_asn”` indicates that the ROA object for the address range was not created successfully. # Use your BYOIP address range in Amazon EC2 Use your address range You can view and use the IPv4 and IPv6 address ranges that you've provisioned in your account. For more information, see [Onboard your address range for use in Amazon EC2](byoip-onboard.md). ## IPv4 address ranges You can create an Elastic IP address from your IPv4 address pool and use it with your AWS resources, such as EC2 instances, NAT gateways, and Network Load Balancers. To view information about the IPv4 address pools that you've provisioned in your account, use the following [describe-public-ipv4-pools](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-public-ipv4-pools.html) command. ``` aws ec2 describe-public-ipv4-pools --region us-east-1 ``` To create an Elastic IP address from your IPv4 address pool, use the [allocate-address](https://docs.aws.amazon.com/cli/latest/reference/ec2/allocate-address.html) command. You can use the `--public-ipv4-pool` option to specify the ID of the address pool returned by `describe-byoip-cidrs`. Or you can use the `--address` option to specify an address from the address range that you provisioned. ## IPv6 address ranges To view information about the IPv6 address pools that you've provisioned in your account, use the following [describe-ipv6-pools](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-ipv6-pools.html) command. ``` aws ec2 describe-ipv6-pools --region us-east-1 ``` To create a VPC and specify an IPv6 CIDR from your IPv6 address pool, use the following [create-vpc](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-vpc.html) command. To let Amazon choose the IPv6 CIDR from your IPv6 address pool, omit the `--ipv6-cidr-block` option. ``` aws ec2 create-vpc --cidr-block 10.0.0.0/16 --ipv6-cidr-block ipv6-cidr --ipv6-pool pool-id --region us-east-1 ``` To associate an IPv6 CIDR block from your IPv6 address pool with a VPC, use the following [associate-vpc-cidr-block](https://docs.aws.amazon.com/cli/latest/reference/ec2/associate-vpc-cidr-block.html) command. To let Amazon choose the IPv6 CIDR from your IPv6 address pool, omit the `--ipv6-cidr-block` option. ``` aws ec2 associate-vpc-cidr-block --vpc-id vpc-123456789abc123ab --ipv6-cidr-block ipv6-cidr --ipv6-pool pool-id --region us-east-1 ``` To view your VPCs and the associated IPv6 address pool information, use the [describe-vpcs](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-vpcs.html) command. To view information about associated IPv6 CIDR blocks from a specific IPv6 address pool, use the following [get-associated-ipv6-pool-cidrs](https://docs.aws.amazon.com/cli/latest/reference/ec2/get-associated-ipv6-pool-cidrs.html) command. ``` aws ec2 get-associated-ipv6-pool-cidrs --pool-id pool-id --region us-east-1 ``` If you disassociate the IPv6 CIDR block from your VPC, it's released back into your IPv6 address pool. # Elastic IP addresses Elastic IP addresses An *Elastic IP address* is a static IPv4 address designed for dynamic cloud computing. An Elastic IP address is allocated to your AWS account, and is yours until you release it. By using an Elastic IP address, you can mask the failure of an instance or software by rapidly remapping the address to another instance in your account. Alternatively, you can specify the Elastic IP address in a DNS record for your domain, so that your domain points to your instance. For more information, see the documentation for your domain registrar. An Elastic IP address is a public IPv4 address, which is reachable from the internet. If you need to connect to an instance that does not have a public IPv4 address, you can associate an Elastic IP address with your instance to enable communication with the internet. **Topics** + [ ## Elastic IP address pricing ](#eip-pricing) + [ ## Elastic IP address basics ](#eip-basics) + [ ## Elastic IP address quota ](#using-instance-addressing-limit) + [ # Associate an Elastic IP address with an instance ](working-with-eips.md) + [ # Transfer an Elastic IP address between AWS accounts ](transfer-EIPs-intro-ec2.md) + [ # Release an Elastic IP address ](using-instance-addressing-eips-releasing.md) + [ # Create a reverse DNS record for email on Amazon EC2 ](Using_Elastic_Addressing_Reverse_DNS.md) ## Elastic IP address pricing There is a charge for all Elastic IP addresses whether they are in use (allocated to a resource, like an EC2 instance) or idle (created in your account but unallocated). AWS charges for all public IPv4 addresses, including public IPv4 addresses associated with running instances and Elastic IP addresses. For more information, see the **Public IPv4 Address** tab on the [Amazon VPC pricing page](https://aws.amazon.com/vpc/pricing/). ## Elastic IP address basics The following are the basic characteristics of an Elastic IP address: + An Elastic IP address is static; it does not change over time. + An Elastic IP address is for use in a specific Region only, and cannot be moved to a different Region. + An Elastic IP address comes from Amazon's pool of IPv4 addresses, or from a custom IPv4 address pool that you have brought to your AWS account. We do not support Elastic IP addresses for IPv6. + To use an Elastic IP address, you first allocate one to your account, and then associate it with your instance or a network interface. + When you associate an Elastic IP address with an instance, it is also associated with the instance's primary network interface. When you associate an Elastic IP address with a network interface that is attached to an instance, it is also associated with the instance. + When you associate an Elastic IP address with an instance or its primary network interface, if the instance already has a public IPv4 address associated with it, that public IPv4 address is released back into Amazon's pool of public IPv4 addresses and the Elastic IP address is associated with the instance instead. You cannot reuse the public IPv4 address previously associated with the instance and you cannot convert that public IPv4 address to an Elastic IP address. For more information, see [Public IPv4 addresses](using-instance-addressing.md#concepts-public-addresses). + You can disassociate an Elastic IP address from a resource, and then associate it with a different resource. To avoid unexpected behavior, ensure that all active connections to the resource named in the existing association are closed before you make the change. After you have associated your Elastic IP address to a different resource, you can reopen your connections to the newly associated resource. + A disassociated Elastic IP address remains allocated to your account until you explicitly release it. You are charged for all Elastic IP addresses in your account, regardless of whether they are associated or disassociated with an instance. For more information, see the **Public IPv4 Address** tab on the [Amazon VPC pricing](https://aws.amazon.com/vpc/pricing/) page. + When you associate an Elastic IP address with an instance that previously had a public IPv4 address, the public DNS host name of the instance changes to match the Elastic IP address. + We resolve a public DNS host name to the public IPv4 address or the Elastic IP address of the instance outside the network of the instance, and to the private IPv4 address of the instance from within the network of the instance. + When you allocate an Elastic IP address from an IP address pool that you have brought to your AWS account, it does not count toward your Elastic IP address limits. For more information, see [Elastic IP address quota](#using-instance-addressing-limit). + When you allocate the Elastic IP addresses, you can associate the Elastic IP addresses with a network border group. This is the location from which we advertise the CIDR block. Setting the network border group limits the CIDR block to this group. If you do not specify the network border group, we set the border group containing all of the Availability Zones in the Region (for example, `us-west-2`). + An Elastic IP address is for use in a specific network border group only. ## Elastic IP address quota By default, all AWS accounts have a quota of five (5) Elastic IP addresses per Region, because public (IPv4) internet addresses are a scarce public resource. We strongly recommend that you use Elastic IP addresses primarily for their ability to remap the address to another instance in the case of instance failure, and to use [DNS hostnames](https://docs.aws.amazon.com/vpc/latest/userguide/AmazonDNS-concepts.html#vpc-dns-hostnames) for all other inter-node communication. If you think your architecture warrants additional Elastic IP addresses, you can request a quota increase directly from the Service Quotas console. To request a quota increase, choose **Request increase at account-level**. For more information, see [Amazon EC2 service quotas](ec2-resource-limits.md). # Associate an Elastic IP address with an instance Associate an Elastic IP address After you allocate an Elastic IP address, you can associate it with an AWS resource, such as an EC2 instance, NAT gateway, or Network Load Balancer. To associate an Elastic IP address with a different AWS resource later on, you can disassociate it from its current resource and then associated it with the new resource. **Topics** + [ ## Allocate an Elastic IP address ](#using-instance-addressing-eips-allocating) + [ ## Associate an Elastic IP address ](#using-instance-addressing-eips-associating) + [ ## Disassociate an Elastic IP address ](#using-instance-addressing-eips-associating-different) ## Allocate an Elastic IP address You can allocate an Elastic IP address for use in a Region. There is a charge for all Elastic IP addresses whether they are in use (associated with a resource, like an EC2 instance) or idle (created in your account but unassociated). ------ #### [ Console ] **To allocate an Elastic IP address** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network & Security**, **Elastic IPs**. 1. Choose **Allocate Elastic IP address**. 1. (Optional) When you allocate an Elastic IP address (EIP), you choose the **Network border group** in which to allocate the EIP. A network border group is a collection of Availability Zones (AZs), Local Zones, or Wavelength Zones from which AWS advertises a public IP address. Local Zones and Wavelength Zones may have different network border groups than the AZs in a Region to ensure minimum latency or physical distance between the AWS network and the customers accessing the resources in these Zones. **Important** You must allocate an EIP in the same network border group as the AWS resource that will be associated with the EIP. An EIP in one network border group can only be advertised in zones in that network border group and not in any other zones represented by other network border groups. If you have Local Zones or Wavelength Zones enabled (for more information, see [Enable a Local Zone](https://docs.aws.amazon.com/local-zones/latest/ug/getting-started.html#getting-started-find-local-zone) or [Enable Wavelength Zones](https://docs.aws.amazon.com//wavelength/latest/developerguide/get-started-wavelength.html#enable-zone-group)), you can choose a network border group for AZs, Local Zones, or Wavelength Zones. Choose the network border group carefully as the EIP and the AWS resource it is associated with must reside in the same network border group. You can use the EC2 console to view the network border group that your Availability Zones, Local Zones, or Wavelength Zones are in. Typically, all Availability Zones in a Region belong to the same network border group, whereas Local Zones or Wavelength Zones belong to their own separate network border groups. If you don't have Local Zones or Wavelength Zones enabled, when you allocate an EIP, the network border group that represents all of the AZs for the Region (such as `us-west-2`) is predefined for you and you cannot change it. This means that the EIP that you allocate to this network border group will be advertised in all AZs in the Region you're in. 1. For **Public IPv4 address pool**, choose one of the following: + **Amazon's pool of IPv4 addresses**—If you want an IPv4 address to be allocated from Amazon's pool of IPv4 addresses. + **Public IPv4 address that you bring to your AWS account**—If you want to allocate a non-contiguous (non-sequential) public IPv4 address from an IP address pool that you have brought to your AWS account. This option is disabled if you do not have any IP address pools. For more information about bringing your own IP address range to your AWS account, see [Bring your own IP addresses (BYOIP) to Amazon EC2](ec2-byoip.md). + **Customer owned pool of IPv4 addresses**—If you want to allocate an IPv4 address from a pool created from your on-premises network for use with an AWS Outpost. This option is disabled if you do not have an AWS Outpost. + **Allocate using an IPAM IPv4 pool**: If you want to allocate sequential Elastic IP addresses from a contiguous public IPv4 block in an IPAM pool. Allocating sequential Elastic IP addresses can significantly reduce management overhead for security access control lists and simplify IP address allocation and tracking for enterprises scaling on AWS. For more information, see [Allocate sequential Elastic IP addresses from an IPAM pool](https://docs.aws.amazon.com/vpc/latest/ipam/tutorials-eip-pool.html) in the *Amazon VPC IPAM User Guide*. 1. (Optional) To add a tag, choose **Add new tag** and enter a tag key and a tag value. ------ #### [ AWS CLI ] **To allocate an Elastic IP address** Use the [allocate-address](https://docs.aws.amazon.com/cli/latest/reference/ec2/allocate-address.html) AWS CLI command. In the following example, Amazon EC2 selects an address from Amazon's address pool. ``` aws ec2 allocate-address ``` In the following example, Amazon EC2 selects an address from the specified pool that you brought to AWS using BYOIP. ``` aws ec2 allocate-address \ --public-ipv4-pool ipv4pool-ec2-012345abcdef67890 ``` The following example specifies an address from the specified IPv4 IPAM pool. ``` aws ec2 allocate-address \ --ipam-pool-id ipam-pool-1234567890abcdef0 \ --address 192.0.2.0 ``` ------ #### [ PowerShell ] **To allocate an Elastic IP address** Use the [New-EC2Address](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2Address.html) cmdlet. In the following example, Amazon EC2 selects an address from Amazon's address pool. ``` New-EC2Address ``` In the following example, Amazon EC2 selects an address from the specified pool that you brought to AWS using BYOIP. ``` New-EC2Address ` -PublicIpv4Pool ipv4pool-ec2-012345abcdef67890 ``` The following example specifies an address from the specified IPv4 IPAM pool. ``` New-EC2Address ` -IpamPoolId ipam-pool-1234567890abcdef0 ` -Address 192.0.2.0 ``` ------ ## Associate an Elastic IP address If you're associating an Elastic IP address with your instance to enable communication with the internet, you must also ensure that your instance is in a public subnet. For more information, see [Enable internet access using an internet gateway](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html) in the *Amazon VPC User Guide*. ------ #### [ Console ] **To associate an Elastic IP address with an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Elastic IPs**. 1. Select the Elastic IP address to associate and choose **Actions**, **Associate Elastic IP address**. 1. For **Resource type**, choose **Instance**. 1. For instance, choose the instance with which to associate the Elastic IP address. You can also enter text to search for a specific instance. 1. (Optional) For **Private IP address**, specify a private IP address with which to associate the Elastic IP address. 1. Choose **Associate**. **To associate an Elastic IP address with a network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Elastic IPs**. 1. Select the Elastic IP address to associate and choose **Actions**, **Associate Elastic IP address**. 1. For **Resource type**, choose **Network interface**. 1. For **Network interface**, choose the network interface with which to associate the Elastic IP address. You can also enter text to search for a specific network interface. 1. (Optional) For **Private IP address**, specify a private IP address with which to associate the Elastic IP address. 1. Choose **Associate**. ------ #### [ AWS CLI ] **To associate an Elastic IP address** Use the [associate-address](https://docs.aws.amazon.com/cli/latest/reference/ec2/associate-address.html) AWS CLI command. ``` aws ec2 associate-address \ --instance-id i-0b263919b6498b123 \ --allocation-id eipalloc-64d5890a ``` ------ #### [ PowerShell ] **To associate an Elastic IP address** Use the [Register-EC2Address](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Address.html) cmdlet. ``` Register-EC2Address ` -InstanceId i-0b263919b6498b123 ` -AllocationId eipalloc-64d5890a ``` ------ ## Disassociate an Elastic IP address You can disassociate an Elastic IP address from an instance or network interface at any time. After you disassociate the Elastic IP address, you can associate it with another resource. ------ #### [ Console ] **To disassociate and reassociate an Elastic IP address** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Elastic IPs**. 1. Select the Elastic IP address to disassociate, choose **Actions**, **Disassociate Elastic IP address**. 1. Choose **Disassociate**. ------ #### [ AWS CLI ] **To disassociate an Elastic IP address** Use the [disassociate-address](https://docs.aws.amazon.com/cli/latest/reference/ec2/disassociate-address.html) AWS CLI command. ``` aws ec2 disassociate-address --association-id eipassoc-12345678 ``` ------ #### [ PowerShell ] **To disassociate an Elastic IP address** Use the [Unregister-EC2Address](https://docs.aws.amazon.com/powershell/latest/reference/items/Unregister-EC2Address.html) cmdlet. ``` Unregister-EC2Address -AssociationId eipassoc-12345678 ``` ------ # Transfer an Elastic IP address between AWS accounts Transfer an Elastic IP address You can transfer an Elastic IP address from one AWS account to another. This can be helpful in the following situations: + **Disaster recovery** – Quickly remap the IP addresses for public-facing internet workloads during emergency events. + **Organizational restructuring** – Quickly move a workload from one AWS account to another. An address transfer avoids the need to wait for new Elastic IP addresses to be allowed by your security groups and network ACLs. + **Centralized security administration** – Use a centralized AWS security account to track and transfer Elastic IP addresses that have been vetted for security compliance. **Pricing** There is no charge for transferring Elastic IP addresses. **Topics** + [ ## Enable Elastic IP address transfer ](#using-instance-addressing-eips-transfer-enable-ec2) + [ ## Accept a transferred Elastic IP address ](#using-instance-addressing-eips-transfer-accept-ec2) + [ ## Disable Elastic IP address transfer ](#using-instance-addressing-eips-transfer-disable-ec2) ## Enable Elastic IP address transfer This section describes how to accept a transferred Elastic IP address. Note the following limitations related to enabling Elastic IP addresses for transfer: + You can transfer Elastic IP addresses from any AWS account (source account) to any other AWS account in the same AWS Region (transfer account). You cannot transfer Elastic IP addresses to a different Region. + When you transfer an Elastic IP address, there is a two-step handshake between the AWS accounts. When the source account starts the transfer, the transfer accounts have seven days to accept the Elastic IP address transfer. During those seven days, the source account can view the pending transfer (for example in the AWS console or by using the [describe-address-transfers](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-address-transfers.html) command). After seven days, the transfer expires and ownership of the Elastic IP address returns to the source account. + Accepted transfers are visible to the source account (for example in the AWS console or by using the [describe-address-transfers](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-address-transfers.html) command) for 14 days after the transfers have been accepted. + AWS does not notify transfer accounts about pending Elastic IP address transfer requests. The owner of the source account must notify the owner of the transfer account that there is an Elastic IP address transfer request that they must accept. + Any tags that are associated with an Elastic IP address being transferred are reset when the transfer is complete. + You cannot transfer Elastic IP addresses allocated from public IPv4 address pools that you bring to your AWS account – commonly referred to as Bring Your Own IP (BYOIP) address pools. + You cannot transfer Elastic IP addresses allocated from an Amazon-provided contiguous public IPv4 Amazon VPC IP Address Manager (IPAM) pool. Instead, IPAM allows you to share IPAM pools across AWS accounts by integrating IPAM with AWS Organizations and using AWS RAM. For more information, see [Allocate sequential Elastic IP addresses from an IPAM pool](https://docs.aws.amazon.com/vpc/latest/ipam/tutorials-eip-pool.html) in the *Amazon VPC IPAM User Guide*. + If you attempt to transfer an Elastic IP address that has a reverse DNS record associated with it, you can begin the transfer process, but the transfer account will not be able to accept the transfer until the associated DNS record is removed. + If you have enabled and configured AWS Outposts, you might have allocated Elastic IP addresses from a customer-owned IP address pool (CoIP). You cannot transfer Elastic IP addresses allocated from a CoIP. However, you can use AWS RAM to share a CoIP with another account. For more information, see [Customer-owned IP addresses](https://docs.aws.amazon.com/outposts/latest/userguide/routing.html#ip-addressing) in the *AWS Outposts User Guide*. + You can use Amazon VPC IPAM to track the transfer of Elastic IP addresses to accounts in an organization from AWS Organizations. For more information, see [View IP address history](https://docs.aws.amazon.com/vpc/latest/ipam/view-history-cidr-ipam.html). If an Elastic IP address is transferred to an AWS account outside of the organization, the IPAM audit history of the Elastic IP address is lost. These steps must be completed by the source account. ------ #### [ Console ] **To enable Elastic IP address transfer** 1. Ensure that you're using the source AWS account. 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Elastic IPs**. 1. Select one or more Elastic IP address to enable for transfer and choose **Actions**, **Enable transfer**. 1. If you are transferring multiple Elastic IP addresses, you’ll see the **Transfer type** option. Choose one of the following options: + Choose **Single account** if you are transferring the Elastic IP addresses to a single AWS account. + Choose **Multiple accounts** if you are transferring the Elastic IP addresses to multiple AWS accounts. 1. Under **Transfer account ID**, enter the IDs of the AWS accounts that you want to transfer the Elastic IP addresses to. 1. Confirm the transfer by entering **enable** in the text box. 1. Choose **Submit**. 1. To accept the transfer, see [Accept a transferred Elastic IP address](#using-instance-addressing-eips-transfer-accept-ec2). To disable the transfer, see [Disable Elastic IP address transfer](#using-instance-addressing-eips-transfer-disable-ec2). ------ #### [ AWS CLI ] **To enable Elastic IP address transfer** Use the [enable-address-transfer](https://docs.aws.amazon.com/cli/latest/reference/ec2/enable-address-transfer.html) command. ``` aws ec2 enable-address-transfer \ --allocation-id eipalloc-09ad461b0d03f6aaf \ --transfer-account-id 123456789012 ``` ------ #### [ PowerShell ] **To enable Elastic IP address transfer** Use the [Enable-EC2AddressTransfer](https://docs.aws.amazon.com/powershell/latest/reference/items/Enable-EC2AddressTransfer.html) cmdlet. ``` Enable-EC2AddressTransfer ` -AllocationId eipalloc-09ad461b0d03f6aaf ` -TransferAccountId 123456789012 ``` ------ ## Accept a transferred Elastic IP address This section describes how to accept a transferred Elastic IP address. When you transfer an Elastic IP address, there is a two-step handshake between the AWS accounts. When the source account starts the transfer, the transfer accounts have seven days to accept the Elastic IP address transfer. During those seven days, the source account can view the pending transfer (for example in the AWS console or by using the [describe-address-transfers](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-address-transfers.html) command). After seven days, the transfer expires and ownership of the Elastic IP address returns to the source account. When accepting transfers, note the following exceptions that might occur and how to resolve them: + **AddressLimitExceeded**: If your transfer account has exceeded the Elastic IP address quota, the source account can enable Elastic IP address transfer, but this exception occurs when the transfer account tries to accept the transfer. By default, all AWS accounts are limited to 5 Elastic IP addresses per Region. See [Elastic IP address quota](elastic-ip-addresses-eip.md#using-instance-addressing-limit) for instructions on increasing the limit. + **InvalidTransfer.AddressCustomPtrSet**: If you or someone in your organization has configured the Elastic IP address that you are attempting to transfer to use reverse DNS lookup, the source account can enable transfer for the Elastic IP address, but this exception occurs when the transfer account tries to accept the transfer. To resolve this issue, the source account must remove the DNS record for the Elastic IP address. For more information, see [Create a reverse DNS record for email on Amazon EC2](Using_Elastic_Addressing_Reverse_DNS.md). + **InvalidTransfer.AddressAssociated**: If an Elastic IP address is associated with an ENI or EC2 instance, the source account can enable transfer for the Elastic IP address, but this exception occurs when the transfer account tries to accept the transfer. To resolve this issue, the source account must disassociate the Elastic IP address. For more information, see [Disassociate an Elastic IP address](working-with-eips.md#using-instance-addressing-eips-associating-different). For any other exceptions, [contact Support](https://aws.amazon.com/contact-us/). These steps must be completed by the transfer account. ------ #### [ Console ] **To accept an Elastic IP address transfer** 1. Ensure that you're using the transfer account. 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Elastic IPs**. 1. Choose **Actions**, **Accept transfer**. 1. No tags that are associated with the Elastic IP address being transferred are transferred with the Elastic IP address when you accept the transfer. If you want to define a **Name** tag for the Elastic IP address that you are accepting, select **Create a tag with a key of 'Name' and a value that you specify**. 1. Enter the Elastic IP address that you want to transfer. 1. If you are accepting multiple transferred Elastic IP addresses, choose **Add address** to enter an additional Elastic IP address. 1. Choose **Submit**. ------ #### [ AWS CLI ] **To accept an Elastic IP address transfer** Use the [accept-address-transfer](https://docs.aws.amazon.com/cli/latest/reference/ec2/accept-address-transfer.html) command. ``` aws ec2 accept-address-transfer --address 100.21.184.216 ``` ------ #### [ PowerShell ] **To accept an Elastic IP address transfer** Use the [Approve-EC2AddressTransfer](https://docs.aws.amazon.com/powershell/latest/reference/items/Approve-EC2AddressTransfer.html) cmdlet. ``` Approve-EC2AddressTransfer -Address 100.21.184.216 ``` ------ ## Disable Elastic IP address transfer This section describes how to disable an Elastic IP transfer after the transfer has been enabled. These steps must be completed by the source account that enabled the transfer. ------ #### [ Console ] **To disable an Elastic IP address transfer** 1. Ensure that you're using the source AWS account. 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Elastic IPs**. 1. In the resource list of Elastic IPs, ensure that you have the property enabled that shows the column **Transfer status**. 1. Select one or more Elastic IP address that have a **Transfer status** of **Pending**, and choose **Actions**, **Disable transfer**. 1. Confirm by entering **disable** in the text box. 1. Choose **Submit**. ------ #### [ AWS CLI ] **To disable Elastic IP address transfer** Use the [disable-address-transfer](https://docs.aws.amazon.com/cli/latest/reference/ec2/disable-address-transfer.html) command. ``` aws ec2 disable-address-transfer --allocation-id eipalloc-09ad461b0d03f6aaf ``` ------ #### [ PowerShell ] **To disable Elastic IP address transfer** Use the [Disable-EC2AddressTransfer](https://docs.aws.amazon.com/powershell/latest/reference/items/Disable-EC2AddressTransfer.html) cmdlet. ``` Disable-EC2AddressTransfer -AllocationId eipalloc-09ad461b0d03f6aaf ``` ------ # Release an Elastic IP address If you no longer need an Elastic IP address, we recommend that you release it. The Elastic IP address to release must not be currently associated with an AWS resource. ------ #### [ Console ] **To release an Elastic IP address** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Elastic IPs**. 1. Select the Elastic IP address to release and choose **Actions**, **Release Elastic IP addresses**. 1. Choose **Release**. ------ #### [ AWS CLI ] **To release an Elastic IP address** Use the [release-address](https://docs.aws.amazon.com/cli/latest/reference/ec2/release-address.html) AWS CLI command. ``` aws ec2 release-address --allocation-id eipalloc-64d5890a ``` ------ #### [ PowerShell ] **To release an Elastic IP address** Use the [Remove-EC2Address](https://docs.aws.amazon.com/powershell/latest/reference/items/Remove-EC2Address.html) cmdlet. ``` Remove-EC2Address -AllocationId eipalloc-64d5890a ``` ------ After you release your Elastic IP address, you might be able to recover. The following rules apply: + You can't recover an Elastic IP address if it has been allocated to another AWS account, or if it will result in your exceeding your Elastic IP address limit. + You can't recover tags associated with an Elastic IP address. ------ #### [ AWS CLI ] **To recover an Elastic IP address** Use the [allocate-address](https://docs.aws.amazon.com/cli/latest/reference/ec2/allocate-address.html) command. ``` aws ec2 allocate-address \ --domain vpc \ --address 203.0.113.3 ``` ------ #### [ PowerShell ] **To recover an Elastic IP address** Use the [New-EC2Address](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2Address.html) cmdlet. ``` New-EC2Address ` -Address 203.0.113.3 ` -Domain vpc ` -Region us-east-1 ``` ------ # Create a reverse DNS record for email on Amazon EC2 Use reverse DNS for email applications If you intend to send email to third parties from an EC2 instance, we recommend that you provision one or more Elastic IP addresses and assign static reverse DNS records to the Elastic IP addresses that you use to send email. This can help you avoid having your email flagged as spam by some anti-spam organizations. AWS works with ISPs and internet anti-spam organizations to reduce the chance that your email sent from these addresses will be flagged as spam. **Considerations** + Before you create a reverse DNS record, you must set a corresponding forward DNS record (record type A) that points to your Elastic IP address. + If a reverse DNS record is associated with an Elastic IP address, the Elastic IP address is locked to your account and cannot be released from your account until the record is removed. + If you contacted Support to set up reverse DNS for an Elastic IP address, you can remove the reverse DNS, but you can't release the Elastic IP address because it is locked by Support. To unlock the Elastic IP address, contact [AWS Support](https://console.aws.amazon.com/support/home#/). After the Elastic IP address is unlocked, you can release it. + [AWS GovCloud (US) Region] You can't create a reverse DNS record. AWS must assign the static reverse DNS records for you. Open a support case to remove reverse DNS and email sending limitations. You must provide your Elastic IP addresses and reverse DNS records. ## Create a reverse DNS record You can create a reverse DNS record for your Elastic IP address as follows. ------ #### [ Console ] **To create a reverse DNS record** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Elastic IPs**. 1. Select the Elastic IP address and choose **Actions**, **Update reverse DNS**. 1. For **Reverse DNS domain name**, enter the domain name. 1. Enter **update** to confirm. 1. Choose **Update**. ------ #### [ AWS CLI ] **To create a reverse DNS record** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-address-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-address-attribute.html) command. ``` aws ec2 modify-address-attribute \ --allocation-id eipalloc-abcdef01234567890 \ --domain-name example.com ``` The following is example output. ``` { "Addresses": [ { "PublicIp": "192.0.2.0", "AllocationId": "eipalloc-abcdef01234567890", "PtrRecord": "example.net.", "PtrRecordUpdate": { "Value": "example.com.", "Status": "PENDING" } } ] } ``` ------ #### [ PowerShell ] **To create a reverse DNS record** Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2AddressAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2AddressAttribute.html) cmdlet. ``` Edit-EC2AddressAttribute ` -AllocationId 'eipalloc-abcdef01234567890' ` -DomainName 'example.com' | Format-List ` AllocationId, PtrRecord, PublicIp, @{Name='PtrRecordUpdate';Expression={$_.PtrRecordUpdate | Format-List | Out-String}} ``` The following is example output. ``` AllocationId : eipalloc-abcdef01234567890 PtrRecord : example.net. PublicIp : 192.0.2.0 PtrRecordUpdate : Reason : Status : PENDING Value : example.com. ``` ------ ## Remove a reverse DNS record You can remove a reverse DNS record from your Elastic IP address as follows. If you receive the following error, you can submit a [Request to remove email sending restrictions](https://repost.aws/knowledge-center/ec2-port-25-throttle) to Support for assistance. ``` The address cannot be released because it is locked to your account. ``` ------ #### [ Console ] **To remove a reverse DNS record** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Elastic IPs**. 1. Select the Elastic IP address and choose **Actions**, **Update reverse DNS**. 1. For **Reverse DNS domain name**, clear the domain name. 1. Enter **update** to confirm. 1. Choose **Update**. ------ #### [ AWS CLI ] **To remove a reverse DNS record** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/reset-address-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/reset-address-attribute.html) command. ``` aws ec2 reset-address-attribute \ --allocation-id eipalloc-abcdef01234567890 \ --attribute domain-name ``` The following is example output. ``` { "Addresses": [ { "PublicIp": "192.0.2.0", "AllocationId": "eipalloc-abcdef01234567890", "PtrRecord": "example.com.", "PtrRecordUpdate": { "Value": "example.net.", "Status": "PENDING" } } ] } ``` ------ #### [ PowerShell ] **To remove a reverse DNS record** Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Reset-EC2AddressAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Reset-EC2AddressAttribute.html) cmdlet. ``` Reset-EC2AddressAttribute ` -AllocationId 'eipalloc-abcdef01234567890' ` -Attribute domain-name | Format-List ` AllocationId, PtrRecord, PublicIp, @{Name='PtrRecordUpdate';Expression={$_.PtrRecordUpdate | Format-List | Out-String}} ``` The following is example output. ``` AllocationId : eipalloc-abcdef01234567890 PtrRecord : example.com. PublicIp : 192.0.2.0 PtrRecordUpdate : Reason : Status : PENDING Value : example.net. ``` ------ # Elastic network interfaces Network interfaces An *elastic network interface* is a logical networking component in a VPC that represents a virtual network card. You can create and configure network interfaces and attach them to instances that you launch in the same Availability Zone. The attributes of a network interface follow it as it's attached or detached from an instance and reattached to another instance. When you move a network interface from one instance to another, network traffic is redirected from the original instance to the new instance. Note that this AWS resource is referred to as a *network interface* in the AWS Management Console and the Amazon EC2 API. Therefore, we use "network interface" in this documentation instead of "elastic network interface". The term "network interface" in this documentation always means "elastic network interface". **Network interface attributes** A network interface can include the following attributes: + A primary private IPv4 address from the IPv4 address range of your subnet + A primary IPv6 address from the IPv6 address range of your subnet + Secondary private IPv4 addresses from the IPv4 address range of your subnet + One Elastic IP address (IPv4) for each private IPv4 address + One public IPv4 address + Secondary IPv6 addresses + Security groups + A MAC address + A source/destination check flag + A description **Monitoring traffic** You can enable a VPC flow log on your network interface to capture information about the traffic going to and from a network interface. After you've created a flow log, you can view and retrieve its data in Amazon CloudWatch Logs. For more information, see [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) in the *Amazon VPC User Guide*. **Topics** + [ ## Network interface concepts ](#eni-basics) + [ ## Network cards ](#network-cards) + [ # Maximum IP addresses per network interface ](AvailableIpPerENI.md) + [ # Create a network interface for your EC2 instance ](create-network-interface.md) + [ # Network interface attachments for your EC2 instance ](network-interface-attachments.md) + [ # Manage the IP addresses for your network interface ](managing-network-interface-ip-addresses.md) + [ # Modify network interface attributes ](modify-network-interface-attributes.md) + [ # Multiple network interfaces for your Amazon EC2 instances ](scenarios-enis.md) + [ # Requester-managed network interfaces ](requester-managed-eni.md) + [ # Prefix delegation for Amazon EC2 network interfaces ](ec2-prefix-eni.md) + [ # Delete a network interface ](delete_eni.md) ## Network interface concepts The following are important concepts to understand as you get started using network interfaces. **Primary network interface** Each instance has a default network interface, called the *primary network interface*. You can't detach a primary network interface from an instance. **Secondary network interfaces** You can create and attach secondary network interfaces to your instance. The maximum number of network interfaces varies by instance type. For more information, see [Maximum IP addresses per network interface](AvailableIpPerENI.md). **IPv4 addresses for network interfaces** When you launch an EC2 instance into an IPv4-only or dual stack subnet, the instance receives a primary private IP address from the IPv4 address range of the subnet. You can also specify additional private IPv4 addresses, known as secondary private IPv4 addresses. Unlike primary private IP addresses, secondary private IP addresses can be reassigned from one instance to another. **Public IPv4 addresses for network interfaces** All subnets have a modifiable attribute that determines whether network interfaces created in that subnet (and therefore instances launched into that subnet) are assigned a public IPv4 address. For more information, see [Subnet settings](https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html#subnet-settings) in the *Amazon VPC User Guide*. When you launch an instance, the IP address is assigned to the primary network interface. If you specify an existing network interface as the primary network interface when you launch an instance, the public IPv4 address is determined by this network interface. When you create a network interface, it inherits the public IPv4 addressing attribute from the subnet. If you later modify the public IPv4 addressing attribute of the subnet, the network interface keeps the setting that was in effect when it was created. We release the public IP address when the instance is stopped, hibernated, or terminated. We assign a new public IP address when you start your stopped or hibernated instance, unless it has a secondary network interface or a secondary private IPv4 address that is associated with an Elastic IP address. **IPv6 addresses for network interfaces** If you associate IPv6 CIDR blocks with your VPC and subnet, you can assign IPv6 addresses from the subnet range to a network interface. Each IPv6 address can be assigned to one network interface. All subnets have a modifiable attribute that determines whether network interfaces created in that subnet (and therefore instances launched into that subnet) are automatically assigned an IPv6 address from the range of the subnet. When you launch an instance, the IPv6 address is assigned to the primary network interface. **Elastic IP addresses for network interfaces** You can associate an Elastic IP address with one of the private IPv4 addresses for the network interface. You can associate one Elastic IP address with each private IPv4 address. If you disassociate an Elastic IP address from a network interface, you can release it or associate it with a different instance. **Termination behavior** You can set the termination behavior for a network interface that's attached to an instance. You can specify whether the network interface should be automatically deleted when you terminate the instance to which it's attached. **Source/destination checking** You can enable or disable source/destination checks, which ensure that the instance is either the source or the destination of any traffic that it receives. Source/destination checks are enabled by default. You must disable source/destination checks if the instance runs services such as network address translation, routing, or firewalls. **Requester-managed network interfaces** These network interfaces are created and managed by AWS services to enable you to use some resources and services. You can't manage these network interfaces yourself. For more information, see [Requester-managed network interfaces](requester-managed-eni.md). **Prefix delegation** A prefix is a reserved private IPv4 or IPv6 CIDR range that you allocate for automatic or manual assignment to network interfaces that are associated with an instance. By using Delegated Prefixes, you can launch services faster by assigning a range of IP addresses as a single prefix. **Managed network interfaces** A *managed network interface* is managed by a service provider, such as Amazon EKS Auto Mode. You can’t directly modify the settings of a managed network interface. Managed network interface are identified by a **true** value in the **Managed** field. For more information, see [Amazon EC2 managed instances](amazon-ec2-managed-instances.md). ## Network cards Most instance types support one network card. Instance types that support multiple network cards provide higher network performance, including bandwidth capabilities above 100 Gbps and improved packet rate performance. When you attach a network interface to an instance that supports multiple network cards, you can select the network card for the network interface. The primary network interface must be assigned to network card index 0. EFA and EFA-only network interfaces count as a network interface. You can assign only one EFA or EFA-only network interface per network card. The primary network interface can't be an EFA-only network interface. The following instance types support multiple network cards. For information about the number of network interfaces that an instance type supports, see [Maximum IP addresses per network interface](AvailableIpPerENI.md). | Instance type | Number of network cards | | --- | --- | | c6in.32xlarge | 2 | | c6in.metal | 2 | | c8gb.48xlarge | 2 | | c8gb.metal-48xl | 2 | | c8gn.48xlarge | 2 | | c8gn.metal-48xl | 2 | | dl1.24xlarge | 4 | | g6e.24xlarge | 2 | | g6e.48xlarge | 4 | | g7e.24xlarge | 2 | | g7e.48xlarge | 4 | | hpc6id.32xlarge | 2 | | hpc7a.12xlarge | 2 | | hpc7a.24xlarge | 2 | | hpc7a.48xlarge | 2 | | hpc7a.96xlarge | 2 | | hpc8a.96xlarge | 2 | | m6idn.32xlarge | 2 | | m6idn.metal | 2 | | m6in.32xlarge | 2 | | m6in.metal | 2 | | m8gb.48xlarge | 2 | | m8gb.metal-48xl | 2 | | m8gn.48xlarge | 2 | | m8gn.metal-48xl | 2 | | p4d.24xlarge | 4 | | p4de.24xlarge | 4 | | p5.48xlarge | 32 | | p5e.48xlarge | 32 | | p5en.48xlarge | 16 | | p6-b200.48xlarge | 8 | | p6-b300.48xlarge | 17 | | p6e-gb200.36xlarge | 17 | | r8gb.48xlarge | 2 | | r8gb.metal-48xl | 2 | | r8gn.48xlarge | 2 | | r8gn.metal-48xl | 2 | | r6idn.32xlarge | 2 | | r6idn.metal | 2 | | r6in.32xlarge | 2 | | r6in.metal | 2 | | trn1.32xlarge | 8 | | trn1n.32xlarge | 16 | | trn2.48xlarge | 16 | | trn2u.48xlarge | 16 | | u7in-16tb.224xlarge | 2 | | u7in-24tb.224xlarge | 2 | | u7in-32tb.224xlarge | 2 | | u7inh-32tb.480xlarge | 2 | # Maximum IP addresses per network interface IP addresses per network interface Each instance type supports a maximum number of network interfaces, maximum number of private IPv4 addresses per network interface, and maximum number of IPv6 addresses per network interface. The limit for IPv6 addresses is separate from the limit for private IPv4 addresses per network interface. Note that all instance types support IPv6 addressing except for the following: C1, M1, M2, M3, and T1. **Available network interfaces** The *Amazon EC2 Instance Types Guide* provides the information about the network interfaces available for each instance type. For more information, see the following: + [Network specifications – General purpose](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html#gp_network) + [Network specifications – Compute optimized](https://docs.aws.amazon.com/ec2/latest/instancetypes/co.html#co_network) + [Network specifications – Memory optimized](https://docs.aws.amazon.com/ec2/latest/instancetypes/mo.html#mo_network) + [Network specifications – Storage optimized](https://docs.aws.amazon.com/ec2/latest/instancetypes/so.html#so_network) + [Network specifications – Accelerated computing](https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html#ac_network) + [Network specifications – High-performance computing](https://docs.aws.amazon.com/ec2/latest/instancetypes/hpc.html#hpc_network) + [Network specifications – Previous generation](https://docs.aws.amazon.com/ec2/latest/instancetypes/pg.html#pg_network) ------ #### [ Console ] **To retrieve the maximum network interfaces** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instance Types**. 1. Add a filter to specify the instance type (**Instance type=c5.12xlarge**) or instance family (**Instance family=c5**). 1. (Optional) Click the **Preferences** icon and then turn on **Maximum number of network interfaces**. This column indicates the maximum number of network interfaces for each instance type. 1. (Optional) Select the instance type. On the **Networking** tab, find **Maximum number of network interfaces**. ------ #### [ AWS CLI ] **To retrieve the maximum network interfaces** You can use the [describe-instance-types](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-types.html) command to display information about an instance type, such as its supported network interfaces and IP addresses per interface. The following example displays this information for all C8i instances. ``` { echo -e "InstanceType\tMaximumNetworkInterfaces\tIpv4AddressesPerInterface"; \ aws ec2 describe-instance-types \ --filters "Name=instance-type,Values=c8i.*" \ --query 'InstanceTypes[*].[InstanceType, NetworkInfo.MaximumNetworkInterfaces, NetworkInfo.Ipv4AddressesPerInterface]' \ --output text | sort -k2 -n; } | column -t ``` The following is example output. ``` InstanceType MaximumNetworkInterfaces Ipv4AddressesPerInterface c8i.large 3 20 c8i.2xlarge 4 30 c8i.xlarge 4 30 c8i.4xlarge 8 50 c8i.8xlarge 10 50 c8i.12xlarge 12 50 c8i.16xlarge 16 64 c8i.24xlarge 16 64 c8i.32xlarge 24 64 c8i.48xlarge 24 64 c8i.96xlarge 24 64 c8i.metal-48xl 24 64 c8i.metal-96xl 24 64 ``` ------ #### [ PowerShell ] **To retrieve the maximum network interfaces** You can use the [Get-EC2InstanceType](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceType.html) PowerShell command to display information about an instance type, such as its supported network interfaces and IP addresses per interface. The following example displays this information for all C8i instances. ``` Get-EC2InstanceType -Filter @{Name="instance-type"; Values="c8i.*"} | Select-Object ` InstanceType, @{Name='MaximumNetworkInterfaces'; Expression={$_.NetworkInfo.MaximumNetworkInterfaces}}, @{Name='Ipv4AddressesPerInterface'; Expression={$_.NetworkInfo.Ipv4AddressesPerInterface}} | Sort-Object MaximumNetworkInterfaces | Format-Table -AutoSize ``` The following is example output. ``` InstanceType MaximumNetworkInterfaces Ipv4AddressesPerInterface ------------ ------------------------ ------------------------- c8i.large 3 20 c8i.xlarge 4 30 c8i.2xlarge 4 30 c8i.4xlarge 8 50 c8i.8xlarge 10 50 c8i.12xlarge 12 50 c8i.24xlarge 16 64 c8i.16xlarge 16 64 c8i.96xlarge 24 64 c8i.48xlarge 24 64 c8i.metal-96xl 24 64 c8i.32xlarge 24 64 c8i.metal-48xl 24 64 ``` ------ # Create a network interface for your EC2 instance Create a network interface You can create a network interface for use by your EC2 instances. When you create a network interface, you specify the subnet for which it is created. You can't move a network interface to another subnet after it's created. You must attach a network interface to an instance in the same Availability Zone. You can detach a secondary network interface from an instance and then attach it to a different instance in the same Availability Zone. You can't detach a primary network interface from an instance. For more information, see [Network interface attachments for your EC2 instance](network-interface-attachments.md). ------ #### [ Console ] **To create a network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. Choose **Create network interface**. 1. (Optional) For **Description**, enter a descriptive name. 1. For **Subnet**, select a subnet. The options available in the subsequent steps change depending on the type of subnet you select (IPv4-only, IPv6-only, or dual-stack (IPv4 and IPv6)). 1. For **Interface type**, choose one of the following: + **ENA**: A high-performance network interface designed to handle high throughput and packet-per-second rates for TCP/IP protocols while minimizing CPU usage. This is the default value. For more information about ENA, see [Elastic Network Adapter](enhanced-networking-ena.md). + **EFA with ENA**: A network interface that supports both ENA and EFA devices for traditional TCP/IP based transport along with SRD based transport. If you choose EFA with ENA, the instance you are attaching it to must [support EFA](efa.md#efa-instance-types). For more information about EFA, see [Elastic Fabric Adapter](efa.md). + **EFA-only**: A high-performance network interface designed to handle high throughput, low latency inter-node communication for SRD based transport while bypassing the operating system stack. If you choose this option, the instance you are attaching it to must [support EFA](efa.md#efa-instance-types). EFA-only network interfaces do not support IP addresses. For more information about EFA, see [Elastic Fabric Adapter](efa.md). 1. For **Private IPv4 address**, do one of the following: + Choose **Auto-assign** to allow Amazon EC2 to select an IPv4 address from the subnet. + Choose **Custom** and enter an IPv4 address that you select from the subnet. 1. (Subnets with IPv6 addresses only) For **IPv6 address**, do one of the following: + Choose **None** if you do not want to assign an IPv6 address to the network interface. + Choose **Auto-assign** to allow Amazon EC2 to select an IPv6 address from the subnet. + Choose **Custom** and enter an IPv6 address that you select from the subnet. 1. (Optional) If you’re creating a network interface in a dual-stack or IPv6-only subnet, you have the option to **Assign Primary IPv6 IP**. This assigns a primary IPv6 global unicast address (GUA) to the network interface. Assigning a primary IPv6 address enables you to avoid disrupting traffic to instances or ENIs. Choose **Enable** if the instance that this ENI will be attached to relies on its IPv6 address not changing. AWS will automatically assign an IPv6 address associated with the ENI attached to your instance to be the primary IPv6 address. Once you enable an IPv6 GUA address to be a primary IPv6, you can't disable it. When you enable an IPv6 GUA address to be a primary IPv6, the first IPv6 GUA will be made the primary IPv6 address until the instance is terminated or the network interface is detached. If you have multiple IPv6 addresses associated with an ENI attached to your instance and you enable a primary IPv6 address, the first IPv6 GUA address associated with the ENI becomes the primary IPv6 address. 1. (Optional) To create an Elastic Fabric Adapter, choose **Elastic Fabric Adapter**, **Enable**. 1. (Optional) Under **Advanced settings**, you can optionally set IP prefix delegation. For more information, see [Prefix delegation](ec2-prefix-eni.md). + **Auto-assign** — AWS chooses the prefix from the IPv4 or IPv6 CIDR blocks for the subnet, and assigns it to the network interface. + **Custom** — You specify the prefix from the IPv4 or IPv6 CIDR blocks for the subnet, and AWS verifies that the prefix is not already assigned to other resources before assigning it to the network interface. 1. (Optional) Under **Advanced settings**, for **Idle connection tracking timeout**, modify the default idle connection timeouts. For more information, see [Idle connection tracking timeout](security-group-connection-tracking.md#connection-tracking-timeouts). + **TCP established timeout**: Timeout (in seconds) for idle TCP connections in an established state. + Min: `60` seconds + Max: `432000` seconds + Default: `350` seconds for [Nitrov6](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-nitro-instances.html) instance types, excluding P6e-GB200. And `432000` seconds for other instance types, including P6e-GB200. + Recommended: Less than `432000` seconds + **UDP timeout**: Timeout (in seconds) for idle UDP flows that have seen traffic only in a single direction or a single request-response transaction. + Min: `30` seconds + Max: `60` seconds + Default: `30` seconds + **UDP stream timeout**: Timeout (in seconds) for idle UDP flows classified as streams which have seen more than one request-response transaction. + Min: `60` seconds + Max: `180` seconds + Default: `180` seconds 1. For **Security groups**, select one or more security groups. 1. (Optional) For each tag, choose **Add new tag** and enter a tag key and an optional tag value. 1. Choose **Create network interface**. ------ #### [ AWS CLI ] **Example 1: To create a network interface with IP addresses chosen by Amazon EC2** Use the following [create-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-network-interface.html) command. This example creates a network interface with a public IPv4 address and an IPv6 address chosen by Amazon EC2. ``` aws ec2 create-network-interface \ --subnet-id subnet-0abcdef1234567890 \ --description "my dual-stack network interface" \ --ipv6-address-count 1 \ --groups sg-1234567890abcdef0 ``` **Example 2: To create a network interface with specific IP addresses** Use the following [create-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-network-interface.html) command. ``` aws ec2 create-network-interface \ --subnet-id subnet-0abcdef1234567890 \ --description "my dual-stack network interface" \ --private-ip-address 10.251.50.12 \ --ipv6-addresses 2001:db8::1234:5678:1.2.3.4 \ --groups sg-1234567890abcdef0 ``` **Example 3: To create a network interface with a count of secondary IP addresses** Use the following [create-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-network-interface.html) command. In this example, Amazon EC2 chooses both the primary IP address and the secondary IP addresses. ``` aws ec2 create-network-interface \ --subnet-id subnet-0abcdef1234567890 \ --description "my network interface" \ --secondary-private-ip-address-count 2 \ --groups sg-1234567890abcdef0 ``` **Example 4: To create a network interface with a specific secondary IP address** Use the following [create-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-network-interface.html) command. This example specifies a primary IP address and a secondary IP address. ``` aws ec2 create-network-interface \ --subnet-id subnet-0abcdef1234567890 \ --description "my network interface" \ --private-ip-addresses PrivateIpAddress=10.0.1.30,Primary=true \ PrivateIpAddress=10.0.1.31,Primary=false --groups sg-1234567890abcdef0 ``` ------ #### [ PowerShell ] **Example 1: To create a network interface with IP addresses chosen by Amazon EC2** Use the [New-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2NetworkInterface.html) cmdlet. This example creates a network interface with a public IPv4 address and an IPv6 address chosen by Amazon EC2. ``` New-EC2NetworkInterface ` -SubnetId subnet-0abcdef1234567890 ` -Description "my dual-stack network interface" ` -Ipv6AddresCount 1 ` -Group sg-1234567890abcdef0 ``` **Example 2: To create a network interface with specific IP addresses** Use the [New-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2NetworkInterface.html) cmdlet. ``` New-EC2NetworkInterface ` -SubnetId subnet-0abcdef1234567890 ` -Description "my dual-stack network interface" ` -PrivateIpAddress 10.251.50.12 ` -Ipv6Address $ipv6addr ` -Group sg-1234567890abcdef0 ``` Define the IPv6 addresses as follows. ``` $ipv6addr = New-Object Amazon.EC2.Model.InstanceIpv6Address $ipv6addr1.Ipv6Address = "2001:db8::1234:5678:1.2.3.4" ``` **Example 3: To create a network interface with a count of secondary IP addresses** Use the [New-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2NetworkInterface.html) cmdlet. In this example, Amazon EC2 chooses both the primary IP address and the secondary IP addresses. ``` New-EC2NetworkInterface ` -SubnetId subnet-0abcdef1234567890 ` -Description "my network interface" ` -SecondaryPrivateIpAddressCount 2 ` -Group sg-1234567890abcdef0 ``` **Example 4: To create a network interface with a specific secondary IP address** Use the [New-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2NetworkInterface.html) cmdlet. This example specifies a primary IP address and a secondary IP address. ``` New-EC2NetworkInterface ` -SubnetId subnet-0abcdef1234567890 ` -Description "my network interface" ` -PrivateIpAddresses @($primary, $secondary) ` -Group sg-1234567890abcdef0 ``` Define the secondary addresses as follows. ``` $primary = New-Object Amazon.EC2.Model.PrivateIpAddressSpecification $primary.PrivateIpAddress = "10.0.1.30" $primary.Primary = $true $secondary = New-Object Amazon.EC2.Model.PrivateIpAddressSpecification $secondary.PrivateIpAddress = "10.0.1.31" $secondary.Primary = $false ``` ------ # Network interface attachments for your EC2 instance Network interface attachments You can create network interfaces to be used by your EC2 instances as primary or secondary network interfaces. You must attach a network interface to an EC2 instance if it is in the same Availability Zone as the network interface. The instance type of an instance determines how many network interfaces you can attach to the instance. For more information, see [Maximum IP addresses per network interface](AvailableIpPerENI.md). **Considerations** + You can attach a network interface to an instance when it's running (hot attach), when it's stopped (warm attach), or when the instance is being launched (cold attach). + You can detach secondary network interfaces when the instance is running or stopped. However, you can't detach the primary network interface. + You can detach a secondary network interface from one instance and attach it to another instance. + When launching an instance using the CLI, API, or an SDK, you can specify the primary network interface and additional network interfaces. Note that you can't enable the auto-assignment of public IPv4 addresses if you add a secondary network interface during launch. + Launching an Amazon Linux or Windows Server instance with multiple network interfaces automatically configures interfaces, private IPv4 addresses, and route tables on the operating system of the instance. + A warm or hot attach of an additional network interface might require you to manually bring up the second interface, configure the private IPv4 address, and modify the route table accordingly. Instances running Amazon Linux or Windows Server automatically recognize the warm or hot attach and configure themselves. + You can't attach another network interface to an instance (for example, a NIC teaming configuration) to increase or double the network bandwidth to or from the dual-homed instance. + If you attach multiple network interfaces from the same subnet to an instance, you might encounter networking issues such as asymmetric routing. If possible, add a secondary private IPv4 address on the primary network interface instead. + For EC2 instances in an IPv6-only subnet, if you attach a secondary network interface, the private DNS hostname of the secondary network interface resolves to the primary IPv6 address for the primary network interface. + [Windows instances] – If you add multiple network interfaces to an instance, you must configure the network interfaces to use static routing. ## Attach a network interface You can attach a network interface to any instance in the same Availability Zone as the network interface, using either the **Instances** or **Network Interfaces** page of the Amazon EC2 console. Alternatively, you can specify existing network interfaces when you [launch instances](ec2-launch-instance-wizard.md). If the public IPv4 address on your instance is released, it does not receive a new one if there is more than one network interface attached to the instance. For more information about the behavior of public IPv4 addresses, see [Public IPv4 addresses](using-instance-addressing.md#concepts-public-addresses). ------ #### [ Console ] **To attach a network interface using the Instances page** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the checkbox for the instance. 1. Choose **Actions**, **Networking**, **Attach network interface**. 1. Choose a VPC. The network interface can reside in the same VPC as your instance or in a different VPC that you own, as long as the network interface is in the same Availability Zone as the instance. This enables you to create multi-homed instances across VPCs with different networking and security configurations. 1. Select a network interface. If the instance supports multiple network cards, you can choose a network card. 1. Choose **Attach**. **To attach a network interface using the Network Interfaces page** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. Select the checkbox for the network interface. 1. Choose **Actions**, **Attach**. 1. Choose an instance. If the instance supports multiple network cards, you can choose a network card. 1. Choose **Attach**. ------ #### [ AWS CLI ] **To attach a network interface** Use the following [attach-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/attach-network-interface.html) command. ``` aws ec2 attach-network-interface \ --network-interface-id eni-1234567890abcdef0 \ --instance-id i-1234567890abcdef0 \ --device-index 1 ``` ------ #### [ PowerShell ] **To attach a network interface** Use the [Add-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/Add-EC2NetworkInterface.html) cmdlet. ``` Add-EC2NetworkInterface ` -NetworkInterfaceId eni-1234567890abcdef0 ` -InstanceId i-1234567890abcdef0 ` -DeviceIndex 1 ``` ------ ## Detach a network interface You can detach a secondary network interface that is attached to an EC2 instance at any time, using either the **Instances** or **Network Interfaces** page of the Amazon EC2 console. If you try to detach a network interface that is attached to a resource from another service, such as an Elastic Load Balancing load balancer, a Lambda function, a WorkSpace, or a NAT gateway, you get an error that you do not have permission to access the resource. To find which service created the resource attached to a network interface, check the description of the network interface. If you delete the resource, then its network interface is deleted. ------ #### [ Console ] **To detach a network interface using the Instances page** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the checkbox for the instance. Check the **Network interfaces** section of the **Networking** tab to verify that the network interface is attached to an instance as a secondary network interface. 1. Choose **Actions**, **Networking**, **Detach network interface**. 1. Select the network interface and choose **Detach**. **To detach a network interface using the Network Interfaces page** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. Select the checkbox for the network interface. Check the **Instance details** section of the **Details** tab to verify that the network interface is attached to an instance as a secondary network interface. 1. Choose **Actions**, **Detach**. 1. When prompted for confirmation, choose **Detach**. 1. If the network interface fails to detach from the instance, choose **Force detachment**, **Enable** and then try again. We recommend that force detachment only as a last resort. Forcing a detachment can prevent you from attaching a different network interface on the same index until you restart the instance. It can also prevent the instance metadata from reflecting that the network interface was detached until you restart the instance. ------ #### [ AWS CLI ] **To detach a network interface** Use the following [detach-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/detach-network-interface.html) command. ``` aws ec2 detach-network-interface --attachment-id eni-attach-016c93267131892c9 ``` ------ #### [ PowerShell ] **To detach a network interface** Use the [Dismount-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/Dismount-EC2NetworkInterface.html) cmdlet. ``` Dismount-EC2NetworkInterface -AttachmentId eni-attach-016c93267131892c9 ``` ------ # Manage the IP addresses for your network interface Manage IP addresses You can manage the following IP addresses for your network interfaces: + Elastic IP addresses (one per private IPv4 address) + IPv4 addresses + IPv6 addresses ------ #### [ Console ] **To manage the IP addresses of a network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. Select the checkbox for the network interface. 1. To manage the IPv4 and IPv6 addresses, do the following: 1. Choose **Actions**, **Manage IP addresses**. 1. Expand the network interface. 1. For **IPv4 addresses**, edit the IP addresses as needed. To assign an additional IPv4 address, choose **Assign new IP address** and then specify an IPv4 address from the subnet range or let AWS choose one for you. 1. (Dual stack or IPv6 only) For **IPv6 addresses**, edit the IP addresses as needed. To assign an additional IPv6 address, choose **Assign new IP address** and then specify an IPv6 address from the subnet range or let AWS choose one for you. 1. To assign or unassign a public IPv4 address to a network interface, choose **Auto-assign public IP**. You can enable or disable this for any network interface, but it only affects the primary network interface. 1. (Dual stack or IPv6-only) For **Assign primary IPv6 IP**, choose **Enable** to assign a primary IPv6 address. The first GUA associated with the network interface is chosen as the primary IPv6 address. After you assign a primary IPv6 address, you can't change it. This address is the primary IPv6 address until the instance is terminated or the network interface is detached. 1. Choose **Save**. 1. To associate an Elastic IP address, do the following: 1. Choose **Actions**, **Associate address**. 1. For **Elastic IP address**, select the Elastic IP address. 1. For **Private IPv4 address**, select the private IPv4 address to associate with the Elastic IP address. 1. (Optional) Choose **Allow the Elastic IP address to be reassociated** if the network interface is currently associated with another instance or network interface. 1. Choose **Associate**. 1. To disassociate an Elastic IP address, do the following: 1. Choose **Actions**, **Disassociate address**. 1. For **Public IP address**, select the Elastic IP address. 1. Choose **Disassociate**. ------ #### [ AWS CLI ] **To manage the IPv4 addresses** Use the following [assign-private-ip-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-private-ip-addresses.html) command to assign an IPv4 address. ``` aws ec2 assign-private-ip-addresses \ --network-interface-id eni-1234567890abcdef0 \ --private-ip-addresses 10.0.0.82 ``` Use the following [unassign-private-ip-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/unassign-private-ip-addresses.html) command to unassign an IPv4 address. ``` aws ec2 unassign-private-ip-addresses \ --network-interface-id eni-1234567890abcdef0 \ --private-ip-addresses 10.0.0.82 ``` **To manage the IPv6 addresses** Use the following [assign-ipv6-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-ipv6-addresses.html) command to assign an IPv6 address. ``` aws ec2 assign-ipv6-addresses \ --network-interface-id eni-1234567890abcdef0 \ --ipv6-addresses 2001:db8:1234:1a00:9691:9503:25ad:1761 ``` Use the following [unassign-ipv6-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/unassign-ipv6-addresses.html) command to unassign an IPv6 address. ``` aws ec2 unassign-ipv6-addresses \ --network-interface-id eni-1234567890abcdef0 \ --ipv6-addresses 2001:db8:1234:1a00:9691:9503:25ad:1761 ``` **To manage the Elastic IP address for the primary private IPv4 address** Use the following [associate-address](https://docs.aws.amazon.com/cli/latest/reference/ec2/associate-address.html) command to associate an Elastic IP address with the primary private IPv4 address. ``` aws ec2 associate-address \ --allocation-id eipalloc-0b263919b6EXAMPLE \ --network-interface-id eni-1234567890abcdef0 ``` Use the following [disassociate-address](https://docs.aws.amazon.com/cli/latest/reference/ec2/disassociate-address.html) command to disassociate an Elastic IP address from the primary private IPv4 address. ``` aws ec2 disassociate-address --association-id eipassoc-2bebb745a1EXAMPLE ``` ------ #### [ PowerShell ] **To manage the IPv4 addresses** Use the [Register-EC2PrivateIpAddress](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2PrivateIpAddress.html) cmdlet to assign an IPv4 address. ``` Register-EC2PrivateIpAddress ` -NetworkInterfaceId eni-1234567890abcdef0 ` -PrivateIpAddress 10.0.0.82 ``` Use the [Unregister-EC2PrivateIpAddress](https://docs.aws.amazon.com/powershell/latest/reference/items/Unregister-EC2PrivateIpAddress.html) cmdlet to unassign an IPv4 address. ``` Unregister-EC2PrivateIpAddress ` -NetworkInterfaceId eni-1234567890abcdef0 ` -PrivateIpAddress 10.0.0.82 ``` **To manage the IPv6 addresses** Use the [Register-EC2Ipv6AddressList](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Ipv6AddressList.html) cmdlet to assign an IPv6 address. ``` Register-EC2Ipv6AddressList ` -NetworkInterfaceId eni-1234567890abcdef0 ` -Ipv6Address 2001:db8:1234:1a00:9691:9503:25ad:1761 ``` Use the [Unregister-EC2Ipv6AddressList](https://docs.aws.amazon.com/powershell/latest/reference/items/Unregister-EC2Ipv6AddressList.html) cmdlet to unassign an IPv6 address. ``` Unregister-EC2Ipv6AddressList ` -NetworkInterfaceId eni-1234567890abcdef0 ` -Ipv6Address 2001:db8:1234:1a00:9691:9503:25ad:1761 ``` **To manage the Elastic IP address for the primary private IPv4 address** Use the [Register-EC2Address](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Address.html) cmdlet to associate an Elastic IP address with the primary private IPv4 address. ``` Register-EC2Address ` -NetworkInterfaceId eni-1234567890abcdef0 ` -AllocationId eipalloc-0b263919b6EXAMPLE ``` Use the [Unregister-EC2Address](https://docs.aws.amazon.com/powershell/latest/reference/items/Unregister-EC2Address.html) cmdlet to disassociate an Elastic IP address from the primary private IPv4 address. ``` Unregister-EC2Address -AssociationId eipassoc-2bebb745a1EXAMPLE ``` ------ # Modify network interface attributes You can change the following network interface attributes: + Description + Security groups + Delete on termination + Source/destination check + Idle connection tracking timeout **Considerations** You can't change the attributes of a requester-managed network interface. ------ #### [ Console ] **To modify network interface attributes** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. Select the checkbox for the network interface. 1. To change the description, do the following 1. Choose **Actions**, **Change description**. 1. For **Description**, enter a description. 1. Choose **Save**. 1. To change the security groups, do the following: 1. Choose **Actions**, **Change security groups**. 1. For **Associated security groups**, add and remove security groups as needed. The security group and network interface must be created for the same VPC. 1. Choose **Save**. 1. To change the termination behavior, do the following: 1. Choose **Actions**, **Change termination behavior**. 1. Select or clear **Delete on termination**, **Enable**. 1. Choose **Save**. 1. To change source/destination checking, do the following: 1. Choose **Actions**, **Change source/dest check**. 1. Select or clear **Source/destination check**, **Enable**. 1. Choose **Save**. 1. To change idle connection tracking timeouts, do the following: 1. Choose **Actions**, **Modify idle connection tracking timeout**. 1. Modify timeout values as needed. For more information, see [Idle connection tracking timeout](security-group-connection-tracking.md#connection-tracking-timeouts). + **TCP established timeout**: Timeout (in seconds) for idle TCP connections in an established state. + Min: `60` seconds + Max: `432000` seconds + Default: `350` seconds for [Nitrov6](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-nitro-instances.html) instance types, excluding P6e-GB200. And `432000` seconds for other instance types, including P6e-GB200. + Recommended: Less than `432000` seconds + **UDP timeout**: Timeout (in seconds) for idle UDP flows that have seen traffic only in a single direction or a single request-response transaction. + Min: `30` seconds + Max: `60` seconds + Default: `30` seconds + **UDP stream timeout**: Timeout (in seconds) for idle UDP flows classified as streams which have seen more than one request-response transaction. + Min: `60` seconds + Max: `180` seconds + Default: `180` seconds 1. Choose **Save**. ------ #### [ AWS CLI ] **Example: To modify the description** Use the following [modify-network-interface-attribute](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-network-interface-attribute.html) command. ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --description "my updated description" ``` **Example: To modify the security groups** Use the following [modify-network-interface-attribute](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-network-interface-attribute.html) command. ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --groups sg-1234567890abcdef0 ``` **Example: To modify the termination behavior** Use the following [modify-network-interface-attribute](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-network-interface-attribute.html) command. ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --attachment AttachmentId=eni-attach-43348162abEXAMPLE,DeleteOnTermination=false ``` **Example: To enable source/destination checking** Use the following [modify-network-interface-attribute](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-network-interface-attribute.html) command. ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --source-dest-check ``` **Example: To modify idle connection tracking timout** Use the following [modify-network-interface-attribute](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-network-interface-attribute.html) command. For more information, see [Idle connection tracking timeout](security-group-connection-tracking.md#connection-tracking-timeouts). ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --connection-tracking-specification TcpEstablishedTimeout=172800,UdpStreamTimeout=90,UdpTimeout=60 ``` ------ #### [ PowerShell ] **Example: To modify the description** Use the [Edit-EC2NetworkInterfaceAttribute](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2NetworkInterfaceAttribute.html) cmdlet. ``` Edit-EC2NetworkInterfaceAttribute ` -NetworkInterfaceId eni-1234567890abcdef0 ` -Description "my updated description" ``` **Example: To modify the security groups** Use the [Edit-EC2NetworkInterfaceAttribute](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2NetworkInterfaceAttribute.html) cmdlet. ``` Edit-EC2NetworkInterfaceAttribute ` -NetworkInterfaceId eni-1234567890abcdef0 ` -Group sg-1234567890abcdef0 ``` **Example: To modify the termination behavior** Use the [Edit-EC2NetworkInterfaceAttribute](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2NetworkInterfaceAttribute.html) cmdlet. ``` Edit-EC2NetworkInterfaceAttribute ` -NetworkInterfaceId eni-1234567890abcdef0 ` -Attachment_AttachmentId eni-attach-43348162abEXAMPLE ` -Attachment_DeleteOnTermination $false ``` **Example: To enable source/destination checking** Use the [Edit-EC2NetworkInterfaceAttribute](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2NetworkInterfaceAttribute.html) cmdlet. ``` Edit-EC2NetworkInterfaceAttribute ` -NetworkInterfaceId eni-1234567890abcdef0 ` -SourceDestCheck $true ``` **Example: To modify idle connection tracking timeouts** Use the [Edit-EC2NetworkInterfaceAttribute](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2NetworkInterfaceAttribute.html) cmdlet. For more information, see [Idle connection tracking timeout](security-group-connection-tracking.md#connection-tracking-timeouts). ``` Edit-EC2NetworkInterfaceAttribute ` -NetworkInterfaceId eni-1234567890abcdef0 ` -ConnectionTrackingSpecification_TcpEstablishedTimeout 172800 ` -ConnectionTrackingSpecification_UdpStreamTimeout 90 ` -ConnectionTrackingSpecification_UdpTimeout 60 ``` ------ # Multiple network interfaces for your Amazon EC2 instances Multiple network interfaces Attaching multiple network interfaces to an instance is useful when you need the following: + A [management network](#creating-a-management-network). + [Network and security appliances](#use-network-and-security-appliances-in-your-vpc). + Dual-homed instances with workloads in different [subnets](#creating-dual-homed-instances-with-workloads-roles-on-distinct-subnets) or [VPCs](#creating-dual-homed-instances-with-workloads-roles-on-distinct-subnets). + A [low-budget, high-availability](#create-a-low-budget-high-availability-solution) solution. ## Management network The following overview describes a management network created using multiple network interfaces. **Criteria** + The primary network interface on the instance (for example, eth0) handles public traffic. + The secondary network interface on the instance (for example, eth1) handles backend management traffic. It's connected to a separate subnet that has more restrictive access controls, and is located within the same Availability Zone as the primary network interface. **Settings** + The primary network interface, which may or may not be behind a load balancer, has an associated security group that allows access to the server from the internet. For example, allow TCP port 80 and 443 from 0.0.0.0/0 or from the load balancer. + The secondary network interface has an associated security group that allows SSH access only, initiated from one of the following locations: + An allowed range of IP addresses, either within the VPC, or from the internet. + A private subnet within the same Availability Zone as the primary network interface. + A virtual private gateway. **Note** To ensure failover capabilities, consider using a secondary private IPv4 for incoming traffic on a network interface. In the event of an instance failure, you can move the interface and/or secondary private IPv4 address to a standby instance. ![\[Creating a management network\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/EC2_ENI_management_network.png) ## Network and security appliances Some network and security appliances, such as load balancers, network address translation (NAT) servers, and proxy servers prefer to be configured with multiple network interfaces. You can create and attach secondary network interfaces to instances that are running these types of applications and configure the additional interfaces with their own public and private IP addresses, security groups, and source/destination checking. ## Dual-homed instances with workloads in different subnets You can place a network interface on each of your web servers that connects to a mid-tier network where an application server resides. The application server can also be dual-homed to a backend network (subnet) where the database server resides. Instead of routing network packets through the dual-homed instances, each dual-homed instance receives and processes requests on the front end, initiates a connection to the backend, and then sends requests to the servers on the backend network. ## Dual-homed instances with workloads in different VPCs in the same account You can launch an EC2 instance in one VPC and attach a secondary ENI from a different VPC, as long as the network interface is in the same Availability Zone as the instance. This enables you to create multi-homed instances across VPCs with different networking and security configurations. You can't create multi-homed instances across VPCs in different AWS accounts. You can use dual-homed instances across VPCs in the following use cases: + **Overcome CIDR overlaps between two VPCs that can’t be peered together**: You can leverage a secondary CIDR in a VPC and allow an instance to communicate across two non-overlapping IP ranges. + **Connect multiple VPCs within a single account**: Enable communication between individual resources that would normally be separated by VPC boundaries. ## Low-budget, high-availability solution If one of your instances serving a particular function fails, its network interface can be attached to a replacement or hot standby instance pre-configured for the same role in order to rapidly recover the service. For example, you can use a network interface as your primary or secondary network interface to a critical service such as a database instance or a NAT instance. If the instance fails, you (or more likely, the code running on your behalf) can attach the network interface to a hot standby instance. Because the interface maintains its private IP addresses, Elastic IP addresses, and MAC address, network traffic begins flowing to the standby instance as soon as you attach the network interface to the replacement instance. Users experience a brief loss of connectivity between the time the instance fails and the time that the network interface is attached to the standby instance, but no changes to the route table or your DNS server are required. # Requester-managed network interfaces A requester-managed network interface is a network interface that an AWS service creates in your VPC on your behalf. The network interface is associated with a resource for another service, such as a DB instance from Amazon RDS, a NAT gateway, or an interface VPC endpoint from AWS PrivateLink. **Considerations** + You can view the requester-managed network interfaces in your account. You can add or remove tags, but you can't change other properties of a requester-managed network interface. + You can't detach a requester-managed network interface. + When you delete the resource associated with a requester-managed network interface, the AWS service detaches the network interface and deletes it. If the service detached a network interface but didn't delete it, you can delete the detached network interface. ------ #### [ Console ] **To view requester-managed network interfaces** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network & Security**, **Network Interfaces**. 1. Select the ID of the network interface to open its details page. 1. The following are the key fields that you can use to determine the purpose of the network interface: + **Description**: A description provided by the AWS service that created the interface. For example, "VPC Endpoint Interface vpce 089f2123488812123". + **Requester-managed**: Indicates whether the network interface is managed by AWS. + **Requester ID**: The alias or AWS account ID of the principal or service that created the network interface. If you created the network interface, this is your AWS account ID. Otherwise, another principal or service created it. ------ #### [ AWS CLI ] **To view requester-managed network interfaces** Use the [describe-network-interfaces](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-network-interfaces.html) command as follows. ``` aws ec2 describe-network-interfaces \ --filters "Name=requester-managed,Values=true" \ --query "NetworkInterfaces[*].[Description, InterfaceType]" \ --output table ``` The following is example output that shows the key fields that you can use to determine the purpose of the network interface: `Description` and `InterfaceType`. ``` ------------------------------------------------------------------------------- | DescribeNetworkInterfaces | +---------------------------------------------------+-------------------------+ | VPC Endpoint Interface: vpce-0f00567fa8477a1e6 | interface | | VPC Endpoint Interface vpce-0d8ddce4be80e4474 | interface | | VPC Endpoint Interface vpce-078221a1e27d1ea5b | vpc_endpoint | | Resource Gateway Interface rgw-0bba03f3d56060135 | interface | | VPC Endpoint Interface: vpce-0cc199f605eaeace7 | interface | | VPC Endpoint Interface vpce-019b90d6f16d4f958 | interface | +---------------------------------------------------+-------------------------+ ``` ------ #### [ PowerShell ] **To view requester-managed network interfaces** Use the [Get-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2NetworkInterface.html) cmdlet as follows. ``` Get-EC2NetworkInterface -Filter @{Name="requester-managed"; Values="true"} | Select Description, InterfaceType ``` The following is example output that shows the key fields that you can use to determine the purpose of a network interface: `Description` and `InterfaceType`. ``` Description InterfaceType ----------- ------------- VPC Endpoint Interface: vpce-0f00567fa8477a1e6 interface VPC Endpoint Interface vpce-0d8ddce4be80e4474 interface VPC Endpoint Interface vpce-078221a1e27d1ea5b vpc_endpoint Resource Gateway Interface rgw-0bba03f3d56060135 interface VPC Endpoint Interface: vpce-0cc199f605eaeace7 interface VPC Endpoint Interface vpce-019b90d6f16d4f958 interface ``` ------ # Prefix delegation for Amazon EC2 network interfaces Prefix delegation You can assign a private IPv4 or IPv6 CIDR range, either automatically or manually, to your network interfaces. By assigning prefixes, you scale and simplify the management of applications, including container and networking applications that require multiple IP addresses on an instance. For more information about IPv4 and IPv6 addresses, see [Amazon EC2 instance IP addressing](using-instance-addressing.md). The following assignment options are available: + **Automatic assignment** — AWS chooses the prefix and assigns it to your network interface. If the subnet for the network interface has a subnet CIDR reservation of type `prefix`, we select the prefixes from the subnet CIDR reservation. Otherwise, we select them from the subnet CIDR range. + **Manual assignment** — You specify the prefix and AWS verifies that it is not already assigned to other resources before assigning it to your network interface. Assigning prefixes has the following benefits: + Increased IP addresses on a network interface — When you use a prefix, you assign a block of IP addresses as opposed to individual IP addresses. This increases the number of IP addresses for a network interface. + Simplified VPC management for containers — In container applications, each container requires a unique IP address. Assigning prefixes to your instance simplifies the management of your VPCs, as you can launch and terminate containers without having to call Amazon EC2 APIs for individual IP assignments. **Contents** + [ ## Basics ](#ec2-prefix-basics) + [ ## Considerations ](#prefix-limit) + [Manage prefixes](work-with-prefixes.md) + [ ## Assign prefixes during network interface creation ](work-with-prefixes.md#assign-auto-creation) + [ ## Assign prefixes to an existing network interface ](work-with-prefixes.md#assign-auto-existing) + [ ## Remove prefixes from your network interfaces ](work-with-prefixes.md#unassign-prefix) ## Basics + You can assign a prefix to new or existing network interfaces. + To use prefixes, you assign a prefix to your network interface, attach the network interface to your instance, and then configure your operating system. + When you choose the option to specify a prefix, the prefix must meet the following requirements: + The IPv4 prefix that you can specify is `/28`. + The IPv6 prefix that you can specify is `/80`. + The prefix is in the subnet CIDR of the network interface, and does not overlap with other prefixes or IP addresses assigned to existing resources in the subnet. + You can assign a prefix to the primary or secondary network interface. + You can assign an Elastic IP address to a network interface that has a prefix assigned to it. + You can also assign an Elastic IP address to the IP address part of the assigned prefix. + We resolve the private DNS host name of an instance to the primary private IPv4 address. + We assign each private IPv4 address for a network interface, including those from prefixes, using the following format: + `us-east-1` Region ``` ip-private-ipv4-address.ec2.internal ``` + All other Regions ``` ip-private-ipv4-address.region.compute.internal ``` ## Considerations Take the following into consideration when you use prefixes: + Network interfaces with prefixes are supported with [Nitro-based instances](instance-types.md#instance-hypervisor-type). + Prefixes for network interfaces must use IPv6 addresses or private IPv4 addresses. + The maximum number of IP addresses that you can assign to a network interface depends on the instance type. Each prefix that you assign to a network interface counts as one IP address. For example, a `c5.large` instance has a limit of `10` IPv4 addresses per network interface. Each network interface for this instance has a primary IPv4 address. If a network interface has no secondary IPv4 addresses, you can assign up to 9 prefixes to the network interface. For each additional IPv4 address that you assign to a network interface, you can assign one less prefix to the network interface. For more information, see [Maximum IP addresses per network interface](AvailableIpPerENI.md). + Prefixes are included in source/destination checks. + You must configure your operating system to work with network interfaces with prefixes. Note the following: + Some Amazon Linux AMIs contain additional scripts installed by AWS, known as `ec2-net-utils`. These scripts optionally automate the configuration of your network interfaces. They are for use only on Amazon Linux. + For containers, you can use a Container Network Interface (CNI) for the Kubernetes plug-in, or `dockerd` if you use Docker to manage your containers. # Manage prefixes for your network interfaces Manage prefixes When you assign prefixes to a network interface, you can choose whether to let us automatically assign the prefixes or you can specify custom prefixes. If you let us automatically assign prefixes and the subnet for the network interface has a subnet CIDR reservation of type `prefix`, we select the prefixes from the subnet CIDR reservation. Otherwise, we select them from the subnet CIDR range. **Topics** + [ ## Assign prefixes during network interface creation ](#assign-auto-creation) + [ ## Assign prefixes to an existing network interface ](#assign-auto-existing) + [ ## Remove prefixes from your network interfaces ](#unassign-prefix) ## Assign prefixes during network interface creation You can assign automatic or custom prefixes when you create a network interface. ------ #### [ Console ] **To assign automatic prefixes during network interface creation** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. Choose **Create network interface**. 1. Enter a description for the network interface, select the subnet in which to create the network interface, and configure the private IPv4 and IPv6 addresses. 1. Expand **Advanced settings**. 1. For **IPv4 prefix delegation** do one of the following: + To automatically assign an IPv4 prefix, choose **Auto-assign**. For **Number of IPv4 prefixes**, enter the number of prefixes to assign. + To assign a specific IPv4 prefix, choose **Custom**. Choose **Add new prefix** and enter the prefix. 1. For **IPv6 prefix delegation** do one of the following: + To automatically assign an IPv6 prefix, choose **Auto-assign**. For **Number of IPv6 prefixes**, enter the number of prefixes to assign. + To assign a specific IPv6 prefix, choose **Custom**. Choose **Add new prefix** and enter the prefix. **Note** **IPv6 prefix delegation** appears only if the selected subnet is enabled for IPv6. 1. Select the security groups to associate with the network interface and assign resource tags if needed. 1. Choose **Create network interface**. ------ #### [ AWS CLI ] **To assign automatic IPv4 prefixes during network interface creation** Use the [create-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-network-interface.html) command and set `--ipv4-prefix-count` to the number of IPv4 prefixes for AWS to assign. In the following example, AWS assigns one IPv4 prefix. ``` aws ec2 create-network-interface \ --subnet-id subnet-047cfed18eEXAMPLE \ --description "IPv4 automatic example" \ --ipv4-prefix-count 1 ``` **To assign specific IPv4 prefixes during network interface creation** Use the [create-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-network-interface.html) command and set `--ipv4-prefixes` to the prefixes. AWS selects IPv4 addresses from this range. In the following example, the prefix CIDR is 10.0.0.208/28. ``` aws ec2 create-network-interface \ --subnet-id subnet-047cfed18eEXAMPLE \ --description "IPv4 manual example" \ --ipv4-prefixes Ipv4Prefix=10.0.0.208/28 ``` **To assign automatic IPv6 prefixes during network interface creation** Use the [create-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-network-interface.html) command and set `--ipv6-prefix-count` to the number of IPv6 prefixes for AWS to assign. In the following example, AWS assigns one IPv6 prefix. ``` aws ec2 create-network-interface \ --subnet-id subnet-047cfed18eEXAMPLE \ --description "IPv6 automatic example" \ --ipv6-prefix-count 1 ``` **To assign specific IPv6 prefixes during network interface creation** Use the [create-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-network-interface.html) command and set `--ipv6-prefixes` to the prefixes. AWS selects IPv6 addresses from this range. In the following example, the prefix CIDR is 2600:1f13:fc2:a700:1768::/80. ``` aws ec2 create-network-interface \ --subnet-id subnet-047cfed18eEXAMPLE \ --description "IPv6 manual example" \ --ipv6-prefixes Ipv6Prefix=2600:1f13:fc2:a700:1768::/80 ``` ------ #### [ PowerShell ] **To assign automatic IPv4 prefixes during network interface creation** Use the [New-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2NetworkInterface.html) cmdlet and set `Ipv4PrefixCount` to the number of IPv4 prefixes for AWS to assign. In the following example, AWS assigns one IPv4 prefix. ``` New-EC2NetworkInterface ` -SubnetId 'subnet-047cfed18eEXAMPLE' ` -Description 'IPv4 automatic example' ` -Ipv4PrefixCount 1 ``` **To assign specific IPv4 prefixes during network interface creation** Use the [New-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2NetworkInterface.html) cmdlet and set `Ipv4Prefix` to the prefixes. AWS selects IPv4 addresses from this range. In the following example, the prefix CIDR is 10.0.0.208/28. ``` Import-Module AWS.Tools.EC2 New-EC2NetworkInterface ` -SubnetId 'subnet-047cfed18eEXAMPLE' ` -Description 'IPv4 manual example' ` -Ipv4Prefix (New-Object ` -TypeName Amazon.EC2.Model.Ipv4PrefixSpecificationRequest ` -Property @{Ipv4Prefix = '10.0.0.208/28'}) ``` **To assign automatic IPv6 prefixes during network interface creation** Use the [New-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2NetworkInterface.html) cmdlet and set `Ipv6PrefixCount` to the number of IPv6 prefixes for AWS to assign. In the following example, AWS assigns one IPv6 prefix. ``` New-EC2NetworkInterface ` -SubnetId 'subnet-047cfed18eEXAMPLE' ` -Description 'IPv6 automatic example' ` -Ipv6PrefixCount 1 ``` **To assign specific IPv6 prefixes during network interface creation** Use the [New-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2NetworkInterface.html) cmdlet and set `Ipv6Prefixes` to the prefixes. AWS selects IPv6 addresses from this range. In the following example, the prefix CIDR is 2600:1f13:fc2:a700:1768::/80. ``` Import-Module AWS.Tools.EC2 New-EC2NetworkInterface ` -SubnetId 'subnet-047cfed18eEXAMPLE' ` -Description 'IPv6 manual example' ` -Ipv6Prefix (New-Object ` -TypeName Amazon.EC2.Model.Ipv6PrefixSpecificationRequest ` -Property @{Ipv6Prefix = '2600:1f13:fc2:a700:1768::/80'}) ``` ------ ## Assign prefixes to an existing network interface You can assign automatic or custom prefixes to an existing network interface. ------ #### [ Console ] **To assign automatic prefixes to an existing network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. Select the network interface to which to assign the prefixes, and choose **Actions**, **Manage prefixes**. 1. For **IPv4 prefix delegation** do one of the following: + To automatically assign an IPv4 prefix, choose **Auto-assign**. For **Number of IPv4 prefixes**, enter the number of prefixes to assign. + To assign a specific IPv4 prefix, choose **Custom**. Choose **Add new prefix** and enter the prefix. 1. For **IPv6 prefix delegation** do one of the following: + To automatically assign an IPv6 prefix, choose **Auto-assign**. For **Number of IPv6 prefixes**, enter the number of prefixes to assign. + To assign a specific IPv6 prefix, choose **Custom**. Choose **Add new prefix** and enter the prefix. **Note** **IPv6 prefix delegation** appears only if the selected subnet is enabled for IPv6. 1. Choose **Save**. ------ #### [ AWS CLI ] Use the [assign-ipv6-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-ipv6-addresses.html) command to assign IPv6 prefixes and the [assign-private-ip-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-private-ip-addresses.html) command to assign IPv4 prefixes to existing network interfaces. **To assign automatic IPv4 prefixes to an existing network interface** Use the [assign-private-ip-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-private-ip-addresses.html) command and set `--ipv4-prefix-count` to the number of IPv4 prefixes for AWS to assign. In the following example, AWS assigns one IPv4 prefix. ``` aws ec2 assign-private-ip-addresses \ --network-interface-id eni-081fbb4095EXAMPLE \ --ipv4-prefix-count 1 ``` **To assign specific IPv4 prefixes to an existing network interface** Use the [assign-private-ip-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-private-ip-addresses.html) command and set `--ipv4-prefixes` to the prefix. AWS selects IPv4 addresses from this range. In the following example, the prefix CIDR is 10.0.0.208/28. ``` aws ec2 assign-private-ip-addresses \ --network-interface-id eni-081fbb4095EXAMPLE \ --ipv4-prefixes 10.0.0.208/28 ``` **To assign automatic IPv6 prefixes to an existing network interface** Use the [assign-ipv6-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-ipv6-addresses.html) command and set `--ipv6-prefix-count` to the number of IPv6 prefixes for AWS to assign. In the following example, AWS assigns one IPv6 prefix. ``` aws ec2 assign-ipv6-addresses \ --network-interface-id eni-00d577338cEXAMPLE \ --ipv6-prefix-count 1 ``` **To assign specific IPv6 prefixes to an existing network interface** Use the [assign-ipv6-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/assign-ipv6-addresses.html) command and set `--ipv6-prefixes` to the prefix. AWS selects IPv6 addresses from this range. In the following example, the prefix CIDR is 2600:1f13:fc2:a700:18bb::/80. ``` aws ec2 assign-ipv6-addresses \ --network-interface-id eni-00d577338cEXAMPLE \ --ipv6-prefixes 2600:1f13:fc2:a700:18bb::/80 ``` ------ #### [ PowerShell ] **To assign automatic IPv4 prefixes to an existing network interface** Use the [Register-EC2PrivateIpAddress](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2PrivateIpAddress.html) cmdlet and set `Ipv4PrefixCount` to the number of IPv4 prefixes for AWS to assign. In the following example, AWS assigns one IPv4 prefix. ``` Register-EC2PrivateIpAddress ` -NetworkInterfaceId 'eni-00d577338cEXAMPLE' ` -Ipv4PrefixCount 1 ``` **To assign specific IPv4 prefixes to an existing network interface** Use the [Register-EC2PrivateIpAddress](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2PrivateIpAddress.html) cmdlet and set `Ipv4Prefix` to the prefix. AWS selects IPv4 addresses from this range. In the following example, the prefix CIDR is 10.0.0.208/28. ``` Register-EC2PrivateIpAddress ` -NetworkInterfaceId 'eni-00d577338cEXAMPLE' ` -Ipv4Prefix '10.0.0.208/28' ``` **To assign automatic IPv6 prefixes to an existing network interface** Use the [Register-EC2Ipv6AddressList](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Ipv6AddressList.html) cmdlet and set `Ipv6PrefixCount` to the number of IPv4 prefixes for AWS to assign. In the following example, AWS assigns one IPv6 prefix. ``` Register-EC2Ipv6AddressList ` -NetworkInterfaceId 'eni-00d577338cEXAMPLE' ` -Ipv6PrefixCount 1 ``` **To assign specific IPv6 prefixes to an existing network interface** Use the [Register-EC2Ipv6AddressList](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Ipv6AddressList.html) cmdlet and set `Ipv6Prefix` to the prefix. AWS selects IPv6 addresses from this range. In the following example, the prefix CIDR is 2600:1f13:fc2:a700:18bb::/80. ``` Register-EC2Ipv6AddressList ` -NetworkInterfaceId 'eni-00d577338cEXAMPLE' ` -Ipv6Prefix '2600:1f13:fc2:a700:18bb::/80' ``` ------ ## Remove prefixes from your network interfaces You can remove prefixes from an existing network interface. ------ #### [ Console ] **To remove the prefixes from a network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. Select the network interface. 1. Choose **Actions**, **Manage prefixes**. 1. For **IPv4 prefix delegation**, to remove specific prefixes, choose **Unassign** next to the prefixes to remove. To remove all prefixes, choose **Do not assign**. 1. For **IPv6 prefix delegation**, to remove specific prefixes, choose **Unassign** next to the prefixes to remove. To remove all prefixes, choose **Do not assign**. **Note** **IPv6 prefix delegation** appears only if the selected subnet is enabled for IPv6. 1. Choose **Save**. ------ #### [ AWS CLI ] You can use the [unassign-ipv6-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/unassign-ipv6-addresses.html) command to remove IPv6 prefixes and the [unassign-private-ip-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/unassign-private-ip-addresses.html) commands to remove IPv4 prefixes from your existing network interfaces. **To remove IPv4 prefixes from a network interface** Use the [unassign-private-ip-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/unassign-private-ip-addresses.html) command and set `--ipv4-prefix` to the prefix CIDR to remove. ``` aws ec2 unassign-private-ip-addresses \ --network-interface-id eni-081fbb4095EXAMPLE \ --ipv4-prefixes 10.0.0.176/28 ``` **To remove IPv6 prefixes from a network interface** Use the [unassign-ipv6-addresses](https://docs.aws.amazon.com/cli/latest/reference/ec2/unassign-ipv6-addresses.html) command and set `--ipv6-prefix` to the prefix CIDR to remove. ``` aws ec2 unassign-ipv6-addresses \ --network-interface-id eni-00d577338cEXAMPLE \ --ipv6-prefix 2600:1f13:fc2:a700:18bb::/80 ``` ------ #### [ PowerShell ] **To remove IPv4 prefixes from a network interface** Use the [Unregister-EC2PrivateIpAddress](https://docs.aws.amazon.com/powershell/latest/reference/items/Unregister-EC2PrivateIpAddress.html) cmdlet and set `Ipv4Prefix` to the prefix CIDR to remove. ``` Unregister-EC2PrivateIpAddress ` -NetworkInterfaceId 'eni-00d577338cEXAMPLE' ` -Ipv4Prefix '10.0.0.208/28' ``` **To remove IPv6 prefixes from a network interface** Use the [Unregister-EC2Ipv6AddressList](https://docs.aws.amazon.com/powershell/latest/reference/items/Unregister-EC2Ipv6AddressList.html) cmdlet and set `Ipv6Prefix` to the prefix CIDR to remove. ``` Unregister-EC2Ipv6AddressList ` -NetworkInterfaceId 'eni-00d577338cEXAMPLE' ` -Ipv6Prefix '2600:1f13:fc2:a700:18bb::/80' ``` ------ # Delete a network interface Deleting a network interface releases all attributes associated with the interface and releases any private IP addresses or Elastic IP addresses to be used by another instance. You can't delete a network interface that is in use. First, you must [detach the network interface](network-interface-attachments.md#detach_eni). ------ #### [ Console ] **To delete a network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces**. 1. Select the checkbox for the network interface, and then choose **Actions**, **Delete**. 1. When prompted for confirmation, choose **Delete**. ------ #### [ AWS CLI ] **To delete a network interface** Use the following [delete-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/delete-network-interface.html) command. ``` aws ec2 delete-network-interface --network-interface-id eni-1234567890abcdef0 ``` ------ #### [ PowerShell ] **To delete a network interface** Use the [Remove-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/Remove-EC2NetworkInterface.html) cmdlet. ``` Remove-EC2NetworkInterface -NetworkInterfaceId eni-1234567890abcdef0 ``` ------ # Amazon EC2 instance network bandwidth Network bandwidth Instance bandwidth specifications apply to both inbound and outbound traffic for the instance. For example, if an instance specifies up to 10 Gbps of bandwidth, that means it has up to 10 Gbps of bandwidth for inbound traffic and, simultaneously, up to 10 Gbps for outbound traffic. The network bandwidth that's available to an EC2 instance depends on several factors, as follows. **Multi-flow traffic** Bandwidth for multi-flow traffic is limited to 50% of the available bandwidth for traffic that goes through an internet gateway or a [local gateway](https://docs.aws.amazon.com/outposts/latest/userguide/outposts-local-gateways.html) for instances with 32 or more vCPUs, or 5 Gbps, whichever is larger. For instances with fewer than 32 vCPUs, bandwidth is limited to 5 Gbps. **Single-flow traffic** Bandwidth for single-flow traffic is limited to 5 Gbps when instances are not in the same [cluster placement group](placement-strategies.md#placement-groups-cluster). To reduce latency and increase single-flow bandwidth, try one of the following: + Use a cluster placement group to achieve up to 10 Gbps bandwidth for instances within the same placement group. + Set up multiple paths between two endpoints to achieve higher bandwidth with Multipath TCP (MPTCP). + Configure ENA Express for eligible instances within the same Availability Zone to achieve up to 25 Gbps between those instances. **Note** A single-flow is considered a unique 5-tuple TCP or UDP flow. For other protocols following the IP header, such as `GRE` or `IPsec`, the 3 tuple of source IP, destination IP, and next protocol is used to define a flow. ## Available instance bandwidth The available network bandwidth of an instance depends on the number of vCPUs that it has. For example, an `m5.8xlarge` instance has 32 vCPUs and 10 Gbps network bandwidth, and an `m5.16xlarge` instance has 64 vCPUs and 20 Gbps network bandwidth. However, instances might not achieve this bandwidth; for example, if they exceed network allowances at the instance level, such as packet per second or number of tracked connections. How much of the available bandwidth the traffic can utilize depends on the number of vCPUs and the destination. For example, an `m5.16xlarge` instance has 64 vCPUs, so traffic to another instance in the Region can utilize the full bandwidth available (20 Gbps). However, traffic that goes through an internet gateway or a [local gateway](https://docs.aws.amazon.com/outposts/latest/userguide/outposts-local-gateways.html) can utilize only 50% of the bandwidth available (10 Gbps). Typically, instances with 16 vCPUs or fewer (size `4xlarge` and smaller) are documented as having "up to" a specified bandwidth; for example, "up to 10 Gbps". These instances have a baseline bandwidth. To meet additional demand, they can use a network I/O credit mechanism to burst beyond their baseline bandwidth. Instances can use burst bandwidth for a limited time, typically from 5 to 60 minutes, depending on the instance size. An instance receives the maximum number of network I/O credits at launch. If the instance exhausts its network I/O credits, it returns to its baseline bandwidth. A running instance earns network I/O credits whenever it uses less network bandwidth than its baseline bandwidth. A stopped instance does not earn network I/O credits. Instance burst is on a best effort basis, even when the instance has credits available, as burst bandwidth is a shared resource. There are separate network I/O credit buckets for inbound and outbound traffic. **Base and burst network performance** The *Amazon EC2 Instance Types Guide* describes the network performance for each instance type, plus the baseline network bandwidth available for instances that can use burst bandwidth. For more information, see the following: + [Network specifications – General purpose](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html#gp_network) + [Network specifications – Compute optimized](https://docs.aws.amazon.com/ec2/latest/instancetypes/co.html#co_network) + [Network specifications – Memory optimized](https://docs.aws.amazon.com/ec2/latest/instancetypes/mo.html#mo_network) + [Network specifications – Storage optimized](https://docs.aws.amazon.com/ec2/latest/instancetypes/so.html#so_network) + [Network specifications – Accelerated computing](https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html#ac_network) + [Network specifications – High-performance computing](https://docs.aws.amazon.com/ec2/latest/instancetypes/hpc.html#hpc_network) + [Network specifications – Previous generation](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html#pg_network) Alternatively, you can use a command line tool to get this information. The Amazon EC2 console does not display the baseline network bandwidth of an instance type. ------ #### [ AWS CLI ] You can use the [describe-instance-types](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-types.html) command to display information about an instance type. The following example displays network performance information for all C5 instances. ``` aws ec2 describe-instance-types \ --filters "Name=instance-type,Values=c5.*" \ --query "InstanceTypes[].[InstanceType, NetworkInfo.NetworkPerformance, NetworkInfo.NetworkCards[0].BaselineBandwidthInGbps] | sort_by(@,&[2])" \ --output table ``` The following is example output. If your output is missing the baseline bandwidth, update to the latest version of the AWS CLI. ``` --------------------------------------------- | DescribeInstanceTypes | +--------------+--------------------+-------+ | c5.large | Up to 10 Gigabit | 0.75 | | c5.xlarge | Up to 10 Gigabit | 1.25 | | c5.2xlarge | Up to 10 Gigabit | 2.5 | | c5.4xlarge | Up to 10 Gigabit | 5.0 | | c5.9xlarge | 12 Gigabit | 12.0 | | c5.12xlarge | 12 Gigabit | 12.0 | | c5.18xlarge | 25 Gigabit | 25.0 | | c5.24xlarge | 25 Gigabit | 25.0 | | c5.metal | 25 Gigabit | 25.0 | +--------------+--------------------+-------+ ``` ------ #### [ PowerShell ] You can use the [Get-EC2InstanceType](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceType.html) PowerShell command to display information about an instance type. The following example displays network performance information for all C5 instances. ``` Get-EC2InstanceType -Filter @{Name = "instance-type"; Values = "c5.*" } | ` Select-Object ` InstanceType, @{Name = 'NetworkPerformance'; Expression = {($_.Networkinfo.NetworkCards.NetworkPerformance)}}, @{Name = 'BaselineBandwidthInGbps'; Expression = {($_.Networkinfo.NetworkCards.BaselineBandwidthInGbps)}} | ` Format-Table -AutoSize ``` The following is example output. ``` InstanceType NetworkPerformance BaselineBandwidthInGbps ------------ ------------------ ----------------------- c5.4xlarge Up to 10 Gigabit 5.00 c5.xlarge Up to 10 Gigabit 1.25 c5.12xlarge 12 Gigabit 12.00 c5.9xlarge 12 Gigabit 12.00 c5.24xlarge 25 Gigabit 25.00 c5.metal 25 Gigabit 25.00 c5.2xlarge Up to 10 Gigabit 2.50 c5.large Up to 10 Gigabit 0.75 c5.18xlarge 25 Gigabit 25.00 ``` ------ # EC2 instance bandwidth weighting configuration Bandwidth weighting Some instance types support configurable bandwidth weighting, where you can select baseline bandwidth weighting that favors either network processing or EBS operations. Default settings for baseline bandwidth are determined by your instance type. You can configure the bandwidth weighting during launch, or modify your instance settings with the following weighting preferences: + **default** – This option uses the standard bandwidth configuration for your instance type. + **vpc-1** – This option increases the baseline bandwidth available for networking and decreases the baseline bandwidth for EBS operations. + **ebs-1** – This option increases the baseline bandwidth available for EBS operations, and decreases the baseline bandwidth for networking. ## Bandwidth weighting considerations Considerations The following are some considerations that might affect your bandwidth weighting strategy. + Setting bandwidth weighting preferences only affects bandwidth specifications. The network packets per second (PPS) and EBS input/output operations per second (IOPS) specifications don't change. + The combined bandwidth specification between networking and EBS does not change. When you select a bandwidth weighting configuration, the baseline bandwidth available for the selected option increases, and the baseline bandwidth for the remaining option is reduced by the same absolute amount. For all instances except Flex instances, the available burst bandwidth remains the same for your selected option, and is reduced for the remaining option. For Flex instances up to 4xlarge, burst bandwidth remains unchanged. For Flex instances 8xlarge and larger, burst bandwidth increases by the same amount as the baseline bandwidth. + It's important to understand how changes in bandwidth allocation can affect I/O performance for EBS. For EC2 instances that have `vpc-1` configuration (increased networking bandwidth), you might experience lower IOPS for EBS volumes if you reach the EBS bandwidth limit before you've reached the IOPS limit. This is more noticeable with larger I/O sizes. For example, on an instance type that normally supports 240,000 IOPS with 16 KiB I/O size, if you select `vpc-1` weighting, that might reduce the achievable IOPS due to the adjusted EBS baseline bandwidth limit. When planning your workload, consider your I/O size and patterns. Smaller I/O sizes are less likely to be affected by bandwidth limitations, while larger I/O sizes or sequential workloads might see more impact from bandwidth changes. Always test your specific workload to ensure optimal performance with your chosen configuration. + The networking multi-flow bandwidth specification for traffic that goes through an internet gateway or a local gateway is adjusted to 50% of the baseline bandwidth of the configured option or 5 Gbps, where applicable. For more information, see [Amazon EC2 instance network bandwidth](ec2-instance-network-bandwidth.md). The following example is based on an instance type that has a default baseline bandwidth of 40 Gbps, and a default border bandwidth of 20 Gbps. If you choose `vpc-1` bandwidth weighting for this instance, the weighted baseline bandwidth changes to 50 Gbps, and the border bandwidth changes to 25 Gbps. + This feature is available in all commercial regions, aligned with EC2 instance availablilty and support. + This feature adds no additional cost to your EC2 instance. ## Supported instance types for bandwidth weighting Supported instance types Instance types in the following instance families support configurable bandwidth weighting. + **General purpose:** M8a, M8g, M8gd, M8i, M8id, M8i-flex + **Compute optimized:** C8a, C8g, C8gd, C8i, C8id, C8i-flex + **Memory optimized:** R8a, R8g, R8gd, R8i, R8id, R8i-flex, X8g, X8aedz, X8i ## Check current bandwidth settings To see the current bandwidth settings for your instance, select one of the tabs for instructions. ------ #### [ Console ] **To get the bandwidth setting for an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the instance that you want to check from the list, and navigate to the **Networking** tab. Your current setting is shown in the **Configured bandwidth** field. Amazon EC2 uses default settings for your instance type if the bandwidth is not set to a specific value. ------ #### [ AWS CLI ] **To get the bandwidth setting for an instance** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html) command. ``` aws ec2 describe-instances \ --instance-ids i-1234567890abcdef0 \ --query Reservations[].Instances[].NetworkPerformanceOptions.BandwidthWeighting \ --output text ``` The following is example output. ``` default ``` This example lists all of your instances that have the bandwidth weighting preference set to `vpc-1`, for higher networking bandwidth. ``` aws ec2 describe-instances \ --filters "Name=network-performance-options.bandwidth-weighting,Values=vpc-1" \ --query Reservations[].Instances[].InstanceId \ --output text ``` ------ #### [ PowerShell ] **To get the bandwidth setting for an instance** Use the [Get-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2Instance.html) cmdlet. ``` (Get-EC2Instance ` -InstanceId i-1234567890abcdef0).Instances.NetworkPerformanceOptions.BandwidthWeighting.Value ``` The following is example output. ``` default ``` This example lists all of your instances that have the bandwidth weighting preference set to `vpc-1`, for higher networking bandwidth. ``` (Get-EC2Instance ` -Filter @{Name="network-performance-options.bandwidth-weighting";Values="vpc-1"}).Instances.InstanceId ``` ------ ## Configure bandwidth weighting for your instance Configure bandwidth weighting You can configure bandwidth weighting either at launch or by modifying existing instances from the EC2 console, API/SDKs or CLI. ### Configure bandwidth weighting when you launch an instance To configure bandwidth settings when you launch an instance, select one of the tabs for instructions. You can also specify bandwidth weighting in a launch template. To create a launch template, see [Create an Amazon EC2 launch template](create-launch-template.md). The parameter to set is in the same location as it is for launching an instance directly from the console. Expand the **Advanced details** section, and set the **Instance bandwidth configuration**. To launch an instance with your launch template, see [Launch EC2 instances using a launch template](launch-instances-from-launch-template.md). ------ #### [ Console ] **To launch an instance with configurable bandwidth weighting** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Choose **Launch instances**. This opens the **Launch an instance** dialog. There are several additional ways that you can get to the launch dialog, depending on your preference. For example, you can launch an instance directly from an AMI or from the Amazon EC2 dashboard itself. 1. The Amazon Machine Image (AMI) that you launch from must be based on `Arm` architecture. Many **Quick Start** images support both `x86` and `Arm` architectures, After you choose the operating system for your instance, select the `Arm` option from the **Architecture** list. 1. The instance type must be one of the [Supported instance types](#config-bw-support) for this feature. 1. When you expand the **Advanced details** section, you can scroll down to find the **Instance bandwidth configuration** settings. Select the bandwidth configuration option for your instance. 1. Configure all of the other settings for your instance as you normally would, and choose **Launch instance**. ------ #### [ AWS CLI ] **To launch an instance with configurable bandwidth weighting** Use the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command with the following option to launch instances that are configured for higher network bandwidth weighting. ``` --network-performance-options BandwidthWeighting=vpc-1 ``` Use the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command with the following option to launch instances that are configured for higher EBS bandwidth weighting. ``` --network-performance-options BandwidthWeighting=ebs-1 ``` ------ #### [ PowerShell ] **To launch an instance with configurable bandwidth weighting** Use the [New-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2Instance.html) cmdlet with the following parameter to launch instances that are configured for higher network bandwidth weighting. ``` -NetworkPerformanceOptions_BandwidthWeighting vpc-1 ``` Use the [New-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2Instance.html) cmdlet with the following parameter to launch instances that are configured for higher EBS bandwidth weighting. ``` -NetworkPerformanceOptions_BandwidthWeighting ebs-1 ``` ------ ### Update bandwidth weighting for an existing instance To update bandwidth weighting for an existing instance, your instance must be in the `Stopped` state. ------ #### [ Console ] **To update bandwidth weighting** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the instance that you want to update from the list. 1. Before you change the bandwidth configuration, your instance must be in a `Stopped` state. If your instance is running, select **Stop instance** from the **Instance state** menu. 1. Choose **Manage bandwidth** from the **Actions > Networking** menu. This opens the **Manage bandwidth** dialog. **Note** If your instance type doesn't support configuration for bandwidth weighting, that menu item is disabled. 1. Select the option to update your instance, and choose **Change** to save your settings. ------ #### [ AWS CLI ] **To update bandwidth weighting** Use the [modify-instance-network-performance-options](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-network-performance-options.html) command to configure higher network bandwidth weighting for the specified instance. ``` aws ec2 modify-instance-network-performance-options \ --instance-id i-1234567890abcdef0 \ --bandwidth-weighting=vpc-1 ``` The following example configures higher EBS bandwidth weighting for the specified instance. ``` aws ec2 modify-instance-network-performance-options \ --instance-id i-1234567890abcdef0 \ --bandwidth-weighting=ebs-1 ``` ------ #### [ PowerShell ] **To update bandwidth weighting** Use the [Edit-EC2InstanceNetworkPerformanceOption](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceNetworkPerformanceOption.html) cmdlet to configure higher network bandwidth weighting for the specified instance. ``` Edit-EC2InstanceNetworkPerformanceOption ` -InstanceId i-1234567890abcdef0 ` -BandwidthWeighting vpc-1 ``` The following example configures higher EBS bandwidth weighting for the specified instance. ``` Edit-EC2InstanceNetworkPerformanceOption ` -InstanceId i-1234567890abcdef0 ` -BandwidthWeighting ebs-1 ``` ------ ## Networking performance The following table shows the networking performance, in Gbps, that can be achieved with the `default`, `vpc-1`, and `ebs-1` configurations. | Instance type | **`default`**(Baseline / Burst) | **`vpc-1`**(Baseline / Burst) | **`ebs-1`**(Baseline / Burst) | | --- | --- | --- | --- | | c8a.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.438 / 10.0 | | c8a.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | c8a.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | c8a.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | c8a.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | c8a.8xlarge | 15.0 | 18.75 | 12.5 | | c8a.12xlarge | 22.5 | 28.125 | 18.75 | | c8a.16xlarge | 30.0 | 37.5 | 25.0 | | c8a.24xlarge | 40.0 | 50.0 | 32.5 | | c8a.48xlarge | 75.0 | 93.75 | 60.0 | | c8a.metal-24xl | 40.0 | 50.0 | 32.5 | | c8a.metal-48xl | 75.0 | 93.75 | 60.0 | | c8g.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.441 / 10.0 | | c8g.large | 0.937 / 12.5 | 1.171 / 12.5 | 0.779 / 10.0 | | c8g.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | c8g.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | c8g.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | c8g.8xlarge | 15.0 | 18.75 | 12.5 | | c8g.12xlarge | 22.5 | 28.125 | 18.75 | | c8g.16xlarge | 30.0 | 37.5 | 25.0 | | c8g.24xlarge | 40.0 | 50.0 | 32.5 | | c8g.48xlarge | 50.0 | 62.5 | 40.0 | | c8g.metal-24xl | 40.0 | 50.0 | 32.5 | | c8g.metal-48xl | 50.0 | 62.5 | 40.0 | | c8gd.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.441 / 10.0 | | c8gd.large | 0.937 / 12.5 | 1.171 / 12.5 | 0.779 / 10.0 | | c8gd.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | c8gd.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | c8gd.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | c8gd.8xlarge | 15.0 | 18.75 | 12.5 | | c8gd.12xlarge | 22.5 | 28.125 | 18.75 | | c8gd.16xlarge | 30.0 | 37.5 | 25.0 | | c8gd.24xlarge | 40.0 | 50.0 | 32.5 | | c8gd.48xlarge | 50.0 | 62.5 | 40.0 | | c8gd.metal-24xl | 40.0 | 50.0 | 32.5 | | c8gd.metal-48xl | 50.0 | 62.5 | 40.0 | | c8i.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | c8i.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | c8i.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | c8i.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | c8i.8xlarge | 15.0 | 18.75 | 12.5 | | c8i.12xlarge | 22.5 | 28.125 | 18.75 | | c8i.16xlarge | 30.0 | 37.5 | 25.0 | | c8i.24xlarge | 40.0 | 50.0 | 32.5 | | c8i.32xlarge | 50.0 | 62.5 | 40.0 | | c8i.48xlarge | 75.0 | 93.75 | 60.0 | | c8i.96xlarge | 100.0 | 125.0 | 80.0 | | c8i.metal-48xl | 75.0 | 93.75 | 60.0 | | c8i.metal-96xl | 100.0 | 125.0 | 80.0 | | c8id.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | c8id.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | c8id.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | c8id.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | c8id.8xlarge | 15.0 | 18.75 | 12.5 | | c8id.12xlarge | 22.5 | 28.125 | 18.75 | | c8id.16xlarge | 30.0 | 37.5 | 25.0 | | c8id.24xlarge | 40.0 | 50.0 | 32.5 | | c8id.32xlarge | 50.0 | 62.5 | 40.0 | | c8id.48xlarge | 75.0 | 93.75 | 60.0 | | c8id.96xlarge | 100.0 | 125.0 | 80.0 | | c8id.metal-48xl | 75.0 | 93.75 | 60.0 | | c8id.metal-96xl | 100.0 | 125.0 | 80.0 | | c8i-flex.large | 0.468 / 12.5 | 0.585 / 12.5 | 0.389 / 10.0 | | c8i-flex.xlarge | 0.937 / 12.5 | 1.172 / 12.5 | 0.779 / 10.0 | | c8i-flex.2xlarge | 1.875 / 15.0 | 2.344 / 15.0 | 1.562 / 12.5 | | c8i-flex.4xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | c8i-flex.8xlarge | 7.5 / 15.0 | 9.375 / 18.75 | 6.25 / 12.5 | | c8i-flex.12xlarge | 11.25 / 22.5 | 14.063 / 28.125 | 9.375 / 18.75 | | c8i-flex.16xlarge | 15.0 / 30.0 | 18.75 / 37.5 | 12.5 / 25.0 | | m8a.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.438 / 10.0 | | m8a.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | m8a.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | m8a.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | m8a.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | m8a.8xlarge | 15.0 | 18.75 | 12.5 | | m8a.12xlarge | 22.5 | 28.125 | 18.75 | | m8a.16xlarge | 30.0 | 37.5 | 25.0 | | m8a.24xlarge | 40.0 | 50.0 | 32.5 | | m8a.48xlarge | 75.0 | 93.75 | 60.0 | | m8a.metal-24xl | 40.0 | 50.0 | 32.5 | | m8a.metal-48xl | 75.0 | 93.75 | 60.0 | | m8g.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.441 / 10.0 | | m8g.large | 0.937 / 12.5 | 1.171 / 12.5 | 0.779 / 10.0 | | m8g.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | m8g.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | m8g.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | m8g.8xlarge | 15.0 | 18.75 | 12.5 | | m8g.12xlarge | 22.5 | 28.125 | 18.75 | | m8g.16xlarge | 30.0 | 37.5 | 25.0 | | m8g.24xlarge | 40.0 | 50.0 | 32.5 | | m8g.48xlarge | 50.0 | 62.5 | 40.0 | | m8g.metal-24xl | 40.0 | 50.0 | 32.5 | | m8g.metal-48xl | 50.0 | 62.5 | 40.0 | | m8gd.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.441 / 10.0 | | m8gd.large | 0.937 / 12.5 | 1.171 / 12.5 | 0.779 / 10.0 | | m8gd.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | m8gd.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | m8gd.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | m8gd.8xlarge | 15.0 | 18.75 | 12.5 | | m8gd.12xlarge | 22.5 | 28.125 | 18.75 | | m8gd.16xlarge | 30.0 | 37.5 | 25.0 | | m8gd.24xlarge | 40.0 | 50.0 | 32.5 | | m8gd.48xlarge | 50.0 | 62.5 | 40.0 | | m8gd.metal-24xl | 40.0 | 50.0 | 32.5 | | m8gd.metal-48xl | 50.0 | 62.5 | 40.0 | | m8i.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | m8i.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | m8i.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | m8i.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | m8i.8xlarge | 15.0 | 18.75 | 12.5 | | m8i.12xlarge | 22.5 | 28.125 | 18.75 | | m8i.16xlarge | 30.0 | 37.5 | 25.0 | | m8i.24xlarge | 40.0 | 50.0 | 32.5 | | m8i.32xlarge | 50.0 | 62.5 | 40.0 | | m8i.48xlarge | 75.0 | 93.75 | 60.0 | | m8i.96xlarge | 100.0 | 125.0 | 80.0 | | m8i.metal-48xl | 75.0 | 93.75 | 60.0 | | m8i.metal-96xl | 100.0 | 125.0 | 80.0 | | m8id.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | m8id.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | m8id.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | m8id.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | m8id.8xlarge | 15.0 | 18.75 | 12.5 | | m8id.12xlarge | 22.5 | 28.125 | 18.75 | | m8id.16xlarge | 30.0 | 37.5 | 25.0 | | m8id.24xlarge | 40.0 | 50.0 | 32.5 | | m8id.32xlarge | 50.0 | 62.5 | 40.0 | | m8id.48xlarge | 75.0 | 93.75 | 60.0 | | m8id.96xlarge | 100.0 | 125.0 | 80.0 | | m8id.metal-48xl | 75.0 | 93.75 | 60.0 | | m8id.metal-96xl | 100.0 | 125.0 | 80.0 | | m8i-flex.large | 0.468 / 12.5 | 0.585 / 12.5 | 0.389 / 10.0 | | m8i-flex.xlarge | 0.937 / 12.5 | 1.172 / 12.5 | 0.779 / 10.0 | | m8i-flex.2xlarge | 1.875 / 15.0 | 2.344 / 15.0 | 1.562 / 12.5 | | m8i-flex.4xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | m8i-flex.8xlarge | 7.5 / 15.0 | 9.375 / 18.75 | 6.25 / 12.5 | | m8i-flex.12xlarge | 11.25 / 22.5 | 14.063 / 28.125 | 9.375 / 18.75 | | m8i-flex.16xlarge | 15.0 / 30.0 | 18.75 / 37.5 | 12.5 / 25.0 | | r8a.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.438 / 10.0 | | r8a.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | r8a.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | r8a.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | r8a.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | r8a.8xlarge | 15.0 | 18.75 | 12.5 | | r8a.12xlarge | 22.5 | 28.125 | 18.75 | | r8a.16xlarge | 30.0 | 37.5 | 25.0 | | r8a.24xlarge | 40.0 | 50.0 | 32.5 | | r8a.48xlarge | 75.0 | 93.75 | 60.0 | | r8a.metal-24xl | 40.0 | 50.0 | 32.5 | | r8a.metal-48xl | 75.0 | 93.75 | 60.0 | | r8g.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.441 / 10.0 | | r8g.large | 0.937 / 12.5 | 1.171 / 12.5 | 0.779 / 10.0 | | r8g.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | r8g.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | r8g.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | r8g.8xlarge | 15.0 | 18.75 | 12.5 | | r8g.12xlarge | 22.5 | 28.125 | 18.75 | | r8g.16xlarge | 30.0 | 37.5 | 25.0 | | r8g.24xlarge | 40.0 | 50.0 | 32.5 | | r8g.48xlarge | 50.0 | 62.5 | 40.0 | | r8g.metal-24xl | 40.0 | 50.0 | 32.5 | | r8g.metal-48xl | 50.0 | 62.5 | 40.0 | | r8gd.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.441 / 10.0 | | r8gd.large | 0.937 / 12.5 | 1.171 / 12.5 | 0.779 / 10.0 | | r8gd.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | r8gd.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | r8gd.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | r8gd.8xlarge | 15.0 | 18.75 | 12.5 | | r8gd.12xlarge | 22.5 | 28.125 | 18.75 | | r8gd.16xlarge | 30.0 | 37.5 | 25.0 | | r8gd.24xlarge | 40.0 | 50.0 | 32.5 | | r8gd.48xlarge | 50.0 | 62.5 | 40.0 | | r8gd.metal-24xl | 40.0 | 50.0 | 32.5 | | r8gd.metal-48xl | 50.0 | 62.5 | 40.0 | | r8i.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | r8i.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | r8i.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | r8i.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | r8i.8xlarge | 15.0 | 18.75 | 12.5 | | r8i.12xlarge | 22.5 | 28.125 | 18.75 | | r8i.16xlarge | 30.0 | 37.5 | 25.0 | | r8i.24xlarge | 40.0 | 50.0 | 32.5 | | r8i.32xlarge | 50.0 | 62.5 | 40.0 | | r8i.48xlarge | 75.0 | 93.75 | 60.0 | | r8i.96xlarge | 100.0 | 125.0 | 80.0 | | r8i.metal-48xl | 75.0 | 93.75 | 60.0 | | r8i.metal-96xl | 100.0 | 125.0 | 80.0 | | r8id.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | r8id.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | r8id.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | r8id.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | r8id.8xlarge | 15.0 | 18.75 | 12.5 | | r8id.12xlarge | 22.5 | 28.125 | 18.75 | | r8id.16xlarge | 30.0 | 37.5 | 25.0 | | r8id.24xlarge | 40.0 | 50.0 | 32.5 | | r8id.32xlarge | 50.0 | 62.5 | 40.0 | | r8id.48xlarge | 75.0 | 93.75 | 60.0 | | r8id.96xlarge | 100.0 | 125.0 | 80.0 | | r8id.metal-48xl | 75.0 | 93.75 | 60.0 | | r8id.metal-96xl | 100.0 | 125.0 | 80.0 | | r8i-flex.large | 0.468 / 12.5 | 0.585 / 12.5 | 0.389 / 10.0 | | r8i-flex.xlarge | 0.937 / 12.5 | 1.172 / 12.5 | 0.779 / 10.0 | | r8i-flex.2xlarge | 1.875 / 15.0 | 2.344 / 15.0 | 1.562 / 12.5 | | r8i-flex.4xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | r8i-flex.8xlarge | 7.5 / 15.0 | 9.375 / 18.75 | 6.25 / 12.5 | | r8i-flex.12xlarge | 11.25 / 22.5 | 14.063 / 28.125 | 9.375 / 18.75 | | r8i-flex.16xlarge | 15.0 / 30.0 | 18.75 / 37.5 | 12.5 / 25.0 | | x8g.medium | 0.52 / 12.5 | 0.65 / 12.5 | 0.441 / 10.0 | | x8g.large | 0.937 / 12.5 | 1.171 / 12.5 | 0.779 / 10.0 | | x8g.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | x8g.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | x8g.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | x8g.8xlarge | 15.0 | 18.75 | 12.5 | | x8g.12xlarge | 22.5 | 28.125 | 18.75 | | x8g.16xlarge | 30.0 | 37.5 | 25.0 | | x8g.24xlarge | 40.0 | 50.0 | 32.5 | | x8g.48xlarge | 50.0 | 62.5 | 40.0 | | x8g.metal-24xl | 40.0 | 50.0 | 32.5 | | x8g.metal-48xl | 50.0 | 62.5 | 40.0 | | x8aedz.large | 1.562 / 18.75 | 1.953 / 18.75 | 1.249 / 15.0 | | x8aedz.xlarge | 3.125 / 18.75 | 3.907 / 18.75 | 2.5 / 15.0 | | x8aedz.3xlarge | 9.375 / 18.75 | 11.719 / 18.75 | 7.5 / 15.0 | | x8aedz.6xlarge | 18.75 | 23.438 | 15.0 | | x8aedz.12xlarge | 37.5 | 46.875 | 30.0 | | x8aedz.24xlarge | 75.0 | 93.75 | 60.0 | | x8aedz.metal-12xl | 37.5 | 46.875 | 30.0 | | x8aedz.metal-24xl | 75.0 | 93.75 | 60.0 | | x8i.large | 0.937 / 12.5 | 1.172 / 12.5 | 0.774 / 10.0 | | x8i.xlarge | 1.875 / 12.5 | 2.344 / 12.5 | 1.562 / 10.0 | | x8i.2xlarge | 3.75 / 15.0 | 4.688 / 15.0 | 3.125 / 12.5 | | x8i.4xlarge | 7.5 / 15.0 | 9.375 / 15.0 | 6.25 / 12.5 | | x8i.8xlarge | 15.0 | 18.75 | 12.5 | | x8i.12xlarge | 22.5 | 28.125 | 18.75 | | x8i.16xlarge | 30.0 | 37.5 | 25.0 | | x8i.24xlarge | 40.0 | 50.0 | 32.5 | | x8i.32xlarge | 50.0 | 62.5 | 40.0 | | x8i.48xlarge | 75.0 | 93.75 | 60.0 | | x8i.64xlarge | 80.0 | 100.0 | 62.5 | | x8i.96xlarge | 100.0 | 125.0 | 80.0 | | x8i.metal-48xl | 75.0 | 93.75 | 60.0 | | x8i.metal-96xl | 100.0 | 125.0 | 80.0 | ## Amazon EBS performance The following table shows the Amazon EBS performance, in Gbps, that can be achieved with the `default`, `vpc-1`, and `ebs-1` configurations. | Instance type | **`default`**(Baseline / Burst) | **`vpc-1`**(Baseline / Burst) | **`ebs-1`**(Baseline / Burst) | | --- | --- | --- | --- | | c8a.medium | 0.325 / 10.0 | 0.195 / 6.25 | 0.407 / 10.0 | | c8a.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | c8a.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | c8a.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | c8a.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | c8a.8xlarge | 10.0 | 6.25 | 12.5 | | c8a.12xlarge | 15.0 | 9.375 | 18.75 | | c8a.16xlarge | 20.0 | 12.5 | 25.0 | | c8a.24xlarge | 30.0 | 20.0 | 37.5 | | c8a.48xlarge | 60.0 | 41.25 | 75.0 | | c8a.metal-24xl | 30.0 | 20.0 | 37.5 | | c8a.metal-48xl | 60.0 | 41.25 | 75.0 | | c8g.medium | 0.315 / 10.0 | 0.185 / 6.25 | 0.394 / 10.0 | | c8g.large | 0.63 / 10.0 | 0.396 / 6.25 | 0.788 / 10.0 | | c8g.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | c8g.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | c8g.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | c8g.8xlarge | 10.0 | 6.25 | 12.5 | | c8g.12xlarge | 15.0 | 9.375 | 18.75 | | c8g.16xlarge | 20.0 | 12.5 | 25.0 | | c8g.24xlarge | 30.0 | 20.0 | 37.5 | | c8g.48xlarge | 40.0 | 27.5 | 50.0 | | c8g.metal-24xl | 30.0 | 20.0 | 37.5 | | c8g.metal-48xl | 40.0 | 27.5 | 50.0 | | c8gd.medium | 0.315 / 10.0 | 0.185 / 6.25 | 0.394 / 10.0 | | c8gd.large | 0.63 / 10.0 | 0.396 / 6.25 | 0.788 / 10.0 | | c8gd.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | c8gd.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | c8gd.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | c8gd.8xlarge | 10.0 | 6.25 | 12.5 | | c8gd.12xlarge | 15.0 | 9.375 | 18.75 | | c8gd.16xlarge | 20.0 | 12.5 | 25.0 | | c8gd.24xlarge | 30.0 | 20.0 | 37.5 | | c8gd.48xlarge | 40.0 | 27.5 | 50.0 | | c8gd.metal-24xl | 30.0 | 20.0 | 37.5 | | c8gd.metal-48xl | 40.0 | 27.5 | 50.0 | | c8i.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | c8i.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | c8i.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | c8i.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | c8i.8xlarge | 10.0 | 6.25 | 12.5 | | c8i.12xlarge | 15.0 | 9.375 | 18.75 | | c8i.16xlarge | 20.0 | 12.5 | 25.0 | | c8i.24xlarge | 30.0 | 20.0 | 37.5 | | c8i.32xlarge | 40.0 | 27.5 | 50.0 | | c8i.48xlarge | 60.0 | 41.25 | 75.0 | | c8i.96xlarge | 80.0 | 55.0 | 100.0 | | c8i.metal-48xl | 60.0 | 41.25 | 75.0 | | c8i.metal-96xl | 80.0 | 55.0 | 100.0 | | c8id.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | c8id.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | c8id.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | c8id.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | c8id.8xlarge | 10.0 | 6.25 | 12.5 | | c8id.12xlarge | 15.0 | 9.375 | 18.75 | | c8id.16xlarge | 20.0 | 12.5 | 25.0 | | c8id.24xlarge | 30.0 | 20.0 | 37.5 | | c8id.32xlarge | 40.0 | 27.5 | 50.0 | | c8id.48xlarge | 60.0 | 41.25 | 75.0 | | c8id.96xlarge | 80.0 | 55.0 | 100.0 | | c8id.metal-48xl | 60.0 | 41.25 | 75.0 | | c8id.metal-96xl | 80.0 | 55.0 | 100.0 | | c8i-flex.large | 0.315 / 10.0 | 0.198 / 6.25 | 0.394 / 10.0 | | c8i-flex.xlarge | 0.63 / 10.0 | 0.395 / 6.25 | 0.788 / 10.0 | | c8i-flex.2xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | c8i-flex.4xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | c8i-flex.8xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 12.5 | | c8i-flex.12xlarge | 7.5 / 15.0 | 4.687 / 9.375 | 9.375 / 18.75 | | c8i-flex.16xlarge | 10.0 / 20.0 | 6.25 / 12.5 | 12.5 / 25.0 | | m8a.medium | 0.325 / 10.0 | 0.195 / 6.25 | 0.407 / 10.0 | | m8a.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | m8a.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | m8a.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | m8a.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | m8a.8xlarge | 10.0 | 6.25 | 12.5 | | m8a.12xlarge | 15.0 | 9.375 | 18.75 | | m8a.16xlarge | 20.0 | 12.5 | 25.0 | | m8a.24xlarge | 30.0 | 20.0 | 37.5 | | m8a.48xlarge | 60.0 | 41.25 | 75.0 | | m8a.metal-24xl | 30.0 | 20.0 | 37.5 | | m8a.metal-48xl | 60.0 | 41.25 | 75.0 | | m8g.medium | 0.315 / 10.0 | 0.185 / 6.25 | 0.394 / 10.0 | | m8g.large | 0.63 / 10.0 | 0.396 / 6.25 | 0.788 / 10.0 | | m8g.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | m8g.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | m8g.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | m8g.8xlarge | 10.0 | 6.25 | 12.5 | | m8g.12xlarge | 15.0 | 9.375 | 18.75 | | m8g.16xlarge | 20.0 | 12.5 | 25.0 | | m8g.24xlarge | 30.0 | 20.0 | 37.5 | | m8g.48xlarge | 40.0 | 27.5 | 50.0 | | m8g.metal-24xl | 30.0 | 20.0 | 37.5 | | m8g.metal-48xl | 40.0 | 27.5 | 50.0 | | m8gd.medium | 0.315 / 10.0 | 0.185 / 6.25 | 0.394 / 10.0 | | m8gd.large | 0.63 / 10.0 | 0.396 / 6.25 | 0.788 / 10.0 | | m8gd.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | m8gd.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | m8gd.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | m8gd.8xlarge | 10.0 | 6.25 | 12.5 | | m8gd.12xlarge | 15.0 | 9.375 | 18.75 | | m8gd.16xlarge | 20.0 | 12.5 | 25.0 | | m8gd.24xlarge | 30.0 | 20.0 | 37.5 | | m8gd.48xlarge | 40.0 | 27.5 | 50.0 | | m8gd.metal-24xl | 30.0 | 20.0 | 37.5 | | m8gd.metal-48xl | 40.0 | 27.5 | 50.0 | | m8i.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | m8i.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | m8i.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | m8i.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | m8i.8xlarge | 10.0 | 6.25 | 12.5 | | m8i.12xlarge | 15.0 | 9.375 | 18.75 | | m8i.16xlarge | 20.0 | 12.5 | 25.0 | | m8i.24xlarge | 30.0 | 20.0 | 37.5 | | m8i.32xlarge | 40.0 | 27.5 | 50.0 | | m8i.48xlarge | 60.0 | 41.25 | 75.0 | | m8i.96xlarge | 80.0 | 55.0 | 100.0 | | m8i.metal-48xl | 60.0 | 41.25 | 75.0 | | m8i.metal-96xl | 80.0 | 55.0 | 100.0 | | m8id.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | m8id.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | m8id.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | m8id.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | m8id.8xlarge | 10.0 | 6.25 | 12.5 | | m8id.12xlarge | 15.0 | 9.375 | 18.75 | | m8id.16xlarge | 20.0 | 12.5 | 25.0 | | m8id.24xlarge | 30.0 | 20.0 | 37.5 | | m8id.32xlarge | 40.0 | 27.5 | 50.0 | | m8id.48xlarge | 60.0 | 41.25 | 75.0 | | m8id.96xlarge | 80.0 | 55.0 | 100.0 | | m8id.metal-48xl | 60.0 | 41.25 | 75.0 | | m8id.metal-96xl | 80.0 | 55.0 | 100.0 | | m8i-flex.large | 0.315 / 10.0 | 0.198 / 6.25 | 0.394 / 10.0 | | m8i-flex.xlarge | 0.63 / 10.0 | 0.395 / 6.25 | 0.788 / 10.0 | | m8i-flex.2xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | m8i-flex.4xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | m8i-flex.8xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 12.5 | | m8i-flex.12xlarge | 7.5 / 15.0 | 4.687 / 9.375 | 9.375 / 18.75 | | m8i-flex.16xlarge | 10.0 / 20.0 | 6.25 / 12.5 | 12.5 / 25.0 | | r8a.medium | 0.325 / 10.0 | 0.195 / 6.25 | 0.407 / 10.0 | | r8a.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | r8a.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | r8a.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | r8a.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | r8a.8xlarge | 10.0 | 6.25 | 12.5 | | r8a.12xlarge | 15.0 | 9.375 | 18.75 | | r8a.16xlarge | 20.0 | 12.5 | 25.0 | | r8a.24xlarge | 30.0 | 20.0 | 37.5 | | r8a.48xlarge | 60.0 | 41.25 | 75.0 | | r8a.metal-24xl | 30.0 | 20.0 | 37.5 | | r8a.metal-48xl | 60.0 | 41.25 | 75.0 | | r8g.medium | 0.315 / 10.0 | 0.185 / 6.25 | 0.394 / 10.0 | | r8g.large | 0.63 / 10.0 | 0.396 / 6.25 | 0.788 / 10.0 | | r8g.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | r8g.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | r8g.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | r8g.8xlarge | 10.0 | 6.25 | 12.5 | | r8g.12xlarge | 15.0 | 9.375 | 18.75 | | r8g.16xlarge | 20.0 | 12.5 | 25.0 | | r8g.24xlarge | 30.0 | 20.0 | 37.5 | | r8g.48xlarge | 40.0 | 27.5 | 50.0 | | r8g.metal-24xl | 30.0 | 20.0 | 37.5 | | r8g.metal-48xl | 40.0 | 27.5 | 50.0 | | r8gd.medium | 0.315 / 10.0 | 0.185 / 6.25 | 0.394 / 10.0 | | r8gd.large | 0.63 / 10.0 | 0.396 / 6.25 | 0.788 / 10.0 | | r8gd.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | r8gd.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | r8gd.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | r8gd.8xlarge | 10.0 | 6.25 | 12.5 | | r8gd.12xlarge | 15.0 | 9.375 | 18.75 | | r8gd.16xlarge | 20.0 | 12.5 | 25.0 | | r8gd.24xlarge | 30.0 | 20.0 | 37.5 | | r8gd.48xlarge | 40.0 | 27.5 | 50.0 | | r8gd.metal-24xl | 30.0 | 20.0 | 37.5 | | r8gd.metal-48xl | 40.0 | 27.5 | 50.0 | | r8i.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | r8i.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | r8i.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | r8i.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | r8i.8xlarge | 10.0 | 6.25 | 12.5 | | r8i.12xlarge | 15.0 | 9.375 | 18.75 | | r8i.16xlarge | 20.0 | 12.5 | 25.0 | | r8i.24xlarge | 30.0 | 20.0 | 37.5 | | r8i.32xlarge | 40.0 | 27.5 | 50.0 | | r8i.48xlarge | 60.0 | 41.25 | 75.0 | | r8i.96xlarge | 80.0 | 55.0 | 100.0 | | r8i.metal-48xl | 60.0 | 41.25 | 75.0 | | r8i.metal-96xl | 80.0 | 55.0 | 100.0 | | r8id.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | r8id.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | r8id.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | r8id.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | r8id.8xlarge | 10.0 | 6.25 | 12.5 | | r8id.12xlarge | 15.0 | 9.375 | 18.75 | | r8id.16xlarge | 20.0 | 12.5 | 25.0 | | r8id.24xlarge | 30.0 | 20.0 | 37.5 | | r8id.32xlarge | 40.0 | 27.5 | 50.0 | | r8id.48xlarge | 60.0 | 41.25 | 75.0 | | r8id.96xlarge | 80.0 | 55.0 | 100.0 | | r8id.metal-48xl | 60.0 | 41.25 | 75.0 | | r8id.metal-96xl | 80.0 | 55.0 | 100.0 | | r8i-flex.large | 0.315 / 10.0 | 0.198 / 6.25 | 0.394 / 10.0 | | r8i-flex.xlarge | 0.63 / 10.0 | 0.395 / 6.25 | 0.788 / 10.0 | | r8i-flex.2xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | r8i-flex.4xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | r8i-flex.8xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 12.5 | | r8i-flex.12xlarge | 7.5 / 15.0 | 4.687 / 9.375 | 9.375 / 18.75 | | r8i-flex.16xlarge | 10.0 / 20.0 | 6.25 / 12.5 | 12.5 / 25.0 | | x8g.medium | 0.315 / 10.0 | 0.185 / 6.25 | 0.394 / 10.0 | | x8g.large | 0.63 / 10.0 | 0.396 / 6.25 | 0.788 / 10.0 | | x8g.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | x8g.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | x8g.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | x8g.8xlarge | 10.0 | 6.25 | 12.5 | | x8g.12xlarge | 15.0 | 9.375 | 18.75 | | x8g.16xlarge | 20.0 | 12.5 | 25.0 | | x8g.24xlarge | 30.0 | 20.0 | 37.5 | | x8g.48xlarge | 40.0 | 27.5 | 50.0 | | x8g.metal-24xl | 30.0 | 20.0 | 37.5 | | x8g.metal-48xl | 40.0 | 27.5 | 50.0 | | x8aedz.large | 1.25 / 15.0 | 0.859 / 10.312 | 1.563 / 15.0 | | x8aedz.xlarge | 2.5 / 15.0 | 1.718 / 10.312 | 3.125 / 15.0 | | x8aedz.3xlarge | 7.5 / 15.0 | 5.156 / 10.312 | 9.375 / 15.0 | | x8aedz.6xlarge | 15.0 | 10.312 | 18.75 | | x8aedz.12xlarge | 30.0 | 20.625 | 37.5 | | x8aedz.24xlarge | 60.0 | 41.25 | 75.0 | | x8aedz.metal-12xl | 30.0 | 20.625 | 37.5 | | x8aedz.metal-24xl | 60.0 | 41.25 | 75.0 | | x8i.large | 0.65 / 10.0 | 0.415 / 6.25 | 0.813 / 10.0 | | x8i.xlarge | 1.25 / 10.0 | 0.781 / 6.25 | 1.563 / 10.0 | | x8i.2xlarge | 2.5 / 10.0 | 1.562 / 6.25 | 3.125 / 10.0 | | x8i.4xlarge | 5.0 / 10.0 | 3.125 / 6.25 | 6.25 / 10.0 | | x8i.8xlarge | 10.0 | 6.25 | 12.5 | | x8i.12xlarge | 15.0 | 9.375 | 18.75 | | x8i.16xlarge | 20.0 | 12.5 | 25.0 | | x8i.24xlarge | 30.0 | 20.0 | 37.5 | | x8i.32xlarge | 40.0 | 27.5 | 50.0 | | x8i.48xlarge | 60.0 | 41.25 | 75.0 | | x8i.64xlarge | 70.0 | 50.0 | 87.5 | | x8i.96xlarge | 80.0 | 55.0 | 100.0 | | x8i.metal-48xl | 60.0 | 41.25 | 75.0 | | x8i.metal-96xl | 80.0 | 55.0 | 100.0 | ## Monitor instance bandwidth You can use CloudWatch metrics to monitor instance network bandwidth and the packets sent and received. You can use the network performance metrics provided by the Elastic Network Adapter (ENA) driver to monitor when traffic exceeds the network allowances that Amazon EC2 defines at the instance level. You can configure whether Amazon EC2 sends metric data for the instance to CloudWatch using one-minute periods or five-minute periods. It is possible that the network performance metrics would show that an allowance was exceeded and packets were dropped while the CloudWatch instance metrics do not. This can happen when the instance has a short spike in demand for network resources (known as a microburst), but the CloudWatch metrics are not granular enough to reflect these microsecond spikes. **Learn more** + [Instance metrics](viewing_metrics_with_cloudwatch.md#ec2-cloudwatch-metrics) + [Monitor network performance](monitoring-network-performance-ena.md) # Enhanced networking on Amazon EC2 instances Enhanced networking Enhanced networking uses single root I/O virtualization (SR-IOV) to provide high-performance networking capabilities on supported instance types. SR-IOV is a method of device virtualization that provides higher I/O performance and lower CPU utilization when compared to traditional virtualized network interfaces. Enhanced networking provides higher bandwidth, higher packet per second (PPS) performance, and consistently lower latency between instances. There is no additional charge for using enhanced networking. For information about the supported network speed for each instance type, see [Amazon EC2 Instance Types](https://aws.amazon.com/ec2/instance-types/). You can enable enhanced networking using one of the following mechanisms: **Elastic Network Adapter (ENA)** The Elastic Network Adapter (ENA) supports network speeds of up to 100 Gbps for supported instance types. All [Nitro-based instances](instance-types.md#instance-hypervisor-type) use ENA for enhanced networking. In addition, the following Xen-based instances use ENA: H1, I3, G3, `m4.16xlarge`, P2, P3, P3dn, and R4. For more information, see [Enable enhanced networking with ENA on your EC2 instances](enhanced-networking-ena.md). **Intel 82599 Virtual Function (VF) interface** The Intel 82599 Virtual Function interface supports network speeds of up to 10 Gbps for supported instance types. The following instance types use the Intel 82599 VF interface for enhanced networking: C3, C4, D2, I2, M4 (excluding m4.16xlarge), and R3. For more information, see [Enhanced networking with the Intel 82599 VF interface](sriov-networking.md). **Topics** + [Elastic Network Adapter (ENA)](enhanced-networking-ena.md) + [ENA Express](ena-express.md) + [Intel 82599 VF](sriov-networking.md) + [Monitor network performance](monitoring-network-performance-ena.md) + [Improve network latency on Linux](ena-improve-network-latency-linux.md) + [Nitro performance considerations](ena-nitro-perf.md) + [Optimize network performance on Windows](enhanced-networking-os.md) # Enable enhanced networking with ENA on your EC2 instances Elastic Network Adapter (ENA) Amazon EC2 provides enhanced networking capabilities through the Elastic Network Adapter (ENA). To use enhanced networking, you must use an AMI that includes the required ENA driver or manually install it. Then you can enable ENA support on your instance. To review release notes or install instructions for an ENA driver, see the tab that matches your instance operating system platform. ------ #### [ Linux ] You can review the following documentation on GitHub: + Review [ENA Linux kernel driver release notes](https://github.com/amzn/amzn-drivers/blob/master/kernel/linux/ena/RELEASENOTES.md) on GitHub. + For an overview of the ENA Linux kernel driver that includes install instructions see [Linux kernel driver for Elastic Network Adapter (ENA) family](https://github.com/amzn/amzn-drivers/blob/master/kernel/linux/ena/README.rst) on GitHub. ------ #### [ Windows ] You can review the following documentation from the **Manage device drivers** section of this guide: + [Track ENA Windows driver version releases](ena-driver-releases-windows.md). + [Install the ENA driver on EC2 Windows instances](ena-adapter-driver-install-upgrade-win.md). ------ For Nitro-based instances, enhanced networking capabilities vary by the Nitro version that the instance type implements. To review network specifications for your instance, choose the instance family link for your instance type. If you're not sure which instance family applies, see [Naming conventions](https://docs.aws.amazon.com/ec2/latest/instancetypes/instance-type-names.html) in the *Amazon EC2 Instance Types* guide. + [Network specifications for accelerated computing instances](https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html#ac_network) + [Network specifications for compute optimized instances](https://docs.aws.amazon.com/ec2/latest/instancetypes/co.html#co_network) + [Network specifications for general purpose instances](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html#gp_network) + [Network specifications for high-performance computing instances](https://docs.aws.amazon.com/ec2/latest/instancetypes/hpc.html#hpc_network) + [Network specifications for memory optimized instances](https://docs.aws.amazon.com/ec2/latest/instancetypes/mo.html#mo_network) + [Network specifications for storage optimized instances](https://docs.aws.amazon.com/ec2/latest/instancetypes/so.html#so_network) **Topics** + [ ## Prerequisites for enhanced networking with ENA ](#ena-requirements) + [ # Test whether enhanced networking is enabled ](test-enhanced-networking-ena.md) + [ # Enable enhanced networking on your instance ](enabling_enhanced_networking.md) + [ # ENA queues ](ena-queues.md) + [ # Troubleshoot the ENA kernel driver on Linux ](troubleshooting-ena.md) + [ # Troubleshoot the Elastic Network Adapter Windows driver ](troubleshoot-ena-driver.md) ## Prerequisites for enhanced networking with ENA To prepare for enhanced networking using the ENA, set up your instance as follows: + Launch a [Nitro-based instance](instance-types.md#instance-hypervisor-type). + Ensure that the instance has internet connectivity. + If you have important data on the instance that you want to preserve, you should back that data up now by creating an AMI from your instance. Updating the ENA kernel driver and enabling the `enaSupport` attribute might render incompatible instances or operating systems unreachable. If you have a recent backup, your data will still be retained if this happens. + **Linux instances** – Launch the instance using a supported version of the Linux kernel and a supported distribution, so that ENA enhanced networking is enabled for your instance automatically. For more information, see [ENA Linux Kernel Driver Release Notes](https://github.com/amzn/amzn-drivers/blob/master/kernel/linux/ena/RELEASENOTES.md). + **Windows instances** – If the instance is running Windows Server 2008 R2 SP1, ensure that is has the [SHA-2 code signing support update](https://support.microsoft.com/en-us/help/4474419/sha-2-code-signing-support-update). + Use [AWS CloudShell](https://console.aws.amazon.com/cloudshell) from the AWS Management Console, or install and configure the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) or the [AWS Tools for Windows PowerShell](https://docs.aws.amazon.com/powershell/latest/userguide/) on any computer you choose, preferably your local desktop or laptop. For more information, see [Access Amazon EC2](concepts.md#access-ec2) or the [AWS CloudShell User Guide](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html). Enhanced networking cannot be managed from the Amazon EC2 console. # Test whether enhanced networking is enabled Check whether ENA is enabled You can test whether enhanced networking is enabled in your instances or your AMIs. **Instance attribute** Check the value of the `enaSupport` instance attribute. ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html) command. ``` aws ec2 describe-instances \ --instance-ids i-1234567890abcdef0 \ --query "Reservations[].Instances[].EnaSupport" ``` If enhanced networking is enabled, the output is as follows. ``` [ true ] ``` ------ #### [ PowerShell ] Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2Instance.html) cmdlet. ``` (Get-EC2Instance -InstanceId i-1234567890abcdef0).Instances.EnaSupport ``` If enhanced networking is enabled, the output is as follows. ``` True ``` ------ **Image attribute** Check the value of the `enaSupport` image attribute. ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html) command. ``` aws ec2 describe-images \ --image-id ami-0abcdef1234567890 \ --query "Images[].EnaSupport" ``` If enhanced networking is enabled, the output is as follows. ``` [ true ] ``` ------ #### [ PowerShell ] Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2Image.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2Image.html) cmdlet. ``` (Get-EC2Image -ImageId ami-0abcdef1234567890).EnaSupport ``` If enhanced networking is enabled, the output is as follows. ``` True ``` ------ **Linux network interface driver** Use the following command to verify that the `ena` kernel driver is being used on a particular interface, substituting the interface name that you want to check. If you are using a single interface (default), this is `eth0`. If your Linux distribution supports predictable network names, this could be a name like `ens5`. For more information, expand the section for RHEL, SUSE, and CentOS in [Enable enhanced networking on your instance](enabling_enhanced_networking.md). In the following example, the `ena` kernel driver is not loaded, because the listed driver is `vif`. ``` [ec2-user ~]$ ethtool -i eth0 driver: vif version: firmware-version: bus-info: vif-0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ``` In this example, the `ena` kernel driver is loaded and at the minimum recommended version. This instance has enhanced networking properly configured. ``` [ec2-user ~]$ ethtool -i eth0 driver: ena version: 1.5.0g firmware-version: expansion-rom-version: bus-info: 0000:00:05.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ``` # Enable enhanced networking on your instance Enable ENA for an instance The procedure that you use depends on the operating system of the instance. ## Amazon Linux The AMIs for Amazon Linux include the kernel driver required for enhanced networking with ENA installed and have ENA support enabled. Therefore, if you launch an instance with an HVM version of Amazon Linux on a supported instance type, enhanced networking is already enabled for your instance. For more information, see [Test whether enhanced networking is enabled](test-enhanced-networking-ena.md). ## Ubuntu The latest Ubuntu HVM AMIs include the kernel driver required for enhanced networking with ENA installed and have ENA support enabled. Therefore, if you launch an instance with the latest Ubuntu HVM AMI on a supported instance type, enhanced networking is already enabled for your instance. For more information, see [Test whether enhanced networking is enabled](test-enhanced-networking-ena.md). If you launched your instance using an older AMI and it does not have enhanced networking enabled already, you can install the `linux-aws` kernel package to get the latest enhanced networking drivers and update the required attribute. **To install the `linux-aws` kernel package (Ubuntu 16.04 or later)** Ubuntu 16.04 and 18.04 ship with the Ubuntu custom kernel (`linux-aws` kernel package). To use a different kernel, contact [Support](https://console.aws.amazon.com/support). **To install the `linux-aws` kernel package (Ubuntu Trusty 14.04)** 1. Connect to your instance. 1. Update the package cache and packages. ``` ubuntu:~$ sudo apt-get update && sudo apt-get upgrade -y linux-aws ``` **Important** If during the update process you are prompted to install `grub`, use `/dev/xvda` to install `grub` onto, and then choose to keep the current version of `/boot/grub/menu.lst`. 1. [EBS-backed instance] From your local computer, stop the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html) (AWS Tools for Windows PowerShell). [Instance store-backed instance] You can't stop the instance to modify the attribute. Instead, proceed to this procedure: [To enable enhanced networking on Ubuntu (instance store-backed instances)](#enhanced-networking-ena-instance-store-ubuntu). 1. From your local computer, enable the enhanced networking attribute using one of the following commands: + [https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html) (AWS CLI) ``` aws ec2 modify-instance-attribute --instance-id i-1234567890abcdef0 --ena-support ``` + [https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html) (Tools for Windows PowerShell) ``` Edit-EC2InstanceAttribute -InstanceId i-1234567890abcdef0 -EnaSupport $true ``` 1. (Optional) Create an AMI from the instance, as described in [Create an Amazon EBS-backed AMI](creating-an-ami-ebs.md). The AMI inherits the enhanced networking `enaSupport` attribute from the instance. Therefore, you can use this AMI to launch another instance with enhanced networking enabled by default. 1. From your local computer, start the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html) (AWS Tools for Windows PowerShell). **To enable enhanced networking on Ubuntu (instance store-backed instances)** Follow the previous procedure until the step where you stop the instance. Create a new AMI as described in [Create an Amazon S3-backed AMI](creating-an-ami-instance-store.md), making sure to enable the enhanced networking attribute when you register the AMI. + [https://docs.aws.amazon.com/cli/latest/reference/ec2/register-image.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/register-image.html) (AWS CLI) ``` aws ec2 register-image --ena-support ... ``` + [https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Image.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Image.html) (AWS Tools for Windows PowerShell) ``` Register-EC2Image -EnaSupport $true ... ``` ## RHEL, SUSE, CentOS The latest AMIs for Red Hat Enterprise Linux, SUSE Linux Enterprise Server, and CentOS include the kernel driver required for enhanced networking with ENA and have ENA support enabled. Therefore, if you launch an instance with the latest AMI on a supported instance type, enhanced networking is already enabled for your instance. For more information, see [Test whether enhanced networking is enabled](test-enhanced-networking-ena.md). The following procedure provides the general steps for enabling enhanced networking on a Linux distribution other than Amazon Linux AMI or Ubuntu. For more information, such as detailed syntax for commands, file locations, or package and tool support, see the documentation for your Linux distribution. **To enable enhanced networking on Linux** 1. Connect to your instance. 1. Clone the source code for the `ena` kernel driver on your instance from GitHub at [https://github.com/amzn/amzn-drivers](https://github.com/amzn/amzn-drivers). (SUSE Linux Enterprise Server 12 SP2 and later include ENA 2.02 by default, so you are not required to download and compile the ENA driver. For SUSE Linux Enterprise Server 12 SP2 and later, you should file a request to add the driver version you want to the stock kernel). ``` git clone https://github.com/amzn/amzn-drivers ``` 1. Compile and install the `ena` kernel driver on your instance. These steps depend on the Linux distribution. For more information about compiling the kernel driver on Red Hat Enterprise Linux, see [How do I install the latest ENS driver for enhanced network support on an Amazon EC2 instance that runs RHEL?](https://repost.aws/knowledge-center/install-ena-driver-rhel-ec2) 1. Run the **sudo depmod** command to update kernel driver dependencies. 1. Update `initramfs` on your instance to ensure that the new kernel driver loads at boot time. For example, if your distribution supports **dracut**, you can use the following command. ``` dracut -f -v ``` 1. Determine if your system uses predictable network interface names by default. Systems that use **systemd** or **udev** versions 197 or greater can rename Ethernet devices and they do not guarantee that a single network interface will be named `eth0`. This behavior can cause problems connecting to your instance. For more information and to see other configuration options, see [Predictable Network Interface Names](https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/) on the freedesktop.org website. 1. You can check the **systemd** or **udev** versions on RPM-based systems with the following command. ``` rpm -qa | grep -e '^systemd-[0-9]\+\|^udev-[0-9]\+' systemd-208-11.el7_0.2.x86_64 ``` In the above Red Hat Enterprise Linux 7 example, the **systemd** version is 208, so predictable network interface names must be disabled. 1. Disable predictable network interface names by adding the `net.ifnames=0` option to the `GRUB_CMDLINE_LINUX` line in `/etc/default/grub`. ``` sudo sed -i '/^GRUB\_CMDLINE\_LINUX/s/\"$/\ net\.ifnames\=0\"/' /etc/default/grub ``` 1. Rebuild the grub configuration file. ``` sudo grub2-mkconfig -o /boot/grub2/grub.cfg ``` 1. [EBS-backed instance] From your local computer, stop the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html) (AWS CLI), [https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html) (AWS Tools for Windows PowerShell). [Instance store-backed instance] You can't stop the instance to modify the attribute. Instead, proceed to this procedure: [To enable enhanced networking on Linux (instance store–backed instances)](#other-linux-enhanced-networking-ena-instance-store). 1. From your local computer, enable the enhanced networking `enaSupport` attribute using one of the following commands: + [https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html) (AWS CLI) ``` aws ec2 modify-instance-attribute --instance-id i-1234567890abcdef0 --ena-support ``` + [https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html) (Tools for Windows PowerShell) ``` Edit-EC2InstanceAttribute -InstanceId i-1234567890abcdef0 -EnaSupport $true ``` 1. (Optional) Create an AMI from the instance, as described in [Create an Amazon EBS-backed AMI](creating-an-ami-ebs.md). The AMI inherits the enhanced networking `enaSupport` attribute from the instance. Therefore, you can use this AMI to launch another instance with enhanced networking enabled by default. If your instance operating system contains an `/etc/udev/rules.d/70-persistent-net.rules` file, you must delete it before creating the AMI. This file contains the MAC address for the Ethernet adapter of the original instance. If another instance boots with this file, the operating system will be unable to find the device and `eth0` might fail, causing boot issues. This file is regenerated at the next boot cycle, and any instances launched from the AMI create their own version of the file. 1. From your local computer, start the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html) (AWS Tools for Windows PowerShell). 1. (Optional) Connect to your instance and verify that the kernel driver is installed. If you are unable to connect to your instance after enabling enhanced networking, see [Troubleshoot the ENA kernel driver on Linux](troubleshooting-ena.md). **To enable enhanced networking on Linux (instance store–backed instances)** Follow the previous procedure until the step where you stop the instance. Create a new AMI as described in [Create an Amazon S3-backed AMI](creating-an-ami-instance-store.md), making sure to enable the enhanced networking attribute when you register the AMI. + [https://docs.aws.amazon.com/cli/latest/reference/ec2/register-image.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/register-image.html) (AWS CLI) ``` aws ec2 register-image --ena-support ... ``` + [https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Image.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Image.html) (AWS Tools for Windows PowerShell) ``` Register-EC2Image -EnaSupport ... ``` ## Ubuntu with DKMS This method is for testing and feedback purposes only. It is not intended for use with production deployments. For production deployments, see [Ubuntu](#enhanced-networking-ena-ubuntu). **Important** Using DKMS voids the support agreement for your subscription. It should not be used for production deployments. **To enable enhanced networking with ENA on Ubuntu (EBS-backed instances)** 1. Follow steps 1 and 2 in [Ubuntu](#enhanced-networking-ena-ubuntu). 1. Install the `build-essential` packages to compile the kernel driver and the `dkms` package so that your `ena` kernel driver is rebuilt every time your kernel is updated. ``` ubuntu:~$ sudo apt-get install -y build-essential dkms ``` 1. Clone the source for the `ena` kernel driver on your instance from GitHub at [https://github.com/amzn/amzn-drivers](https://github.com/amzn/amzn-drivers). ``` ubuntu:~$ git clone https://github.com/amzn/amzn-drivers ``` 1. Move the `amzn-drivers` package to the `/usr/src/` directory so DKMS can find it and build it for each kernel update. Append the version number (you can find the current version number in the release notes) of the source code to the directory name. For example, version `1.0.0` is shown in the following example. ``` ubuntu:~$ sudo mv amzn-drivers /usr/src/amzn-drivers-1.0.0 ``` 1. Create the DKMS configuration file with the following values, substituting your version of `ena`. Create the file. ``` ubuntu:~$ sudo touch /usr/src/amzn-drivers-1.0.0/dkms.conf ``` Edit the file and add the following values. ``` ubuntu:~$ sudo vim /usr/src/amzn-drivers-1.0.0/dkms.conf PACKAGE_NAME="ena" PACKAGE_VERSION="1.0.0" CLEAN="make -C kernel/linux/ena clean" MAKE="make -C kernel/linux/ena/ BUILD_KERNEL=${kernelver}" BUILT_MODULE_NAME[0]="ena" BUILT_MODULE_LOCATION="kernel/linux/ena" DEST_MODULE_LOCATION[0]="/updates" DEST_MODULE_NAME[0]="ena" AUTOINSTALL="yes" ``` 1. Add, build, and install the `ena` kernel driver on your instance using DKMS. Add the kernel driver to DKMS. ``` ubuntu:~$ sudo dkms add -m amzn-drivers -v 1.0.0 ``` Build the kernel driver using the **dkms** command. ``` ubuntu:~$ sudo dkms build -m amzn-drivers -v 1.0.0 ``` Install the kernel driver using **dkms**. ``` ubuntu:~$ sudo dkms install -m amzn-drivers -v 1.0.0 ``` 1. Rebuild `initramfs` so the correct kernel driver is loaded at boot time. ``` ubuntu:~$ sudo update-initramfs -u -k all ``` 1. Verify that the `ena` kernel driver is installed using the modinfo ena command from [Test whether enhanced networking is enabled](test-enhanced-networking-ena.md). ``` ubuntu:~$ modinfo ena filename: /lib/modules/3.13.0-74-generic/updates/dkms/ena.ko version: 1.0.0 license: GPL description: Elastic Network Adapter (ENA) author: Amazon.com, Inc. or its affiliates srcversion: 9693C876C54CA64AE48F0CA alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i* alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i* alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i* alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i* depends: vermagic: 3.13.0-74-generic SMP mod_unload modversions parm: debug:Debug level (0=none,...,16=all) (int) parm: push_mode:Descriptor / header push mode (0=automatic,1=disable,3=enable) 0 - Automatically choose according to device capability (default) 1 - Don't push anything to device memory 3 - Push descriptors and header buffer to device memory (int) parm: enable_wd:Enable keepalive watchdog (0=disable,1=enable,default=1) (int) parm: enable_missing_tx_detection:Enable missing Tx completions. (default=1) (int) parm: numa_node_override_array:Numa node override map (array of int) parm: numa_node_override:Enable/Disable numa node override (0=disable) (int) ``` 1. Continue with Step 3 in [Ubuntu](#enhanced-networking-ena-ubuntu). ## Enable enhanced networking on Windows If you launched your instance and it does not have enhanced networking enabled already, you must download and install the required network adapter driver on your instance, and then set the `enaSupport` instance attribute to activate enhanced networking. **To enable enhanced networking** 1. Connect to your instance and log in as the local administrator. 1. [Windows Server 2016 and 2019 only] Run the following EC2Launch PowerShell script to configure the instance after the driver is installed. ``` PS C:\> C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule ``` 1. From the instance, install the driver as follows: 1. [Download](https://s3.amazonaws.com/ec2-windows-drivers-downloads/ENA/Latest/AwsEnaNetworkDriver.zip) the latest driver to the instance. 1. Extract the zip archive. 1. Install the driver by running the `install.ps1` PowerShell script. **Note** If you get an execution policy error, set the policy to `Unrestricted` (by default it is set to `Restricted` or `RemoteSigned`). In a command line, run `Set-ExecutionPolicy -ExecutionPolicy Unrestricted`, and then run the `install.ps1` PowerShell script again. 1. From your local computer, stop the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html) (AWS Tools for Windows PowerShell). 1. Enable ENA support on your instance as follows: 1. From your local computer, check the EC2 instance ENA support attribute on your instance by running one of the following commands. If the attribute is not enabled, the output will be "[]" or blank. `EnaSupport` is set to `false` by default. + [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html) (AWS CLI) ``` aws ec2 describe-instances --instance-ids i-1234567890abcdef0 --query "Reservations[].Instances[].EnaSupport" ``` + [https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2Instance.html) (Tools for Windows PowerShell) ``` (Get-EC2Instance -InstanceId i-1234567890abcdef0).Instances.EnaSupport ``` 1. To enable ENA support, run one of the following commands: + [https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html) (AWS CLI) ``` aws ec2 modify-instance-attribute --instance-id i-1234567890abcdef0 --ena-support ``` + [https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html) (AWS Tools for Windows PowerShell) ``` Edit-EC2InstanceAttribute -InstanceId i-1234567890abcdef0 -EnaSupport $true ``` If you encounter problems when you restart the instance, you can also disable ENA support using one of the following commands: + [https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html) (AWS CLI) ``` aws ec2 modify-instance-attribute --instance-id i-1234567890abcdef0 --no-ena-support ``` + [https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html) (AWS Tools for Windows PowerShell) ``` Edit-EC2InstanceAttribute -InstanceId i-1234567890abcdef0 -EnaSupport $false ``` 1. Verify that the attribute has been set to `true` using **describe-instances** or **Get-EC2Instance** as shown previously. You should now see the following output: ``` [ true ] ``` 1. From your local computer, start the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html) (AWS Tools for Windows PowerShell). 1. On the instance, validate that the ENA driver is installed and enabled as follows: 1. Right-click the network icon and choose **Open Network and Sharing Center**. 1. Choose the Ethernet adapter (for example, **Ethernet 2**). 1. Choose **Details**. For **Network Connection Details**, check that **Description** is **Amazon Elastic Network Adapter**. 1. (Optional) Create an AMI from the instance. The AMI inherits the `enaSupport` attribute from the instance. Therefore, you can use this AMI to launch another instance with ENA enabled by default. # ENA queues ENA queues are allocated to network interfaces with default static limits based on the instance type and size. On supported instance types, you can dynamically allocate these queues across Elastic Network Interfaces (ENIs). While the total queue count per instance depends on its type and size, you can configure multiple ENIs with ENA queues until you meet the maximum queue count for the ENI and the instance. Flexible ENA queue allocation optimizes resource distribution, enabling maximum vCPU utilization. High network performance workloads typically require multiple ENA queues. You can fine-tune network performance and packets per second (PPS) by adjusting queue counts according to your specific workload needs. For example, network-intensive applications may require more queues compared to CPU-intensive applications. **Topics** + [ ## Supported instances ](#supported-instances) + [ ## Modify the number of queues ](#modify) ## Supported instances The following instances support dynamic allocation of multiple ENA queues. ### General purpose | Instance type | Default ENA queues per interface | Maximum ENA queues per interface | Maximum ENA queues per instance | | --- | --- | --- | --- | | M6i | | m6i.large | 2 | 2 | 6 | | m6i.xlarge | 4 | 4 | 16 | | m6i.2xlarge | 8 | 8 | 32 | | m6i.4xlarge | 8 | 16 | 64 | | m6i.8xlarge | 8 | 32 | 64 | | m6i.12xlarge | 8 | 32 | 64 | | m6i.16xlarge | 8 | 32 | 120 | | m6i.24xlarge | 8 | 32 | 120 | | m6i.32xlarge | 8 | 32 | 120 | | M6id | | m6id.large | 2 | 2 | 6 | | m6id.xlarge | 4 | 4 | 16 | | m6id.2xlarge | 8 | 8 | 32 | | m6id.4xlarge | 8 | 16 | 64 | | m6id.8xlarge | 8 | 32 | 64 | | m6id.12xlarge | 8 | 32 | 64 | | m6id.16xlarge | 8 | 32 | 120 | | m6id.24xlarge | 8 | 32 | 120 | | m6id.32xlarge | 8 | 32 | 120 | | M6idn | | m6idn.large | 2 | 2 | 6 | | m6idn.xlarge | 4 | 4 | 16 | | m6idn.2xlarge | 8 | 8 | 32 | | m6idn.4xlarge | 8 | 16 | 64 | | m6idn.8xlarge | 16 | 32 | 128 | | m6idn.12xlarge | 16 | 32 | 128 | | m6idn.16xlarge | 16 | 32 | 240 | | m6idn.24xlarge | 32 | 32 | 480 | | m6idn.32xlarge | 32 | 32 | 512 \$1 | | M6in | | m6in.large | 2 | 2 | 6 | | m6in.xlarge | 4 | 4 | 16 | | m6in.2xlarge | 8 | 8 | 32 | | m6in.4xlarge | 8 | 16 | 64 | | m6in.8xlarge | 16 | 32 | 128 | | m6in.12xlarge | 16 | 32 | 128 | | m6in.16xlarge | 16 | 32 | 240 | | m6in.24xlarge | 32 | 32 | 480 | | m6in.32xlarge | 32 | 32 | 512 \$1 | | M8a | | m8a.medium | 1 | 1 | 3 | | m8a.large | 2 | 2 | 6 | | m8a.xlarge | 4 | 4 | 16 | | m8a.2xlarge | 8 | 8 | 32 | | m8a.4xlarge | 8 | 16 | 64 | | m8a.8xlarge | 8 | 32 | 128 | | m8a.12xlarge | 16 | 64 | 192 | | m8a.16xlarge | 16 | 64 | 256 | | m8a.24xlarge | 16 | 128 | 384 | | m8a.48xlarge | 32 | 128 | 768 | | m8a.metal-24xl | 16 | 128 | 384 | | m8a.metal-48xl | 32 | 128 | 768 | | M8azn | | m8azn.medium | 1 | 1 | 3 | | m8azn.large | 2 | 2 | 8 | | m8azn.xlarge | 4 | 4 | 16 | | m8azn.3xlarge | 4 | 16 | 48 | | m8azn.6xlarge | 8 | 32 | 96 | | m8azn.12xlarge | 8 | 64 | 192 | | m8azn.24xlarge | 16 | 128 | 384 | | m8azn.metal-12xl | 8 | 64 | 192 | | m8azn.metal-24xl | 16 | 128 | 384 | | M8gb | | m8gb.medium | 1 | 1 | 2 | | m8gb.large | 2 | 2 | 6 | | m8gb.xlarge | 4 | 4 | 16 | | m8gb.2xlarge | 8 | 8 | 32 | | m8gb.4xlarge | 8 | 16 | 64 | | m8gb.8xlarge | 8 | 32 | 128 | | m8gb.12xlarge | 16 | 64 | 192 | | m8gb.16xlarge | 16 | 64 | 256 | | m8gb.24xlarge | 16 | 128 | 384 | | m8gb.48xlarge | 32 | 128 | 768 \$1 | | m8gb.metal-24xl | 32 | 128 | 768 | | m8gb.metal-48xl | 32 | 128 | 768 \$1 | | M8gn | | m8gn.medium | 1 | 1 | 2 | | m8gn.large | 2 | 2 | 6 | | m8gn.xlarge | 4 | 4 | 16 | | m8gn.2xlarge | 8 | 8 | 32 | | m8gn.4xlarge | 8 | 16 | 64 | | m8gn.8xlarge | 8 | 32 | 128 | | m8gn.12xlarge | 16 | 64 | 192 | | m8gn.16xlarge | 16 | 64 | 256 | | m8gn.24xlarge | 16 | 128 | 384 | | m8gn.48xlarge | 32 | 128 | 768 \$1 | | m8gn.metal-24xl | 32 | 128 | 768 | | m8gn.metal-48xl | 32 | 128 | 768 \$1 | | M8i | | m8i.large | 2 | 2 | 6 | | m8i.xlarge | 4 | 4 | 16 | | m8i.2xlarge | 8 | 8 | 32 | | m8i.4xlarge | 8 | 16 | 64 | | m8i.8xlarge | 8 | 32 | 128 | | m8i.12xlarge | 16 | 64 | 192 | | m8i.16xlarge | 16 | 64 | 256 | | m8i.24xlarge | 16 | 128 | 384 | | m8i.32xlarge | 16 | 128 | 512 | | m8i.48xlarge | 32 | 128 | 768 | | m8i.96xlarge | 32 | 128 | 1536 | | m8i.metal-48xl | 32 | 128 | 768 | | m8i.metal-96xl | 32 | 128 | 1536 | | M8id | | m8id.large | 2 | 2 | 6 | | m8id.xlarge | 4 | 4 | 16 | | m8id.2xlarge | 8 | 8 | 32 | | m8id.4xlarge | 8 | 16 | 64 | | m8id.8xlarge | 8 | 32 | 128 | | m8id.12xlarge | 16 | 64 | 192 | | m8id.16xlarge | 16 | 64 | 256 | | m8id.24xlarge | 16 | 128 | 384 | | m8id.32xlarge | 16 | 128 | 512 | | m8id.48xlarge | 32 | 128 | 768 | | m8id.96xlarge | 32 | 128 | 1536 | | m8id.metal-48xl | 32 | 128 | 768 | | m8id.metal-96xl | 32 | 128 | 1536 | | M8i-flex | | m8i-flex.large | 1 | 1 | 3 | | m8i-flex.xlarge | 2 | 2 | 8 | | m8i-flex.2xlarge | 4 | 4 | 16 | | m8i-flex.4xlarge | 4 | 8 | 32 | | m8i-flex.8xlarge | 4 | 16 | 64 | | m8i-flex.12xlarge | 8 | 32 | 96 | | m8i-flex.16xlarge | 8 | 32 | 128 | **Note** \$1 These instance types feature multiple network cards. Other instance types feature a single network card. For more information, see [Network cards](using-eni.md#network-cards). ### Compute optimized | Instance type | Default ENA queues per interface | Maximum ENA queues per interface | Maximum ENA queues per instance | | --- | --- | --- | --- | | C6i | | c6i.large | 2 | 2 | 6 | | c6i.xlarge | 4 | 4 | 16 | | c6i.2xlarge | 8 | 8 | 32 | | c6i.4xlarge | 8 | 16 | 64 | | c6i.8xlarge | 8 | 32 | 64 | | c6i.12xlarge | 8 | 32 | 64 | | c6i.16xlarge | 8 | 32 | 120 | | c6i.24xlarge | 8 | 32 | 120 | | c6i.32xlarge | 8 | 32 | 120 | | C6id | | c6id.large | 2 | 2 | 6 | | c6id.xlarge | 4 | 4 | 16 | | c6id.2xlarge | 8 | 8 | 32 | | c6id.4xlarge | 8 | 16 | 64 | | c6id.8xlarge | 8 | 32 | 64 | | c6id.12xlarge | 8 | 32 | 64 | | c6id.16xlarge | 8 | 32 | 120 | | c6id.24xlarge | 8 | 32 | 120 | | c6id.32xlarge | 8 | 32 | 120 | | C6in | | c6in.large | 2 | 2 | 6 | | c6in.xlarge | 4 | 4 | 16 | | c6in.2xlarge | 8 | 8 | 32 | | c6in.4xlarge | 8 | 16 | 64 | | c6in.8xlarge | 16 | 32 | 128 | | c6in.12xlarge | 16 | 32 | 128 | | c6in.16xlarge | 16 | 32 | 240 | | c6in.24xlarge | 32 | 32 | 480 | | c6in.32xlarge | 32 | 32 | 512 \$1 | | C8a | | c8a.medium | 1 | 1 | 3 | | c8a.large | 2 | 2 | 6 | | c8a.xlarge | 4 | 4 | 16 | | c8a.2xlarge | 8 | 8 | 32 | | c8a.4xlarge | 8 | 16 | 64 | | c8a.8xlarge | 8 | 32 | 128 | | c8a.12xlarge | 16 | 64 | 192 | | c8a.16xlarge | 16 | 64 | 256 | | c8a.24xlarge | 16 | 128 | 384 | | c8a.48xlarge | 32 | 128 | 768 | | c8a.metal-24xl | 16 | 128 | 384 | | c8a.metal-48xl | 32 | 128 | 768 | | C8gb | | c8gb.medium | 1 | 1 | 2 | | c8gb.large | 2 | 2 | 6 | | c8gb.xlarge | 4 | 4 | 16 | | c8gb.2xlarge | 8 | 8 | 32 | | c8gb.4xlarge | 8 | 16 | 64 | | c8gb.8xlarge | 8 | 32 | 128 | | c8gb.12xlarge | 16 | 64 | 192 | | c8gb.16xlarge | 16 | 64 | 256 | | c8gb.24xlarge | 16 | 128 | 384 | | c8gb.48xlarge | 32 | 128 | 768 \$1 | | c8gb.metal-24xl | 32 | 128 | 768 | | c8gb.metal-48xl | 32 | 128 | 768 \$1 | | C8gn | | c8gn.medium | 1 | 1 | 2 | | c8gn.large | 2 | 2 | 6 | | c8gn.xlarge | 4 | 4 | 16 | | c8gn.2xlarge | 8 | 8 | 32 | | c8gn.4xlarge | 8 | 16 | 64 | | c8gn.8xlarge | 8 | 32 | 128 | | c8gn.12xlarge | 16 | 64 | 192 | | c8gn.16xlarge | 16 | 64 | 256 | | c8gn.24xlarge | 16 | 128 | 384 | | c8gn.48xlarge | 32 | 128 | 768 \$1 | | c8gn.metal-24xl | 32 | 128 | 768 | | c8gn.metal-48xl | 32 | 128 | 768 \$1 | | C8i | | c8i.large | 2 | 2 | 6 | | c8i.xlarge | 4 | 4 | 16 | | c8i.2xlarge | 8 | 8 | 32 | | c8i.4xlarge | 8 | 16 | 64 | | c8i.8xlarge | 8 | 32 | 128 | | c8i.12xlarge | 16 | 64 | 192 | | c8i.16xlarge | 16 | 64 | 256 | | c8i.24xlarge | 16 | 128 | 384 | | c8i.32xlarge | 16 | 128 | 512 | | c8i.48xlarge | 32 | 128 | 768 | | c8i.96xlarge | 32 | 128 | 1536 | | c8i.metal-48xl | 32 | 128 | 768 | | c8i.metal-96xl | 32 | 128 | 1536 | | C8id | | c8id.large | 2 | 2 | 6 | | c8id.xlarge | 4 | 4 | 16 | | c8id.2xlarge | 8 | 8 | 32 | | c8id.4xlarge | 8 | 16 | 64 | | c8id.8xlarge | 8 | 32 | 128 | | c8id.12xlarge | 16 | 64 | 192 | | c8id.16xlarge | 16 | 64 | 256 | | c8id.24xlarge | 16 | 128 | 384 | | c8id.32xlarge | 16 | 128 | 512 | | c8id.48xlarge | 32 | 128 | 768 | | c8id.96xlarge | 32 | 128 | 1536 | | c8id.metal-48xl | 32 | 128 | 768 | | c8id.metal-96xl | 32 | 128 | 1536 | | C8i-flex | | c8i-flex.large | 1 | 1 | 3 | | c8i-flex.xlarge | 2 | 2 | 8 | | c8i-flex.2xlarge | 4 | 4 | 16 | | c8i-flex.4xlarge | 4 | 8 | 32 | | c8i-flex.8xlarge | 4 | 16 | 64 | | c8i-flex.12xlarge | 8 | 32 | 96 | | c8i-flex.16xlarge | 8 | 32 | 128 | **Note** \$1 These instance types feature multiple network cards. Other instance types feature a single network card. For more information, see [Network cards](using-eni.md#network-cards). ### Memory optimized | Instance type | Default ENA queues per interface | Maximum ENA queues per interface | Maximum ENA queues per instance | | --- | --- | --- | --- | | R6i | | r6i.large | 2 | 2 | 6 | | r6i.xlarge | 4 | 4 | 16 | | r6i.2xlarge | 8 | 8 | 32 | | r6i.4xlarge | 8 | 16 | 64 | | r6i.8xlarge | 8 | 32 | 64 | | r6i.12xlarge | 8 | 32 | 64 | | r6i.16xlarge | 8 | 32 | 120 | | r6i.24xlarge | 8 | 32 | 120 | | r6i.32xlarge | 8 | 32 | 120 | | R6id | | r6id.large | 2 | 2 | 6 | | r6id.xlarge | 4 | 4 | 16 | | r6id.2xlarge | 8 | 8 | 32 | | r6id.4xlarge | 8 | 16 | 64 | | r6id.8xlarge | 8 | 32 | 64 | | r6id.12xlarge | 8 | 32 | 64 | | r6id.16xlarge | 8 | 32 | 120 | | r6id.24xlarge | 8 | 32 | 120 | | r6id.32xlarge | 8 | 32 | 120 | | R6idn | | r6idn.large | 2 | 2 | 6 | | r6idn.xlarge | 4 | 4 | 16 | | r6idn.2xlarge | 8 | 8 | 32 | | r6idn.4xlarge | 8 | 16 | 64 | | r6idn.8xlarge | 16 | 32 | 128 | | r6idn.12xlarge | 16 | 32 | 128 | | r6idn.16xlarge | 16 | 32 | 240 | | r6idn.24xlarge | 32 | 32 | 480 | | r6idn.32xlarge | 32 | 32 | 512 \$1 | | R6in | | r6in.large | 2 | 2 | 6 | | r6in.xlarge | 4 | 4 | 16 | | r6in.2xlarge | 8 | 8 | 32 | | r6in.4xlarge | 8 | 16 | 64 | | r6in.8xlarge | 16 | 32 | 128 | | r6in.12xlarge | 16 | 32 | 128 | | r6in.16xlarge | 16 | 32 | 240 | | r6in.24xlarge | 32 | 32 | 480 | | r6in.32xlarge | 32 | 32 | 512 \$1 | | R8a | | r8a.medium | 1 | 1 | 3 | | r8a.large | 2 | 2 | 6 | | r8a.xlarge | 4 | 4 | 16 | | r8a.2xlarge | 8 | 8 | 32 | | r8a.4xlarge | 8 | 16 | 64 | | r8a.8xlarge | 8 | 32 | 128 | | r8a.12xlarge | 16 | 64 | 192 | | r8a.16xlarge | 16 | 64 | 256 | | r8a.24xlarge | 16 | 128 | 384 | | r8a.48xlarge | 32 | 128 | 768 | | r8a.metal-24xl | 16 | 128 | 384 | | r8a.metal-48xl | 32 | 128 | 768 | | R8gb | | r8gb.medium | 1 | 1 | 2 | | r8gb.large | 2 | 2 | 6 | | r8gb.xlarge | 4 | 4 | 16 | | r8gb.2xlarge | 8 | 8 | 32 | | r8gb.4xlarge | 8 | 16 | 64 | | r8gb.8xlarge | 8 | 32 | 128 | | r8gb.12xlarge | 16 | 64 | 192 | | r8gb.16xlarge | 16 | 64 | 256 | | r8gb.24xlarge | 16 | 128 | 384 | | r8gb.48xlarge | 32 | 128 | 768 \$1 | | r8gb.metal-24xl | 32 | 128 | 768 | | r8gb.metal-48xl | 32 | 128 | 768 \$1 | | R8gn | | r8gn.medium | 1 | 1 | 2 | | r8gn.large | 2 | 2 | 6 | | r8gn.xlarge | 4 | 4 | 16 | | r8gn.2xlarge | 8 | 8 | 32 | | r8gn.4xlarge | 8 | 16 | 64 | | r8gn.8xlarge | 8 | 32 | 128 | | r8gn.12xlarge | 16 | 64 | 192 | | r8gn.16xlarge | 16 | 64 | 256 | | r8gn.24xlarge | 16 | 128 | 384 | | r8gn.48xlarge | 32 | 128 | 768 \$1 | | r8gn.metal-24xl | 32 | 128 | 768 | | r8gn.metal-48xl | 32 | 128 | 768 \$1 | | R8i | | r8i.large | 2 | 2 | 6 | | r8i.xlarge | 4 | 4 | 16 | | r8i.2xlarge | 8 | 8 | 32 | | r8i.4xlarge | 8 | 16 | 64 | | r8i.8xlarge | 8 | 32 | 128 | | r8i.12xlarge | 16 | 64 | 192 | | r8i.16xlarge | 16 | 64 | 256 | | r8i.24xlarge | 16 | 128 | 384 | | r8i.32xlarge | 16 | 128 | 512 | | r8i.48xlarge | 32 | 128 | 768 | | r8i.96xlarge | 32 | 128 | 1536 | | r8i.metal-48xl | 32 | 128 | 768 | | r8i.metal-96xl | 32 | 128 | 1536 | | R8id | | r8id.large | 2 | 2 | 6 | | r8id.xlarge | 4 | 4 | 16 | | r8id.2xlarge | 8 | 8 | 32 | | r8id.4xlarge | 8 | 16 | 64 | | r8id.8xlarge | 8 | 32 | 128 | | r8id.12xlarge | 16 | 64 | 192 | | r8id.16xlarge | 16 | 64 | 256 | | r8id.24xlarge | 16 | 128 | 384 | | r8id.32xlarge | 16 | 128 | 512 | | r8id.48xlarge | 32 | 128 | 768 | | r8id.96xlarge | 32 | 128 | 1536 | | r8id.metal-48xl | 32 | 128 | 768 | | r8id.metal-96xl | 32 | 128 | 1536 | | R8i-flex | | r8i-flex.large | 1 | 1 | 3 | | r8i-flex.xlarge | 2 | 2 | 8 | | r8i-flex.2xlarge | 4 | 4 | 16 | | r8i-flex.4xlarge | 4 | 8 | 32 | | r8i-flex.8xlarge | 4 | 16 | 64 | | r8i-flex.12xlarge | 8 | 32 | 96 | | r8i-flex.16xlarge | 8 | 32 | 128 | | X8aedz | | x8aedz.large | 2 | 2 | 8 | | x8aedz.xlarge | 4 | 4 | 16 | | x8aedz.3xlarge | 4 | 16 | 48 | | x8aedz.6xlarge | 8 | 32 | 96 | | x8aedz.12xlarge | 8 | 64 | 192 | | x8aedz.24xlarge | 16 | 128 | 384 | | x8aedz.metal-12xl | 8 | 64 | 192 | | x8aedz.metal-24xl | 16 | 128 | 384 | | X8i | | x8i.large | 2 | 2 | 6 | | x8i.xlarge | 4 | 4 | 16 | | x8i.2xlarge | 8 | 8 | 32 | | x8i.4xlarge | 8 | 16 | 64 | | x8i.8xlarge | 8 | 32 | 128 | | x8i.12xlarge | 16 | 64 | 192 | | x8i.16xlarge | 16 | 64 | 256 | | x8i.24xlarge | 16 | 128 | 384 | | x8i.32xlarge | 16 | 128 | 512 | | x8i.48xlarge | 32 | 128 | 768 | | x8i.64xlarge | 32 | 128 | 1024 | | x8i.96xlarge | 32 | 128 | 1536 | | x8i.metal-48xl | 32 | 128 | 768 | | x8i.metal-96xl | 32 | 128 | 1536 | **Note** \$1 These instance types feature multiple network cards. Other instance types feature a single network card. For more information, see [Network cards](using-eni.md#network-cards). ## Modify the number of queues You can modify the number of ENA queues using AWS Management Console or AWS CLI. In the AWS Management Console, the ENA queues configuration is available under each **Network interface** setting. To modify the number of ENA queues using the AWS CLI, use either one of the following commands. Before modifying the queue count, use the following command to check your current queue count. ``` aws ec2 describe-instances --instance-id i-1234567890abcdef0 ``` **Note** Your instance must be stopped before modifying the number of ENA queues. The value for ENA queues must be a power of 2, such as, 1, 2, 4, 8, 16, 32, etc. The number of queues allocated to any single ENI cannot exceed the number of vCPUs available on your instance. `[attach-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/attach-network-interface.html)` In the following example, 32 ENA queues are configured on an ENI. ``` aws ec2 attach-network-interface \ --network-interface-id eni-001aa1bb223cdd4e4 \ --instance-id i-1234567890abcdef0 \ --device-index 1 \ --ena-queue-count 32 ``` `[run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html)` In the following example, 2 ENA queues each are configured on 3 ENIs. ``` aws ec2 run-instances \ --image-id ami-12ab3c30 \ --instance-type c6i.large \ --min-count 1 \ --max-count 1 \ --network-interfaces \ "[{\"DeviceIndex\":0,\"SubnetId\":\"subnet-123456789012a345a\",\"EnaQueueCount\":2}, {\"DeviceIndex\":1,\"SubnetId\":\"subnet-123456789012a345a\",\"EnaQueueCount\":2}, {\"DeviceIndex\":2,\"SubnetId\":\"subnet-123456789012a345a\",\"EnaQueueCount\":2}]" ``` `[modify-network-interface-attribute](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-network-interface-attribute.html)` In the following example, 32 ENA queues are configured on an ENI. ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --attachment AttachmentId=eni-attach-12345678,EnaQueueCount=32 ``` In the following example, the ENA count is reset to the default value. ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --attachment AttachmentId=eni-attach-12345678,DefaultEnaQueueCount=true ``` # Troubleshoot the ENA kernel driver on Linux Troubleshoot ENA on Linux The Elastic Network Adapter (ENA) is designed to improve operating system health and reduce the chances of long-term disruption because of unexpected hardware behavior and or failures. The ENA architecture keeps device or driver failures as transparent to the system as possible. This topic provides troubleshooting information for ENA. If you are unable to connect to your instance, start with the [Troubleshoot connectivity issues](#ena-connectivity-issues) section. If you experience performance degradation after migrating to a sixth generation instance type, see the article [What do I need to do before I migrate my EC2 instance to a sixth generation instance to make sure that I get maximum network performance?](https://repost.aws/knowledge-center/migrate-to-gen6-ec2-instance) If you are able to connect to your instance, you can gather diagnostic information by using the failure detection and recovery mechanisms that are covered in the later sections of this topic. **Topics** + [ ## Troubleshoot connectivity issues ](#ena-connectivity-issues) + [ ## Keep-alive mechanism ](#ena-keep-alive) + [ ## Register read timeout ](#register-read-timeout-ena) + [ ## Statistics ](#statistics-ena) + [ ## Driver error logs in syslog ](#driver-error-logs-ena) + [ ## Sub-optimal configuration notifications ](#ts-ena-sub-opt-config-notification) ## Troubleshoot connectivity issues If you lose connectivity while enabling enhanced networking, the `ena` module might be incompatible with your instance's current running kernel. This can happen if you install the module for a specific kernel version (without **dkms**, or with an improperly configured **dkms.conf** file) and then your instance kernel is updated. If the instance kernel that is loaded at boot time does not have the `ena` module properly installed, your instance will not recognize the network adapter and your instance becomes unreachable. If you enable enhanced networking for a PV instance or AMI, this can also make your instance unreachable. If your instance becomes unreachable after enabling enhanced networking with ENA, you can disable the `enaSupport` attribute for your instance and it will fall back to the stock network adapter. **To disable enhanced networking with ENA (EBS-backed instances)** 1. From your local computer, stop the instance by using the Amazon EC2 console, the [stop-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html) command (AWS CLI), or the [Stop-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html) cmdlet (AWS Tools for PowerShell). 1. From your local computer, disable the enhanced networking attribute by using the [modify-instance-attribute](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html) command with the `--no-ena-support` option or the [Edit-EC2InstanceAttribute](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html) cmdlet with the `-EnaSupport $false` parameter. 1. From your local computer, start the instance using the Amazon EC2 console, the [start-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html) command, or the [Start-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html) cmdlet. 1. (Optional) Connect to your instance and try reinstalling the `ena` module with your current kernel version by following the steps in [Enable enhanced networking with ENA on your EC2 instances](enhanced-networking-ena.md). **To disable enhanced networking with ENA (instance store-backed instances)** 1. Create a new AMI as described in [Create an Amazon S3-backed AMI](creating-an-ami-instance-store.md). 1. When you register the AMI, be sure to include the `--no-ena-support` option in the [stop-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html) command (AWS CLI) or the `-EnaSupport $false` parameter in the [Register-EC2Image](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Image.html) cmdlet. ## Keep-alive mechanism The ENA device posts keep-alive events at a fixed rate (usually once every second). The ENA driver implements a watchdog mechanism, which checks for the presence of these keep-alive messages. If a message or messages are present, the watchdog is rearmed, otherwise the driver concludes that the device experienced a failure and then does the following: + Dumps its current statistics to syslog + Resets the ENA device + Resets the ENA driver state The above reset procedure may result in some traffic loss for a short period of time (TCP connections should be able to recover), but should not otherwise affect the user. The ENA device may also indirectly request a device reset procedure, by not sending a keep-alive notification, for example, if the ENA device reaches an unknown state after loading an irrecoverable configuration. The following is an example of the reset procedure: ``` [18509.800135] ena 0000:00:07.0 eth1: Keep alive watchdog timeout. // The watchdog process initiates a reset [18509.815244] ena 0000:00:07.0 eth1: Trigger reset is on [18509.825589] ena 0000:00:07.0 eth1: tx_timeout: 0 // The driver logs the current statistics [18509.834253] ena 0000:00:07.0 eth1: io_suspend: 0 [18509.842674] ena 0000:00:07.0 eth1: io_resume: 0 [18509.850275] ena 0000:00:07.0 eth1: wd_expired: 1 [18509.857855] ena 0000:00:07.0 eth1: interface_up: 1 [18509.865415] ena 0000:00:07.0 eth1: interface_down: 0 [18509.873468] ena 0000:00:07.0 eth1: admin_q_pause: 0 [18509.881075] ena 0000:00:07.0 eth1: queue_0_tx_cnt: 0 [18509.888629] ena 0000:00:07.0 eth1: queue_0_tx_bytes: 0 [18509.895286] ena 0000:00:07.0 eth1: queue_0_tx_queue_stop: 0 ....... ........ [18511.280972] ena 0000:00:07.0 eth1: free uncompleted tx skb qid 3 idx 0x7 // At the end of the down process, the driver discards incomplete packets. [18511.420112] [ENA_COM: ena_com_validate_version] ena device version: 0.10 //The driver begins its up process [18511.420119] [ENA_COM: ena_com_validate_version] ena controller version: 0.0.1 implementation version 1 [18511.420127] [ENA_COM: ena_com_admin_init] ena_defs : Version:[b9692e8] Build date [Wed Apr 6 09:54:21 IDT 2016] [18512.252108] ena 0000:00:07.0: Device watchdog is Enabled [18512.674877] ena 0000:00:07.0: irq 46 for MSI/MSI-X [18512.674933] ena 0000:00:07.0: irq 47 for MSI/MSI-X [18512.674990] ena 0000:00:07.0: irq 48 for MSI/MSI-X [18512.675037] ena 0000:00:07.0: irq 49 for MSI/MSI-X [18512.675085] ena 0000:00:07.0: irq 50 for MSI/MSI-X [18512.675141] ena 0000:00:07.0: irq 51 for MSI/MSI-X [18512.675188] ena 0000:00:07.0: irq 52 for MSI/MSI-X [18512.675233] ena 0000:00:07.0: irq 53 for MSI/MSI-X [18512.675279] ena 0000:00:07.0: irq 54 for MSI/MSI-X [18512.772641] [ENA_COM: ena_com_set_hash_function] Feature 10 isn't supported [18512.772647] [ENA_COM: ena_com_set_hash_ctrl] Feature 18 isn't supported [18512.775945] ena 0000:00:07.0: Device reset completed successfully // The reset process is complete ``` ## Register read timeout The ENA architecture suggests a limited usage of memory mapped I/O (MMIO) read operations. MMIO registers are accessed by the ENA device driver only during its initialization procedure. If the driver logs (available in **dmesg** output) indicate failures of read operations, this may be caused by an incompatible or incorrectly compiled driver, a busy hardware device, or hardware failure. Intermittent log entries that indicate failures on read operations should not be considered an issue; the driver will retry them in this case. However, a sequence of log entries containing read failures indicate a driver or hardware problem. Below is an example of driver log entry indicating a read operation failure due to a timeout: ``` [ 47.113698] [ENA_COM: ena_com_reg_bar_read32] reading reg failed for timeout. expected: req id[1] offset[88] actual: req id[57006] offset[0]  [ 47.333715] [ENA_COM: ena_com_reg_bar_read32] reading reg failed for timeout. expected: req id[2] offset[8] actual: req id[57007] offset[0]  [ 47.346221] [ENA_COM: ena_com_dev_reset] Reg read32 timeout occurred ``` ## Statistics If you experience insufficient network performance or latency issues, you should retrieve the device statistics and examine them. These statistics can be obtained using **ethtool** as follows. ``` [ec2-user ~]$ ethtool -S ethN NIC statistics: tx_timeout: 0 suspend: 0 resume: 0 wd_expired: 0 interface_up: 1 interface_down: 0 admin_q_pause: 0 bw_in_allowance_exceeded: 0 bw_out_allowance_exceeded: 0 pps_allowance_exceeded: 0 conntrack_allowance_available: 450878 conntrack_allowance_exceeded: 0 linklocal_allowance_exceeded: 0 queue_0_tx_cnt: 4329 queue_0_tx_bytes: 1075749 queue_0_tx_queue_stop: 0 ... ``` The following command output parameters are described below: `tx_timeout`: *N* The number of times that the Netdev watchdog was activated. `suspend`: *N* The number of times the driver performed a suspend operation. `resume`: *N* The number of times the driver performed a resume operation. `wd_expired`: *N* The number of times that the driver did not receive the keep-alive event in the preceding three seconds. `interface_up`: *N* The number of times that the ENA interface was brought up. `interface_down`: *N* The number of times that the ENA interface was brought down. `admin_q_pause`: *N* The number of times the admin queue was not found in a running state. `bw_in_allowance_exceeded`: *N* The number of packets queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance. `bw_out_allowance_exceeded`: *N* The number of packets queued or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance. `pps_allowance_exceeded`: *N* The number of packets queued or dropped because the bidirectional PPS exceeded the maximum for the instance. \$1 `conntrack_allowance_available`: *N* The number of tracked connections that can be established by the instance before hitting the Connections Tracked allowance of that instance type. Only available for Nitro-based instances. Not supported with FreeBSD instances or DPDK environments. `conntrack_allowance_exceeded`: *N* The number of packets dropped because connection tracking exceeded the maximum for the instance and new connections could not be established. This can result in packet loss for traffic to or from the instance. `linklocal_allowance_exceeded`: *N* The number of packets dropped because the PPS of the traffic to local proxy services exceeded the maximum for the network interface. This impacts traffic to the Amazon DNS service, the Instance Metadata Service, and the Amazon Time Sync Service, but does not impact traffic to custom DNS resolvers. `queue_N_tx_cnt`: *N* The number of transmitted packets for this queue. `queue_N_tx_bytes`: *N* The number of transmitted bytes for this queue. `queue_N_tx_queue_stop`: *N* The number of times that queue *N* was full and stopped. `queue_N_tx_queue_wakeup`: *N* The number of times that queue *N* resumed after being stopped. `queue_N_tx_dma_mapping_err`: *N* Direct memory access error count. If this value is not 0, it indicates low system resources. `queue_N_tx_linearize`: *N* The number of times SKB linearization was attempted for this queue. `queue_N_tx_linearize_failed`: *N* The number of times SKB linearization failed for this queue. `queue_N_tx_napi_comp`: *N* The number of times the `napi` handler called `napi_complete` for this queue. `queue_N_tx_tx_poll`: *N* The number of times the `napi` handler was scheduled for this queue. `queue_N_tx_doorbells`: *N* The number of transmission doorbells for this queue. `queue_N_tx_prepare_ctx_err`: *N* The number of times `ena_com_prepare_tx` failed for this queue. `queue_N_tx_bad_req_id`: *N* Invalid `req_id` for this queue. The valid `req_id` is zero, minus the `queue_size`, minus 1. `queue_N_tx_llq_buffer_copy`: *N* The number of packets whose headers size are larger than llq entry for this queue. `queue_N_tx_missed_tx`: *N* The number of packets that were left uncompleted for this queue. `queue_N_tx_unmask_interrupt`: *N* The number of times the tx interrupt was unmasked for this queue. `queue_N_rx_cnt`: *N* The number of received packets for this queue. `queue_N_rx_bytes`: *N* The number of received bytes for this queue. `queue_N_rx_rx_copybreak_pkt`: *N* The number of times the rx queue received a packet that is less than the rx\$1copybreak packet size for this queue. `queue_N_rx_csum_good`: *N* The number of times the rx queue received a packet where the checksum was checked and was correct for this queue. `queue_N_rx_refil_partial`: *N* The number of times the driver did not succeed in refilling the empty portion of the rx queue with the buffers for this queue. If this value is not zero, it indicates low memory resources. `queue_N_rx_bad_csum`: *N* The number of times the `rx` queue had a bad checksum for this queue (only if rx checksum offload is supported). `queue_N_rx_page_alloc_fail`: *N* The number of time that page allocation failed for this queue. If this value is not zero, it indicates low memory resources. `queue_N_rx_skb_alloc_fail`: *N* The number of time that SKB allocation failed for this queue. If this value is not zero, it indicates low system resources. `queue_N_rx_dma_mapping_err`: *N* Direct memory access error count. If this value is not 0, it indicates low system resources. `queue_N_rx_bad_desc_num`: *N* Too many buffers per packet. If this value is not 0, it indicates the use of very small buffers. `queue_N_rx_bad_req_id`: *N* The req\$1id for this queue is not valid. The valid req\$1id is from [0, queue\$1size - 1 ]. `queue_N_rx_empty_rx_ring`: *N* The number of times the rx queue was empty for this queue. `queue_N_rx_csum_unchecked`: *N* The number of times the rx queue received a packet whose checksum wasn't checked for this queue. `queue_N_rx_xdp_aborted`: *N* The number of times that an XDP packet was classified as XDP\$1ABORT. `queue_N_rx_xdp_drop`: *N* The number of times that an XDP packet was classified as XDP\$1DROP. `queue_N_rx_xdp_pass`: *N* The number of times that an XDP packet was classified as XDP\$1PASS. `queue_N_rx_xdp_tx`: *N* The number of times that an XDP packet was classified as XDP\$1TX. `queue_N_rx_xdp_invalid`: *N* The number of times that the XDP return code for the packet was not valid. `queue_N_rx_xdp_redirect`: *N* The number of times that an XDP packet was classified as XDP\$1REDIRECT. `queue_N_xdp_tx_cnt`: *N* The number of transmitted packets for this queue. `queue_N_xdp_tx_bytes`: *N* The number of transmitted bytes for this queue. `queue_N_xdp_tx_queue_stop`: *N* The number of times that this queue was full and stopped. `queue_N_xdp_tx_queue_wakeup`: *N* The number of times that this queue resumed after being stopped. `queue_N_xdp_tx_dma_mapping_err`: *N* Direct memory access error count. If this value is not 0, it indicates low system resources. `queue_N_xdp_tx_linearize`: *N* The number of times XDP buffer linearization was attempted for this queue. `queue_N_xdp_tx_linearize_failed`: *N* The number of times XDP buffer linearization failed for this queue. `queue_N_xdp_tx_napi_comp`: *N* The number of times the napi handler called napi\$1complete for this queue. `queue_N_xdp_tx_tx_poll`: *N* The number of times the napi handler was scheduled for this queue. `queue_N_xdp_tx_doorbells`: *N* The number of transmission doorbells for this queue. `queue_N_xdp_tx_prepare_ctx_err`: *N* The number of times ena\$1com\$1prepare\$1tx failed for this queue. This value should always be zero; if not, see the driver logs. `queue_N_xdp_tx_bad_req_id`: *N* The req\$1id for this queue is not valid. The valid req\$1id is from [0, queue\$1size - 1 ]. `queue_N_xdp_tx_llq_buffer_copy`: *N* The number of packets that had their headers copied using llq buffer copy for this queue. `queue_N_xdp_tx_missed_tx`: *N* The number of times a tx queue entry missed a completion timeout for this queue. `queue_N_xdp_tx_unmask_interrupt`: *N* The number of times the tx interrupt was unmasked for this queue. `ena_admin_q_aborted_cmd`: *N* The number of admin commands that were aborted. This usually happens during the auto-recovery procedure. `ena_admin_q_submitted_cmd`: *N* The number of admin queue doorbells. `ena_admin_q_completed_cmd`: *N* The number of admin queue completions. `ena_admin_q_out_of_space`: *N* The number of times that the driver tried to submit new admin command, but the queue was full. `ena_admin_q_no_completion`: *N* The number of times that the driver did not get an admin completion for a command. ## Driver error logs in syslog The ENA driver writes log messages to **syslog** during system boot. You can examine these logs to look for errors if you are experiencing issues. Below is an example of information logged by the ENA driver in **syslog** during system boot, along with some annotations for select messages. ``` Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 478.416939] [ENA_COM: ena_com_validate_version] ena device version: 0.10 Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 478.420915] [ENA_COM: ena_com_validate_version] ena controller version: 0.0.1 implementation version 1 Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 479.256831] ena 0000:00:03.0: Device watchdog is Enabled Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 479.672947] ena 0000:00:03.0: creating 8 io queues. queue size: 1024 Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 479.680885] [ENA_COM: ena_com_init_interrupt_moderation] Feature 20 isn't supported // Interrupt moderation is not supported by the device Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 479.691609] [ENA_COM: ena_com_get_feature_ex] Feature 10 isn't supported // RSS HASH function configuration is not supported by the device Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 479.694583] [ENA_COM: ena_com_get_feature_ex] Feature 18 isn't supported //RSS HASH input source configuration is not supported by the device Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 479.697433] [ENA_COM: ena_com_set_host_attributes] Set host attribute isn't supported Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 479.701064] ena 0000:00:03.0 (unnamed net_device) (uninitialized): Cannot set host attributes Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 479.704917] ena 0000:00:03.0: Elastic Network Adapter (ENA) found at mem f3000000, mac addr 02:8a:3c:1e:13:b5 Queues 8 Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 480.805037] EXT4-fs (xvda1): re-mounted. Opts: (null) Jun 3 22:37:46 ip-172-31-2-186 kernel: [ 481.025842] NET: Registered protocol family 10 ``` **Which errors can I ignore?** The following warnings that may appear in your system's error logs can be ignored for the Elastic Network Adapter: Set host attribute isn't supported Host attributes are not supported for this device. failed to alloc buffer for rx queue This is a recoverable error, and it indicates that there may have been a memory pressure issue when the error was thrown. Feature *X* isn't supported The referenced feature is not supported by the Elastic Network Adapter. Possible values for *X* include: + 10: RSS Hash function configuration is not supported for this device. + 12: RSS Indirection table configuration is not supported for this device. + 18: RSS Hash Input configuration is not supported for this device. + 20: Interrupt moderation is not supported for this device. + 27: The Elastic Network Adapter driver does not support polling the Ethernet capabilities from snmpd. Failed to config AENQ The Elastic Network Adapter does not support AENQ configuration. Trying to set unsupported AENQ events This error indicates an attempt to set an AENQ events group that is not supported by the Elastic Network Adapter. ## Sub-optimal configuration notifications The ENA device detects sub-optimal configuration settings in the driver that you can change. The device notifies the ENA driver and logs a warning to the console. The following example shows the format of the warning message. ``` Sub-optimal configuration notification code: 1. Refer to AWS ENA documentation for additional details and mitigation options. ``` The following list shows notification code details and recommended actions for sub-optimal configuration findings. + **Code 1: ENA Express with wide LLQ configuration is not recommended** ENA Express ENI is configured with wide LLQ. This configuration is sub-optimal and could impact performance for ENA Express. We recommend that you disable wide LLQ settings when you use ENA Express ENIs as follows. ``` sudo rmmod ena && sudo modprobe ena force_large_llq_header=0 ``` For more information about optimal configuration for ENA Express, see [Improve network performance between EC2 instances with ENA Express](ena-express.md). + **Code 2: ENA Express ENI with sub-optimal Tx queue depth is not recommended** ENA Express ENI is configured with sub-optimal Tx queue depth. This configuration could impact performance for ENA Express. We recommend that you enlarge all Tx queues to the maximum value for the network interface when you use ENA Express ENIs as follows. You can run the following **ethtool** commands to adjust LLQ size. To learn more about how to control, query, and enable wide-LLQ, see the [Large Low-Latency Queue (Large LLQ)](https://github.com/amzn/amzn-drivers/tree/master/kernel/linux/ena#large-low-latency-queue-large-llq) topic of the Linux kernel driver for ENA documentation in the *Amazon Drivers GitHub repository*. ``` ethtool -g interface ``` Set your Tx queues to the maximum depth: ``` ethtool -G interface tx depth ``` For more information about optimal configuration for ENA Express, see [Improve network performance between EC2 instances with ENA Express](ena-express.md). + **Code 3: ENA with regular LLQ size and Tx packet traffic exceeds the maximum header supported size** By default, ENA LLQ supports Tx packet header size up to 96 bytes. If the packet header size is larger than 96 bytes, the packet is dropped. To mitigate this issue, we recommend that you enable wide-LLQ, which increases the supported Tx packet header size to a maximum of 224 bytes. However, when you enable wide-LLQ, the maximum Tx ring size is reduced from 1000 to 512 entries. Wide-LLQ is enabled by default for all Nitro v4 and later instance types. + Nitro v4 instance types have a default maximum wide-LLQ Tx ring size of 512 entries, which can't be changed. + Nitro v5 instance types have a default wide-LLQ Tx ring size of 512 entries, which you can increase up to 1000 entries. You can run the following **ethtool** commands to adjust LLQ size. To learn more about how to control, query, and enable wide-LLQ, see the [Large Low-Latency Queue (Large LLQ)](https://github.com/amzn/amzn-drivers/tree/master/kernel/linux/ena#large-low-latency-queue-large-llq) topic of the Linux kernel driver for ENA documentation in the *Amazon Drivers GitHub repository*. Find the maximum depth for your Tx queues: ``` ethtool -g interface ``` Set your Tx queues to the maximum depth: ``` ethtool -G interface tx depth ``` # Troubleshoot the Elastic Network Adapter Windows driver Troubleshoot ENA on Windows The Elastic Network Adapter (ENA) is designed to improve operating system health and to reduce unexpected hardware behavior or failures that can disrupt the operation of your Windows instance. The ENA architecture keeps device or driver failures as transparent to the operating system as possible. ## Collect diagnostic information on the instance Collect diagnostic information The steps to open Windows operating system (OS) tools vary, depending on what version of the OS is installed on your instance. In the following sections, we use the **Run** dialog to open the tools, which works the same across all OS versions. However, you can access these tools using any method that you prefer. **Access the Run dialog** + Using the Windows logo key combination: `Windows` \$1 `R` + Using the search bar: + Enter `run` in the search bar. + Select the **Run** application from the search results. Some steps require the context menu to access properties or context-sensitive actions. There are several ways to do this, depending on your OS version and hardware. **Access the context menu** + Using your mouse: right-click an item to bring up its context menu. + Using your keyboard: + Depending on your OS version, use `Shift` \$1 `F10`, or `Ctrl` \$1 `Shift` \$1 `F10`. + If you have the context key on your keyboard (three horizontal lines in a box), select the item you want and then press the context key. If you can connect to your instance, use the following techniques to gather diagnostic information for troubleshooting. ### Check ENA device status To check the status of your ENA Windows driver using the Windows Device Manager, follow these steps: 1. Open the **Run** dialog using one of the methods described in the preceding section. 1. To open the Windows Device Manager, enter `devmgmt.msc` in the **Run** box. 1. Choose **OK**. This opens the Device Manager window. 1. Select the arrow to the left of **Network adapters** to expand the list. 1. Choose the name, or open the context menu for the **Amazon Elastic Network Adapter**, and then choose **Properties**. This opens the **Amazon Elastic Network Adapter Properties** dialog. 1. Verify that the message in the **General** tab says "This device is working properly." ### Investigate driver event messages To review ENA Windows driver event logs using the Windows Event Viewer, follow these steps: 1. Open the **Run** dialog using one of the methods described in the preceding section. 1. To open the Windows Event Viewer, enter `eventvwr.msc` in the **Run** box. 1. Choose **OK**. This opens the Event Viewer window. 1. Expand the **Windows Logs** menu, and then choose **System**. 1. Under **Actions**, in the top-right panel, choose **Filter Current Log**. This displays the filtering dialog. 1. In the **Event sources** box, enter `ena`. This limits results to events that were generated by the ENA Windows driver. 1. Choose **OK**. This shows filtered event log results in the detail sections of the window. 1. To drill down into the details, select an event message from the list. The following example shows an ENA driver event in the Windows Event Viewer system events list: ![\[Example: ENA driver event shown in the Windows Event Viewer system messages list.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/ena-event-viewer-example.png) #### Event message summary The following table shows event messages that the ENA Windows driver generates. **Input** | Event ID | ENA driver event description | Type | | --- | --- | --- | | 5001 | Hardware is out of resources | Error | | 5002 | Adapter has detected a hardware error | Error | | 5005 | Adapter has timed out on NDIS operation that did not complete in a timely manner | Error | | 5032 | Adapter has failed to reset the device | Error | | 5200 | Adapter has been initialized | Informational | | 5201 | Adapter has been halted | Informational | | 5202 | Adapter has been paused | Informational | | 5203 | Adapter has been restarted | Informational | | 5204 | Adapter has been shut down | Informational | | 5205 | Adapter has been reset | Error | | 5206 | Adapter has been surprise removed | Error | | 5208 | Adapter initialization routine has failed | Error | | 5210 | Adapter has encountered and successfully recovered an internal issue | Error | ### Review performance metrics The ENA Windows driver publishes network performance metrics from the instances where metrics are enabled. You can view and enable metrics on the instance using the native Performance Monitor application. For more information about the metrics that the ENA Windows driver produces, see [Monitor network performance for ENA settings on your EC2 instance](monitoring-network-performance-ena.md). On instances where ENA metrics are enabled, and the Amazon CloudWatch agent is installed, CloudWatch collects the metrics that are associated with the counters in Windows Performance Monitor, as well as some advanced metrics for ENA. These metrics are collected in addition to the metrics enabled by default on EC2 instances. For more information about the metrics, see [Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html) in the *Amazon CloudWatch User Guide*. **Note** Performance metrics are available for ENA driver versions 2.4.0 and later (also for version 2.2.3). ENA driver version 2.2.4 was rolled back due to potential performance degradation on the sixth generation EC2 instances. We recommend that you upgrade to the current version of the driver to ensure that you have the latest updates. Some of the ways that you can use performance metrics include: + Troubleshoot instance performance issues. + Choose the right instance size for a workload. + Proactively plan scaling activities. + Benchmark applications to determine if they maximize the performance available on an instance. **Refresh rate** By default, the driver refreshes metrics using a 1-second interval. However, the application that retrieves the metrics might use a different interval for polling. You can change the refresh interval in Device Manager, using the advanced properties for the driver. To change the metrics refresh interval for the ENA Windows driver, follow these steps: 1. Open the **Run** dialog using one of the methods described in the preceding section. 1. To open the Windows Device Manager, enter `devmgmt.msc` in the **Run** box. 1. Choose **OK**. This opens the Device Manager window. 1. Select the arrow to the left of **Network adapters** to expand the list. 1. Choose the name, or open the context menu for the **Amazon Elastic Network Adapter**, and then choose **Properties**. This opens the **Amazon Elastic Network Adapter Properties** dialog. 1. Open the **Advanced** tab in the pop-up window. 1. From the **Property** list, choose **Metrics Refresh Interval** to change the value. 1. When you are done, choose **OK**. ## Investigate sub-optimal configuration notifications The ENA device detects sub-optimal configuration settings in the driver that you can change. The device notifies the ENA driver and logs an event notification. To review sub-optimal events in the Windows Event Viewer 1. Open the **Run** dialog using one of the methods described in the preceding section. 1. To open the Windows Event Viewer, enter `eventvwr.msc` in the **Run** box. 1. Choose **OK**. This opens the Event Viewer window. 1. Expand the **Windows Logs** menu, and then choose **System**. 1. Under **Actions**, in the top-right panel, choose **Filter Current Log**. This displays the filtering dialog. 1. In the **Event sources** box, enter `ena`. This limits results to events that were generated by the ENA Windows driver. 1. Choose **OK**. This shows filtered event log results in the detail sections of the window. Events with ID `59000` notify you of sub-optimal configuration findings. Right-click an event and choose **Event Properties** to open a detailed view, or select **Preview Pane** from the **View** menu to see the same detail. ![\[Example: System event ID 59000 shown in the Windows Event Viewer preview pane.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/ena-sub-opt-event-general.png) Open the **Details** tab to see the event code. In the **Binary Data: In words** section, the last word is the code. ![\[Example: The last word in the Binary Data section is shown highlighted.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/ena-sub-opt-event-detail.png) The following list shows notification code details and recommended actions for sub-optimal configuration findings. + **Code `1`: ENA Express with wide LLQ configuration is not recommended** ENA Express ENI is configured with wide LLQ. This configuration is sub-optimal and could impact performance for ENA Express. We recommend that you disable wide LLQ settings when you use ENA Express ENIs as follows. 1. To open the Windows Device Manager, enter `devmgmt.msc` in the **Run** box. 1. Choose **OK**. This opens the Device Manager window. 1. Select the arrow to the left of **Network adapters** to expand the list. 1. Open the device properties for the `Amazon Elastic Network Adapter`. 1. From there, open the **Advanced** tab to make your changes. 1. Select the **LLQ Header Size Policy** property, and set its value to `Normal (128 Bytes)`. 1. Choose **OK** to save your changes. + **Code `2`: ENA Express ENI with sub-optimal Tx queue depth is not recommended** ENA Express ENI is configured with sub-optimal Tx queue depth. This configuration could impact performance for ENA Express. We recommend that you enlarge all Tx queues to the maximum value for the network interface when you use ENA Express ENIs as follows. Follow these steps to enlarge the Tx queues to the maximum depth: 1. To open the Windows Device Manager, enter `devmgmt.msc` in the **Run** box. 1. Choose **OK**. This opens the Device Manager window. 1. Select the arrow to the left of **Network adapters** to expand the list. 1. Open the device properties for the `Amazon Elastic Network Adapter`. 1. From there, open the **Advanced** tab to make your changes. 1. Select the **Transmit Buffers** property, and set its value to the maximum supported. 1. Choose **OK** to save your changes. ## ENA adapter reset The reset process starts when the ENA Windows driver detects an error on an adapter, and marks the adapter as unhealthy. The driver cannot reset itself, so it depends on the operating system to check the adapter health status, and call the reset handle for the ENA Windows driver. The reset process might result in a brief period of time where traffic loss occurs. However, TCP connections should be able to recover. The ENA adapter might also indirectly request a device reset procedure, by failing to send a keep-alive notification. For example, if the ENA adapter reaches an unknown state after loading an irrecoverable configuration, it might stop sending keep-alive notifications. **Common causes for ENA adapter reset** + Keep-alive messages are missing The ENA adapter posts keep-alive events at a fixed rate (usually once every second). The ENA Windows driver implements a watchdog mechanism, which periodically checks for the presence of these keep-alive messages. If it detects one or more new messages since the last time it checked, it records a successful outcome. Otherwise, the driver concludes that the device experienced a failure, and initiates a reset sequence. + Packets are stuck in transmit queues The ENA adapter verifies that packets are flowing through the transmit queues as expected. The ENA Windows driver detects if packets are getting stuck, and initiates a reset sequence if they are. + Read timeout for Memory Mapped I/O (MMIO) registers To limit memory mapped I/O (MMIO) read operations, the ENA Windows driver accesses MMIO registers only during initialization and reset processes. If the driver detects a timeout, it takes one of the following actions, depending on what process was running: + If a timeout is detected during initialization, it fails the flow, which results in the driver displaying a yellow exclamation mark by the ENA adapter in Windows Device Manager. + If a timeout is detected during reset, it fails the flow. The OS then initiates a surprise removal of the ENA adapter, and recovers it by stopping and starting the adapter that was removed. For more information about surprise removal of a network interface card (NIC), see [Handling the Surprise Removal of a NIC](https://learn.microsoft.com/en-us/windows-hardware/drivers/network/handling-the-surprise-removal-of-a-nic) in the *Microsoft Windows Hardware Developer* documentation. ## Troubleshooting scenarios The following scenarios can help you troubleshoot issues that you might experience with the ENA Windows driver. We recommend that you start with upgrading your ENA driver, if you don't have the latest version. To find the latest driver for your Windows OS version, see [Track ENA Windows driver version releases](ena-driver-releases-windows.md). ### Unexpected ENA driver version installed #### Description After you go through the steps to install a specific version of the ENA driver, the Windows Device Manager shows that Windows installed a different version of the ENA driver. #### Cause When you run the install for a driver package, Windows ranks all of the driver packages that are valid for the given device in the local [Driver Store](https://learn.microsoft.com/en-us/windows-hardware/drivers/install/driver-store) before it begins. Then it selects the package with the lowest rank value as the best match. This can be different from the package that you intended to install. For more information about the device driver package selection process, see [How Windows selects a driver package for a device](https://learn.microsoft.com/en-us/windows-hardware/drivers/install/how-windows-selects-a-driver-for-a-device) on the *Microsoft documentation website*. #### Solution To ensure that Windows installs your chosen driver package version, you can remove lower ranked driver packages from the Driver Store with the [PnPUtil](https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/pnputil) command line tool. Follow these steps to update the ENA driver: 1. Connect to your instance and log in as the local administrator. 1. Open the Device Manager properties window, as described in the [Check ENA device status](#ts-ena-diagnostics-device-mgr) section. This opens the **General** tab of the **Amazon Elastic Network Adapter Properties** window. 1. Open the **Driver** tab. 1. Choose **Update Driver**. This opens the **Update Driver Software – Amazon Elastic Network Adapter** dialog box. 1. On the **How do you want to search for driver software?** page, choose **Browse my computer for driver software**. 1. On the **Browse for driver software on your computer** page, choose **Let me pick from a list of device drivers on my computer**, located below the search bar. 1. On the **Select the device driver you want to install for this hardware** page, choose **Have Disk...**. 1. In the **Install from Disk** window, choose **Browse...**, next to the file location from the dropdown list. 1. Navigate to the location where you downloaded the target ENA driver package. Select the file named `ena.inf` and choose **Open**. 1. To start the install, choose **OK**, and then choose **Next**. 1. If the installer doesn’t automatically reboot your instance, run the **Restart-Computer** PowerShell cmdlet. ``` PS C:\> Restart-Computer ``` ### Device warning for ENA driver #### Description The ENA adapter icon in the Device Manager **Network adapters** section displays a warning sign (a yellow triangle with an exclamation mark inside). The following example shows an ENA adapter with the warning icon in Windows Device Manager: ![\[Example: ENA adapter with warning icon shown in the Windows Device Manager.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/ena-adapter-device-mgr-warn.png) #### Cause This device warning is commonly caused by environment issues, which might require more research, and often require a process of elimination to determine the underlying cause. For a full list of device errors, see [Device Manager Error Messages](https://learn.microsoft.com/en-us/windows-hardware/drivers/install/device-manager-error-messages) in the Microsoft documentation. #### Solution The solution for this device warning depends on the root cause. The process of elimination described here includes a few basic steps to help identify and resolve the most common issues that might have a simple solution. Additional root cause analysis is required when these steps do not resolve the issue. Follow these steps to help identify and resolve common issues: 1. **Stop and start the device** Open the Device Manager properties window, as described in the [Check ENA device status](#ts-ena-diagnostics-device-mgr) section. This opens the **General** tab of the **Amazon Elastic Network Adapter Properties** window, where the **Device status** displays the error code and a short message. 1. Open the **Driver** tab. 1. Choose **Disable Device**, and respond **Yes** to the warning message that displays. 1. Choose **Enable Device**. 1. **Stop and start the EC2 instance** If the adapter still shows the warning icon in Device Manager, the next step is to stop and start the EC2 instance. This relaunches the instance on different hardware in most cases. 1. **Investigate possible instance resource issue** If you have stopped and started your EC2 instance, and the problem persists, this might indicate a resource issue on your instance, such as insufficient memory. ### Connection timeout with adapter reset (error codes 5007, 5205) #### Description The Windows Event Viewer shows adapter timeout and reset events occurring in combination for ENA adapters. Messages resemble the following examples: + **Event ID 5007**: Amazon Elastic Network Adapter : Timed out during an operation. + **Event ID 5205**: Amazon Elastic Network Adapter : Adapter reset has been started. Adapter resets cause minimal traffic disruption. Even when there are multiple resets, it would be unusual for them to cause any severe network disruption. #### Cause This sequence of events indicates that the ENA Windows driver initiated a reset for an ENA adapter that was unresponsive. However, the mechanism that the device driver uses to detect this issue is subject to false positives resulting from CPU 0 starvation. #### Solution If this combination of errors happens frequently, check your resource allocations to see where adjustments might be helpful. 1. Open the **Run** dialog using one of the methods described in the preceding section. 1. To open the Windows Resource Monitor, enter `resmon` in the **Run** box. 1. Choose **OK**. This opens the Resource Monitor window. 1. Open the **CPU** tab. Per-CPU usage graphs are shown along the right side of the Resource Monitor window. 1. Check the usage levels for CPU 0 to see if they are too high. We recommend that you configure RSS to exclude CPU 0 for the ENA adapter on larger instance types (more than 16 vCPU). For smaller instance types, configuring RSS might improve the experience, but due to the lower number of available cores, testing is necessary to ensure that constraining CPU cores does not negatively impact performance. Use the **Set-NetAdapterRss** command to configure RSS for your ENA adapter, as shown in the following example. ``` Set-NetAdapterRss -name (Get-NetAdapter | Where-Object {$_.InterfaceDescription -like "*Elastic*"}).Name -Baseprocessorgroup 0 -BaseProcessorNumber 1 ``` ### Migrating to a sixth generation instance infrastructure impacts performance or attachment #### Description If you migrate to a sixth generation EC2 instance, you might experience reduced performance or ENA attachment failures if you haven't updated your ENA Windows driver version. #### Cause The sixth generation EC2 instance types require the following minimum version of the ENA Windows driver, based on the instance operating system (OS). **Minimum version** | Windows Server version | ENA driver version | | --- | --- | | Windows Server 2008 R2 | 2.2.3 or 2.4.0 | | Windows Server 2012 and later | 2.2.3 and later | | Windows Workstation | 2.2.3 and later | #### Solution Before you upgrade to a sixth generation EC2 instance, make sure that the AMI you launch from has compatible drivers based on the instance OS as shown in the previous table. For more information, see [What do I need to do before migrating my EC2 instance to a sixth generation instance to make sure that I get maximum network performance?](https://repost.aws/knowledge-center/migrate-to-gen6-ec2-instance) in the *AWS re:Post Knowledge Center*. ### Suboptimal performance for the elastic network interface #### Description The ENA interface is not performing as expected. #### Cause Root cause analysis for performance issues is a process of elimination. There are too many variables involved to name a common cause. #### Solution The first step in your root cause analysis is to review the diagnostic information for the instance that is not performing as expected, to determine if there are errors that might be causing the issue. For more information, see the [Collect diagnostic information on the instance](#ts-ena-drv-collect-diagnostics) section. You might need to modify the default operating system configuration to achieve maximum network performance on instances with enhanced networking. Some optimizations, such as turning on checksum offloading and enabling RSS, are configured by default in official Windows AMIs. For other optimizations that you can apply to the ENA adapter, see the performance adjustments shown in [ENA adapter performance adjustments](#ts-ena-drv-perf-adj). We recommend that you proceed with caution, and limit device property adjustments to those that are listed in this section, or to specific changes that are recommended by the AWS support team. To change ENA adapter properties, follow these steps: 1. Open the **Run** dialog using one of the methods described in the preceding section. 1. To open the Windows Device Manager, enter `devmgmt.msc` in the **Run** box. 1. Choose **OK**. This opens the Device Manager window. 1. Select the arrow to the left of **Network adapters** to expand the list. 1. Choose the name, or open the context menu for the **Amazon Elastic Network Adapter**, and then choose **Properties**. This opens the **Amazon Elastic Network Adapter Properties** dialog. 1. To make your changes, open the **Advanced** tab. 1. When you're done, choose **OK** to save your changes. The following example shows an ENA adapter property in the Windows Device Manager: ![\[Example: ENA adapter property shown in the Windows Device Manager.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/ena-adapter-device-mgr-property.png) ##### ENA adapter performance adjustments The following table includes properties that can be adjusted to improve performance for the ENA interface. **Input** | Property | Description | Default value | Adjustment | | --- | --- | --- | --- | | Receive Buffers | Controls the number of entries in the software receive queues. | 1024 | Can be increased up to a maximum of 8192. | | Receive Side Scaling (RSS) | Enables the efficient distribution of network receive processing across multiple CPUs in multiprocessor systems. | Enabled | You can spread the load across multiple processors. To learn more, see [Optimize network performance on EC2 Windows instances](enhanced-networking-os.md). | | Maximum Number of RSS Queues | Sets the maximum number of RSS queues allowed when `RSS` is enabled. | 32 | The number of RSS queues is determined during driver initialization, and includes the following limitations (among others): [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshoot-ena-driver.html) You can set the value from 1-32, depending on your instance and hardware generation limits. To learn more, see [Optimize network performance on EC2 Windows instances](enhanced-networking-os.md). | | Jumbo packet | Enables the use of jumbo ethernet frames (more than 1500 bytes of payload). | Disabled (this limits payload to 1500 bytes or less) | Value can be set up to `9015`, which translates to 9001 bytes of payload. This is the maximum payload for jumbo ethernet frames. See [Considerations for using jumbo ethernet frames](#ts-ena-drv-jumbo-frames). | ##### Considerations for using jumbo ethernet frames Jumbo frames allow more than 1500 bytes of data by increasing the payload size per packet, which increases the percentage of the packet that is not packet overhead. Fewer packets are needed to send the same amount of usable data. However, traffic is limited to a maximum MTU of 1500 in the following cases: + Traffic outside of a given AWS Region for EC2 Classic. + Traffic outside of a single VPC. + Traffic over an inter-Region VPC peering connection. + Traffic over VPN connections. + Traffic over an internet gateway. **Note** Packets over 1500 bytes are fragmented. If you have the `Don't Fragment` flag set in the IP header, these packets are dropped. Jumbo frames should be used with caution for internet-bound traffic, or any traffic that leaves a VPC. Packets are fragmented by intermediate systems, which slows down this traffic. To use jumbo frames inside of a VPC without impacting outbound traffic that's leaving the VPC, try one of the following options: Configure the MTU size by route. Use multiple network interfaces with different MTU sizes and different routes. **Recommended use cases for jumbo frames** Jumbo frames can be useful for traffic inside of and between VPCs. We recommend using jumbo frames for the following use cases: + For instances that are collocated inside of a cluster placement group, jumbo frames help to achieve the maximum network throughput possible. For more information, see [Placement groups for your Amazon EC2 instances](placement-groups.md). + You can use jumbo frames for traffic between your VPCs and your on-premises networks over Direct Connect. For more information about using Direct Connect, and verifying jumbo frame capability, see [MTU for private virtual interfaces or transit virtual interfaces](https://docs.aws.amazon.com/directconnect/latest/UserGuide/WorkingWithVirtualInterfaces.html#set-jumbo-frames-vif.html) in the *Direct Connect User Guide*. + For more information about supported MTU sizes for transit gateways, see [Quotas for your transit gateways](https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-quotas.html#mtu-quota) in the *Amazon VPC Transit Gateways*. # Improve network performance between EC2 instances with ENA Express ENA Express ENA Express is powered by AWS Scalable Reliable Datagram (SRD) technology. SRD is a high performance network transport protocol that uses dynamic routing to increase throughput and minimize tail latency. With ENA Express, you can communicate between two EC2 instances in the same Availability Zone. **Benefits of ENA Express** + Increases the maximum bandwidth a single flow can use from 5 Gbps up to 25 Gbps within the Availability Zone, up to the aggregate instance limit. + Reduces tail latency of network traffic between EC2 instances, especially during periods of high network load. + Detects and avoids congested network paths. + Handles some tasks directly in the network layer, such as packet reordering on the receiving end, and most retransmits that are needed. This frees up the application layer for other work. **Note** If your application sends or receives a high volume of packets per second, and needs to optimize for latency most of the time, especially during periods when there is no congestion on the network, [Enhanced networking](enhanced-networking.md) might be a better fit for your network. ENA Express traffic can't be sent in a Local Zone. After you've enabled ENA Express for the network interface attachment on an instance, the sending instance initiates communication with the receiving instance, and SRD detects if ENA Express is operating on both the sending instance and the receiving instance. If ENA Express is operating, the communication can use SRD transmission. If ENA Express is not operating, the communication falls back to standard ENA transmission. During periods of time when network traffic is light, you might notice a slight increase in packet latency (tens of microseconds) when the packet uses ENA Express. During those times, applications that prioritize specific network performance characteristics can benefit from ENA Express as follows: + Processes can benefit from increased maximum single flow bandwidth from 5 Gbps up to 25 Gbps within the same Availability Zone, up to the aggregate instance limit. For example, if a specific instance type supports up to 12.5 Gbps, the single flow bandwidth is also limited to 12.5 Gbps. + Longer running processes should experience reduced tail latency during periods of network congestion. + Processes can benefit from a smoother and more standard distribution for network response times. **Topics** + [ ## How ENA Express works ](#ena-express-how-it-works) + [ ## Supported instance types for ENA Express ](#ena-express-supported-instance-types) + [ ## Prerequisites for Linux instances ](#ena-express-prereq-linux) + [ ## Tune performance for ENA Express settings on Linux instances ](#ena-express-tune) + [ # Review ENA Express settings for your EC2 instance ](ena-express-list-view.md) + [ # Configure ENA Express settings for your EC2 instance ](ena-express-configure.md) ## How ENA Express works ENA Express is powered by AWS Scalable Reliable Datagram (SRD) technology. It distributes packets for each network flow across different AWS network paths, and dynamically adjusts distribution when it detects signs of congestion. It also manages packet reordering on the receiving end. To ensure that ENA Express can manage network traffic as intended, sending and receiving instances and the communication between them must meet all of the following requirements: + Both sending and receiving instance types are supported. See the [Supported instance types for ENA Express](#ena-express-supported-instance-types) table for more information. + Both sending and receiving instances must have ENA Express configured. If there are differences in the configuration, you can run into situations where traffic defaults to standard ENA transmission. The following scenario shows what can happen. **Scenario: Differences in configuration** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ena-express.html) In this case, TCP traffic between the two instances can use ENA Express, as both instances have enabled it. However, since one of the instances does not use ENA Express for UDP traffic, communication between these two instances over UDP uses standard ENA transmission. + The sending and receiving instances must run in the same Availability Zone. + The network path between the instances must not include middleware boxes. ENA Express doesn't currently support middleware boxes. + (Linux instances only) To utilize full bandwidth potential, use driver version 2.2.9 or higher. + (Linux instances only) To produce metrics, use driver version 2.8 or higher. If any requirement is unmet, the instances use the standard TCP/UDP protocol but without SRD to communicate. To ensure that your instance network driver is configured for optimum performance, review the recommended best practices for ENA drivers. These best practices apply to ENA Express, as well. For more information, see the [ENA Linux Driver Best Practices and Performance Optimization Guide](https://github.com/amzn/amzn-drivers/blob/master/kernel/linux/ena/ENA_Linux_Best_Practices.rst) on the GitHub website. **Note** Amazon EC2 refers to the relationship between an instance and a network interface that's attached to it as an *attachment*. ENA Express settings apply to the attachment. If the network interface is detached from the instance, the attachment no longer exists, and the ENA Express settings that applied to it are no longer in force. The same is true when an instance is terminated, even if the network interface remains. After you've enabled ENA Express for the network interface attachments on both the sending instance and the receiving instance, you can use ENA Express metrics to help ensure that your instances take full advantage of the performance improvements that SRD technology provides. For more information about ENA Express metrics, see [Metrics for ENA Express](monitoring-network-performance-ena.md#network-performance-metrics-ena-express). ## Supported instance types for ENA Express The following instance types support ENA Express. ------ #### [ General purpose ] | Instance type | Architecture | | --- | --- | | m6a.12xlarge | x86\$164 | | m6a.16xlarge | x86\$164 | | m6a.24xlarge | x86\$164 | | m6a.32xlarge | x86\$164 | | m6a.48xlarge | x86\$164 | | m6a.metal | x86\$164 | | m6i.8xlarge | x86\$164 | | m6i.12xlarge | x86\$164 | | m6i.16xlarge | x86\$164 | | m6i.24xlarge | x86\$164 | | m6i.32xlarge | x86\$164 | | m6i.metal | x86\$164 | | m6id.8xlarge | x86\$164 | | m6id.12xlarge | x86\$164 | | m6id.16xlarge | x86\$164 | | m6id.24xlarge | x86\$164 | | m6id.32xlarge | x86\$164 | | m6id.metal | x86\$164 | | m6idn.8xlarge | x86\$164 | | m6idn.12xlarge | x86\$164 | | m6idn.16xlarge | x86\$164 | | m6idn.24xlarge | x86\$164 | | m6idn.32xlarge | x86\$164 | | m6idn.metal | x86\$164 | | m6in.8xlarge | x86\$164 | | m6in.12xlarge | x86\$164 | | m6in.16xlarge | x86\$164 | | m6in.24xlarge | x86\$164 | | m6in.32xlarge | x86\$164 | | m6in.metal | x86\$164 | | m7a.12xlarge | x86\$164 | | m7a.16xlarge | x86\$164 | | m7a.24xlarge | x86\$164 | | m7a.32xlarge | x86\$164 | | m7a.48xlarge | x86\$164 | | m7a.metal-48xl | x86\$164 | | m7g.12xlarge | arm64 | | m7g.16xlarge | arm64 | | m7g.metal | arm64 | | m7gd.12xlarge | arm64 | | m7gd.16xlarge | arm64 | | m7gd.metal | arm64 | | m7i.12xlarge | x86\$164 | | m7i.16xlarge | x86\$164 | | m7i.24xlarge | x86\$164 | | m7i.48xlarge | x86\$164 | | m7i.metal-24xl | x86\$164 | | m7i.metal-48xl | x86\$164 | | m8a.16xlarge | x86\$164 | | m8a.24xlarge | x86\$164 | | m8a.48xlarge | x86\$164 | | m8a.metal-24xl | x86\$164 | | m8a.metal-48xl | x86\$164 | | m8azn.12xlarge | x86\$164 | | m8azn.24xlarge | x86\$164 | | m8azn.metal-12xl | x86\$164 | | m8azn.metal-24xl | x86\$164 | | m8g.12xlarge | arm64 | | m8g.16xlarge | arm64 | | m8g.24xlarge | arm64 | | m8g.48xlarge | arm64 | | m8g.metal-24xl | arm64 | | m8g.metal-48xl | arm64 | | m8gb.8xlarge | arm64 | | m8gb.12xlarge | arm64 | | m8gb.16xlarge | arm64 | | m8gb.24xlarge | arm64 | | m8gb.48xlarge | arm64 | | m8gb.metal-24xl | arm64 | | m8gb.metal-48xl | arm64 | | m8gd.12xlarge | arm64 | | m8gd.16xlarge | arm64 | | m8gd.24xlarge | arm64 | | m8gd.48xlarge | arm64 | | m8gd.metal-24xl | arm64 | | m8gd.metal-48xl | arm64 | | m8gn.8xlarge | arm64 | | m8gn.12xlarge | arm64 | | m8gn.16xlarge | arm64 | | m8gn.24xlarge | arm64 | | m8gn.48xlarge | arm64 | | m8gn.metal-24xl | arm64 | | m8gn.metal-48xl | arm64 | | m8i.24xlarge | x86\$164 | | m8i.32xlarge | x86\$164 | | m8i.48xlarge | x86\$164 | | m8i.96xlarge | x86\$164 | | m8i.metal-48xl | x86\$164 | | m8i.metal-96xl | x86\$164 | | m8id.24xlarge | x86\$164 | | m8id.32xlarge | x86\$164 | | m8id.48xlarge | x86\$164 | | m8id.96xlarge | x86\$164 | | m8id.metal-48xl | x86\$164 | | m8id.metal-96xl | x86\$164 | ------ #### [ Compute optimized ] | Instance type | Architecture | | --- | --- | | c6a.12xlarge | x86\$164 | | c6a.16xlarge | x86\$164 | | c6a.24xlarge | x86\$164 | | c6a.32xlarge | x86\$164 | | c6a.48xlarge | x86\$164 | | c6a.metal | x86\$164 | | c6gn.4xlarge | arm64 | | c6gn.8xlarge | arm64 | | c6gn.12xlarge | arm64 | | c6gn.16xlarge | arm64 | | c6i.8xlarge | x86\$164 | | c6i.12xlarge | x86\$164 | | c6i.16xlarge | x86\$164 | | c6i.24xlarge | x86\$164 | | c6i.32xlarge | x86\$164 | | c6i.metal | x86\$164 | | c6id.8xlarge | x86\$164 | | c6id.12xlarge | x86\$164 | | c6id.16xlarge | x86\$164 | | c6id.24xlarge | x86\$164 | | c6id.32xlarge | x86\$164 | | c6id.metal | x86\$164 | | c6in.8xlarge | x86\$164 | | c6in.12xlarge | x86\$164 | | c6in.16xlarge | x86\$164 | | c6in.24xlarge | x86\$164 | | c6in.32xlarge | x86\$164 | | c6in.metal | x86\$164 | | c7a.12xlarge | x86\$164 | | c7a.16xlarge | x86\$164 | | c7a.24xlarge | x86\$164 | | c7a.32xlarge | x86\$164 | | c7a.48xlarge | x86\$164 | | c7a.metal-48xl | x86\$164 | | c7g.12xlarge | arm64 | | c7g.16xlarge | arm64 | | c7g.metal | arm64 | | c7gd.12xlarge | arm64 | | c7gd.16xlarge | arm64 | | c7gd.metal | arm64 | | c7gn.4xlarge | arm64 | | c7gn.8xlarge | arm64 | | c7gn.12xlarge | arm64 | | c7gn.16xlarge | arm64 | | c7gn.metal | arm64 | | c7i.12xlarge | x86\$164 | | c7i.16xlarge | x86\$164 | | c7i.24xlarge | x86\$164 | | c7i.48xlarge | x86\$164 | | c7i.metal-24xl | x86\$164 | | c7i.metal-48xl | x86\$164 | | c8a.16xlarge | x86\$164 | | c8a.24xlarge | x86\$164 | | c8a.48xlarge | x86\$164 | | c8a.metal-24xl | x86\$164 | | c8a.metal-48xl | x86\$164 | | c8g.12xlarge | arm64 | | c8g.16xlarge | arm64 | | c8g.24xlarge | arm64 | | c8g.48xlarge | arm64 | | c8g.metal-24xl | arm64 | | c8g.metal-48xl | arm64 | | c8gb.8xlarge | arm64 | | c8gb.12xlarge | arm64 | | c8gb.16xlarge | arm64 | | c8gb.24xlarge | arm64 | | c8gb.48xlarge | arm64 | | c8gb.metal-24xl | arm64 | | c8gb.metal-48xl | arm64 | | c8gd.12xlarge | arm64 | | c8gd.16xlarge | arm64 | | c8gd.24xlarge | arm64 | | c8gd.48xlarge | arm64 | | c8gd.metal-24xl | arm64 | | c8gd.metal-48xl | arm64 | | c8gn.8xlarge | arm64 | | c8gn.12xlarge | arm64 | | c8gn.16xlarge | arm64 | | c8gn.24xlarge | arm64 | | c8gn.48xlarge | arm64 | | c8gn.metal-24xl | arm64 | | c8gn.metal-48xl | arm64 | | c8i.24xlarge | x86\$164 | | c8i.32xlarge | x86\$164 | | c8i.48xlarge | x86\$164 | | c8i.96xlarge | x86\$164 | | c8i.metal-48xl | x86\$164 | | c8i.metal-96xl | x86\$164 | | c8id.24xlarge | x86\$164 | | c8id.32xlarge | x86\$164 | | c8id.48xlarge | x86\$164 | | c8id.96xlarge | x86\$164 | | c8id.metal-48xl | x86\$164 | | c8id.metal-96xl | x86\$164 | ------ #### [ Memory optimized ] | Instance type | Architecture | | --- | --- | | r6a.12xlarge | x86\$164 | | r6a.16xlarge | x86\$164 | | r6a.24xlarge | x86\$164 | | r6a.32xlarge | x86\$164 | | r6a.48xlarge | x86\$164 | | r6a.metal | x86\$164 | | r6i.8xlarge | x86\$164 | | r6i.12xlarge | x86\$164 | | r6i.16xlarge | x86\$164 | | r6i.24xlarge | x86\$164 | | r6i.32xlarge | x86\$164 | | r6i.metal | x86\$164 | | r6id.8xlarge | x86\$164 | | r6id.12xlarge | x86\$164 | | r6id.16xlarge | x86\$164 | | r6id.24xlarge | x86\$164 | | r6id.32xlarge | x86\$164 | | r6id.metal | x86\$164 | | r6idn.8xlarge | x86\$164 | | r6idn.12xlarge | x86\$164 | | r6idn.16xlarge | x86\$164 | | r6idn.24xlarge | x86\$164 | | r6idn.32xlarge | x86\$164 | | r6idn.metal | x86\$164 | | r6in.8xlarge | x86\$164 | | r6in.12xlarge | x86\$164 | | r6in.16xlarge | x86\$164 | | r6in.24xlarge | x86\$164 | | r6in.32xlarge | x86\$164 | | r6in.metal | x86\$164 | | r7a.12xlarge | x86\$164 | | r7a.16xlarge | x86\$164 | | r7a.24xlarge | x86\$164 | | r7a.32xlarge | x86\$164 | | r7a.48xlarge | x86\$164 | | r7a.metal-48xl | x86\$164 | | r7g.12xlarge | arm64 | | r7g.16xlarge | arm64 | | r7g.metal | arm64 | | r7gd.12xlarge | arm64 | | r7gd.16xlarge | arm64 | | r7gd.metal | arm64 | | r7i.12xlarge | x86\$164 | | r7i.16xlarge | x86\$164 | | r7i.24xlarge | x86\$164 | | r7i.48xlarge | x86\$164 | | r7i.metal-24xl | x86\$164 | | r7i.metal-48xl | x86\$164 | | r7iz.8xlarge | x86\$164 | | r7iz.12xlarge | x86\$164 | | r7iz.16xlarge | x86\$164 | | r7iz.32xlarge | x86\$164 | | r7iz.metal-16xl | x86\$164 | | r7iz.metal-32xl | x86\$164 | | r8a.16xlarge | x86\$164 | | r8a.24xlarge | x86\$164 | | r8a.48xlarge | x86\$164 | | r8a.metal-24xl | x86\$164 | | r8a.metal-48xl | x86\$164 | | r8g.12xlarge | arm64 | | r8g.16xlarge | arm64 | | r8g.24xlarge | arm64 | | r8g.48xlarge | arm64 | | r8g.metal-24xl | arm64 | | r8g.metal-48xl | arm64 | | r8gb.8xlarge | arm64 | | r8gb.12xlarge | arm64 | | r8gb.16xlarge | arm64 | | r8gb.24xlarge | arm64 | | r8gb.48xlarge | arm64 | | r8gb.metal-24xl | arm64 | | r8gb.metal-48xl | arm64 | | r8gd.12xlarge | arm64 | | r8gd.16xlarge | arm64 | | r8gd.24xlarge | arm64 | | r8gd.48xlarge | arm64 | | r8gd.metal-24xl | arm64 | | r8gd.metal-48xl | arm64 | | r8gn.8xlarge | arm64 | | r8gn.12xlarge | arm64 | | r8gn.16xlarge | arm64 | | r8gn.24xlarge | arm64 | | r8gn.48xlarge | arm64 | | r8gn.metal-24xl | arm64 | | r8gn.metal-48xl | arm64 | | r8i.24xlarge | x86\$164 | | r8i.32xlarge | x86\$164 | | r8i.48xlarge | x86\$164 | | r8i.96xlarge | x86\$164 | | r8i.metal-48xl | x86\$164 | | r8i.metal-96xl | x86\$164 | | r8id.24xlarge | x86\$164 | | r8id.32xlarge | x86\$164 | | r8id.48xlarge | x86\$164 | | r8id.96xlarge | x86\$164 | | r8id.metal-48xl | x86\$164 | | r8id.metal-96xl | x86\$164 | | u7i-6tb.112xlarge | x86\$164 | | u7i-8tb.112xlarge | x86\$164 | | u7i-12tb.224xlarge | x86\$164 | | u7in-16tb.224xlarge | x86\$164 | | u7in-24tb.224xlarge | x86\$164 | | u7in-32tb.224xlarge | x86\$164 | | u7inh-32tb.480xlarge | x86\$164 | | x2idn.16xlarge | x86\$164 | | x2idn.24xlarge | x86\$164 | | x2idn.32xlarge | x86\$164 | | x2idn.metal | x86\$164 | | x2iedn.8xlarge | x86\$164 | | x2iedn.16xlarge | x86\$164 | | x2iedn.24xlarge | x86\$164 | | x2iedn.32xlarge | x86\$164 | | x2iedn.metal | x86\$164 | | x8g.12xlarge | arm64 | | x8g.16xlarge | arm64 | | x8g.24xlarge | arm64 | | x8g.48xlarge | arm64 | | x8g.metal-24xl | arm64 | | x8g.metal-48xl | arm64 | | x8aedz.24xlarge | x86\$164 | | x8aedz.metal-24xl | x86\$164 | | x8i.24xlarge | x86\$164 | | x8i.32xlarge | x86\$164 | | x8i.48xlarge | x86\$164 | | x8i.64xlarge | x86\$164 | | x8i.96xlarge | x86\$164 | | x8i.metal-48xl | x86\$164 | | x8i.metal-96xl | x86\$164 | ------ #### [ Accelerated computing ] | Instance type | Architecture | | --- | --- | | g6.48xlarge | x86\$164 | | g6e.12xlarge | x86\$164 | | g6e.24xlarge | x86\$164 | | g6e.48xlarge | x86\$164 | | g7e.12xlarge | x86\$164 | | g7e.24xlarge | x86\$164 | | g7e.48xlarge | x86\$164 | | p5.4xlarge | x86\$164 | | p5.48xlarge | x86\$164 | | p5e.48xlarge | x86\$164 | | p5en.48xlarge | x86\$164 | | p6-b200.48xlarge | x86\$164 | | p6-b300.48xlarge | x86\$164 | ------ #### [ Storage optimized ] | Instance type | Architecture | | --- | --- | | i4g.4xlarge | arm64 | | i4g.8xlarge | arm64 | | i4g.16xlarge | arm64 | | i4i.8xlarge | x86\$164 | | i4i.12xlarge | x86\$164 | | i4i.16xlarge | x86\$164 | | i4i.24xlarge | x86\$164 | | i4i.32xlarge | x86\$164 | | i4i.metal | x86\$164 | | i7i.12xlarge | x86\$164 | | i7i.16xlarge | x86\$164 | | i7i.24xlarge | x86\$164 | | i7i.48xlarge | x86\$164 | | i7i.metal-24xl | x86\$164 | | i7i.metal-48xl | x86\$164 | | i7ie.12xlarge | x86\$164 | | i7ie.18xlarge | x86\$164 | | i7ie.24xlarge | x86\$164 | | i7ie.48xlarge | x86\$164 | | i7ie.metal-24xl | x86\$164 | | i7ie.metal-48xl | x86\$164 | | i8g.12xlarge | arm64 | | i8g.16xlarge | arm64 | | i8g.24xlarge | arm64 | | i8g.48xlarge | arm64 | | i8g.metal-24xl | arm64 | | i8g.metal-48xl | arm64 | | i8ge.12xlarge | arm64 | | i8ge.18xlarge | arm64 | | i8ge.24xlarge | arm64 | | i8ge.48xlarge | arm64 | | i8ge.metal-24xl | arm64 | | i8ge.metal-48xl | arm64 | | im4gn.4xlarge | arm64 | | im4gn.8xlarge | arm64 | | im4gn.16xlarge | arm64 | ------ ## Prerequisites for Linux instances To ensure that ENA Express can operate effectively, update the settings for your Linux instance as follows. + If your instance uses jumbo frames, run the following command to set your maximum transmission unit (MTU) to `8900`. ``` [ec2-user ~]$ sudo ip link set dev eth0 mtu 8900 ``` + Increase the receiver (Rx) ring size, as follows: ``` [ec2-user ~]$ ethtool -G device rx 8192 ``` + To maximize ENA Express bandwidth, configure your TCP queue limits as follows: 1. Set the TCP small queue limit to 1MB or higher. This increases the amount of data that's queued for transmission on a socket. ``` sudo sh -c 'echo 1048576 > /proc/sys/net/ipv4/tcp_limit_output_bytes' ``` 1. Disable byte queue limits on the eth device if they're enabled for your Linux distribution. This increases data queued for transmission for the device queue. ``` sudo sh -c 'for txq in /sys/class/net/eth0/queues/tx-*; do echo max > ${txq}/byte_queue_limits/limit_min; done' ``` **Note** The ENA driver for the Amazon Linux distribution disables byte queue limits by default. + To minimize ENA Express TCP traffic latency, you can disable the TCP autocorking feature. This might result in a minimal increase in packet overhead: ``` sudo bash -c 'echo 0 > /proc/sys/net/ipv4/tcp_autocorking' ``` ## Tune performance for ENA Express settings on Linux instances To check your Linux instance configuration for optimal ENA Express performance, you can run the following script that's available on the Amazon GitHub repository: [https://github.com/amzn/amzn-ec2-ena-utilities/blob/main/ena-express/check-ena-express-settings.sh](https://github.com/amzn/amzn-ec2-ena-utilities/blob/main/ena-express/check-ena-express-settings.sh) The script runs a series of tests and suggests both recommended and required configuration changes. # Review ENA Express settings for your EC2 instance Review instance settings You can verify the ENA Express settings by instance or by network interface. To update the ENA Express settings, see [Configure ENA Express settings for your EC2 instance](ena-express-configure.md). ------ #### [ Console ] **To view ENA Express settings for a network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the left navigation pane, choose **Network interfaces**. 1. Select a network interface to see the details for that instance. You can choose the **Network interface ID** link to open the detail page, or you can select the checkbox on the left side of the list to view details in the detail pane at the bottom of the page. 1. In the **Network interface attachment** section on the **Details** tab or detail page, review settings for **ENA Express** and **ENA Express UDP**. **To view ENA Express settings for an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the left navigation pane, choose **Instances**. 1. Select an instance to see the details for that instance. You can choose the **Instance ID** link to open the detail page, or you can select the checkbox on the left side of the list to view details in the detail pane at the bottom of the page. 1. In the **Network interfaces** section on the **Networking** tab, scroll right to review settings for **ENA Express** and **ENA Express UDP**. ------ #### [ AWS CLI ] **To get the ENA Express settings for an instance** Use the [https://docs.aws.amazon.com/cli/latest/reference/describe-instances.html](https://docs.aws.amazon.com/cli/latest/reference/describe-instances.html) command. This command example returns a list of ENA Express configurations for the network interfaces attached to each of the running instances that are specified by the `--instance-ids` parameter. ``` aws ec2 describe-instances \ --instance-ids i-1234567890abcdef0 i-0598c7d356eba48d7 \ --query 'Reservations[*].Instances[*].[InstanceId, NetworkInterfaces[*].Attachment.EnaSrdSpecification]' ``` The following is example output. ``` [ [ [ "i-1234567890abcdef0", [ { "EnaSrdEnabled": true, "EnaSrdUdpSpecification": { "EnaSrdUdpEnabled": false } } ] ] ], [ [ "i-0598c7d356eba48d7", [ { "EnaSrdEnabled": true, "EnaSrdUdpSpecification": { "EnaSrdUdpEnabled": false } } ] ] ] ] ``` **To get the ENA Express settings for a network interface** Use the [https://docs.aws.amazon.com/cli/latest/reference/describe-network-interfaces.html](https://docs.aws.amazon.com/cli/latest/reference/describe-network-interfaces.html) command. ``` aws ec2 describe-network-interfaces \ --network-interface-ids eni-1234567890abcdef0 \ --query NetworkInterfaces[].[NetworkInterfaceId,Attachment.EnaSrdSpecification] ``` The following is example output. ``` [ [ "eni-1234567890abcdef0", { "EnaSrdEnabled": true, "EnaSrdUdpSpecification": { "EnaSrdUdpEnabled": false } } ] ] ``` ------ #### [ PowerShell ] **To get the ENA Express settings for a network interface** Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2NetworkInterface.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2NetworkInterface.html) cmdlet. ``` Get-EC2NetworkInterface ` -NetworkInterfaceId eni-1234567890abcdef0 | ` Select-Object ` Association, NetworkInterfaceId, OwnerId, @{Name = 'AttachTime'; Expression = { $_.Attachment.AttachTime } }, @{Name = 'AttachmentId'; Expression = { $_.Attachment.AttachmentId } }, @{Name = 'DeleteOnTermination'; Expression = { $_.Attachment.DeleteOnTermination } }, @{Name = 'NetworkCardIndex'; Expression = { $_.Attachment.NetworkCardIndex } }, @{Name = 'InstanceId'; Expression = { $_.Attachment.InstanceId } }, @{Name = 'InstanceOwnerId'; Expression = { $_.Attachment.InstanceOwnerId } }, @{Name = 'Status'; Expression = { $_.Attachment.Status } }, @{Name = 'EnaSrdEnabled'; Expression = { $_.Attachment.EnaSrdSpecification.EnaSrdEnabled } }, @{Name = 'EnaSrdUdpEnabled'; Expression = { $_.Attachment.EnaSrdSpecification.EnaSrdUdpSpecification.EnaSrdUdpEnabled } } ``` The following is example output. ``` Association : NetworkInterfaceId : eni-0d1234e5f6a78901b OwnerId : 111122223333 AttachTime : 6/11/2022 1:13:11 AM AttachmentId : eni-attach-0d1234e5f6a78901b DeleteOnTermination : True NetworkCardIndex : 0 InstanceId : i-1234567890abcdef0 InstanceOwnerId : 111122223333 Status : attached EnaSrdEnabled : True EnaSrdUdpEnabled : False ``` ------ # Configure ENA Express settings for your EC2 instance Configure instance settings You can configure ENA Express for supported EC2 instance types without needing to install any additional software. For more information, see [Supported instance types for ENA Express](ena-express.md#ena-express-supported-instance-types). ------ #### [ Console ] **To manage ENA Express for a network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the left navigation pane, choose **Network interfaces**. 1. Select a network interface that is attached to an instance. You can choose the **Network interface ID** link to open the detail page, or you can select the checkbox on the left side of the list. 1. Choose **Manage ENA Express** from the **Action** menu at the top right side of the page. This opens the **Manage ENA Express** dialog, with the selected network interface ID and current settings displayed. If the network interface you selected is not attached to an instance, this action does not appear in the menu. 1. To use **ENA Express**, select the **Enable** checkbox. 1. When ENA Express is enabled, you can configure UDP settings. To use **ENA Express UDP**, select the **Enable** checkbox. 1. To save your settings, choose **Save**. **To manage ENA Express for an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the left navigation pane, choose **Instances**. 1. Select the instance that you want to manage. You can choose the **Instance ID** to open the detail page, or you can select the checkbox on the left side of the list. 1. Select the **Network interface** to configure for your instance. 1. Choose **Manage ENA Express** from the **Action** menu at the top right side of the page. 1. To configure ENA Express for a network interface that's attached to your instance, select it from the **Network interface** list. 1. To use **ENA Express** for the selected network interface attachment, select the **Enable** checkbox. 1. When ENA Express is enabled, you can configure UDP settings. To use **ENA Express UDP**, select the **Enable** checkbox. 1. To save your settings, choose **Save**. **To configure ENA Express when you attach a network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the left navigation pane, choose **Network interfaces**. 1. Select a network interface that is not attached to an instance (**Status** is **Available**). You can choose the **Network interface ID** link to open the detail page, or you can select the checkbox on the left side of the list. 1. Select the **Instance** that you'll attach to. 1. To use **ENA Express** after you attach the network interface to the instance, select the **Enable** checkbox. 1. When ENA Express is enabled, you can configure UDP settings. To use **ENA Express UDP**, select the **Enable** checkbox. 1. To attach the network interface to the instance and save your ENA Express settings, choose **Attach**. ------ #### [ AWS CLI ] **To configure ENA Express when you attach a network interface** Use the [https://docs.aws.amazon.com/cli/latest/reference/;attach-network-interface.html](https://docs.aws.amazon.com/cli/latest/reference/;attach-network-interface.html) command, as shown in the following examples. **Example 1: Use ENA Express for TCP traffic, but not UDP traffic** This example configures `EnaSrdEnabled` as `true`, and allows `EnaSrdUdpEnabled` to default to `false`. ``` aws ec2 attach-network-interface \ --network-interface-id eni-1234567890abcdef0 \ --instance-id i-1234567890abcdef0 \ --device-index 1 \ --ena-srd-specification 'EnaSrdEnabled=true' ``` **Example 2: Use ENA Express for both TCP traffic and UDP traffic** This example configures both `EnaSrdEnabled` and `EnaSrdUdpEnabled` as `true`. ``` aws ec2 attach-network-interface \ --network-interface-id eni-1234567890abcdef0 \ --instance-id i-1234567890abcdef0 \ --device-index 1 \ --ena-srd-specification 'EnaSrdEnabled=true,EnaSrdUdpSpecification={EnaSrdUdpEnabled=true}' ``` **To update ENA Express settings for a network interface attachment** Use the [https://docs.aws.amazon.com/cli/latest/reference/modify-network-interface-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/modify-network-interface-attribute.html) command as shown in the following examples. **Example 1: Use ENA Express for TCP traffic, but not UDP traffic** This example configures `EnaSrdEnabled` as `true`, and allows `EnaSrdUdpEnabled` to default to `false` if it has never been set previously. ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --ena-srd-specification 'EnaSrdEnabled=true' ``` **Example 2: Use ENA Express for both TCP traffic and UDP traffic** This example configures both `EnaSrdEnabled` and `EnaSrdUdpEnabled` as `true`. ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --ena-srd-specification 'EnaSrdEnabled=true,EnaSrdUdpSpecification={EnaSrdUdpEnabled=true}' ``` **Example 3: Stop using ENA Express for UDP traffic** This example configures `EnaSrdUdpEnabled` as `false`. ``` aws ec2 modify-network-interface-attribute \ --network-interface-id eni-1234567890abcdef0 \ --ena-srd-specification 'EnaSrdUdpSpecification={EnaSrdUdpEnabled=false}' ``` ------ #### [ PowerShell ] **To configure ENA Express when you attach a network interface** Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Add-EC2NetworkInterface.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Add-EC2NetworkInterface.html) cmdlet as shown in the following examples. **Example 1: Use ENA Express for TCP traffic, but not UDP traffic** This example configures `EnaSrdEnabled` as `true`, and allows `EnaSrdUdpEnabled` to default to `false`. ``` Add-EC2NetworkInterface ` -NetworkInterfaceId eni-1234567890abcdef0 ` -InstanceId i-1234567890abcdef0 ` -DeviceIndex 1 ` -EnaSrdSpecification_EnaSrdEnabled $true ``` **Example 2: Use ENA Express for both TCP traffic and UDP traffic** This example configures both `EnaSrdEnabled` and `EnaSrdUdpEnabled` as `true`. ``` Add-EC2NetworkInterface ` -NetworkInterfaceId eni-1234567890abcdef0 ` -InstanceId i-1234567890abcdef0 ` -DeviceIndex 1 ` -EnaSrdSpecification_EnaSrdEnabled $true ` -EnaSrdUdpSpecification_EnaSrdUdpEnabled $true ``` **To configure ENA Express settings for your network interface attachment** Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2NetworkInterfaceAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2NetworkInterfaceAttribute.html) cmdlet as shown in the following examples. **Example 1: Use ENA Express for TCP traffic, but not UDP traffic** This example configures `EnaSrdEnabled` as `true`, and allows `EnaSrdUdpEnabled` to default to `false` if it has never been set previously. ``` Edit-EC2NetworkInterfaceAttribute ` -NetworkInterfaceId eni-1234567890abcdef0 ` -EnaSrdSpecification_EnaSrdEnabled $true ; Get-EC2NetworkInterface -NetworkInterfaceId eni-0123f4567890a1b23 | ` Select-Object ` NetworkInterfaceId, @{Name = 'EnaSrdEnabled'; Expression = { $_.Attachment.EnaSrdSpecification.EnaSrdEnabled }}, @{Name = 'EnaSrdUdpEnabled'; Expression = { $_.Attachment.EnaSrdSpecification.EnaSrdUdpSpecification.EnaSrdUdpEnabled }} | ` Format-List ``` **Example 2: Use ENA Express for both TCP traffic and UDP traffic** This example configures both `EnaSrdEnabled` and `EnaSrdUdpEnabled` as `true`. ``` Edit-EC2NetworkInterfaceAttribute ` -NetworkInterfaceId eni-1234567890abcdef0 ` -EnaSrdSpecification_EnaSrdEnabled $true ` -EnaSrdSpecification_EnaSrdUdpSpecification_EnaSrdUdpEnabled $true ; Get-EC2NetworkInterface -NetworkInterfaceId eni-1234567890abcdef0 | ` Select-Object ` NetworkInterfaceId, @{Name = 'EnaSrdEnabled'; Expression = { $_.Attachment.EnaSrdSpecification.EnaSrdEnabled }}, @{Name = 'EnaSrdUdpEnabled'; Expression = { $_.Attachment.EnaSrdSpecification.EnaSrdUdpSpecification.EnaSrdUdpEnabled }} | ` Format-List ``` **Example 3: Stop using ENA Express for UDP traffic** This example configures `EnaSrdUdpEnabled` as `false`. ``` Edit-EC2NetworkInterfaceAttribute ` -NetworkInterfaceId eni-0123f4567890a1b23 ` -EnaSrdSpecification_EnaSrdUdpSpecification_EnaSrdUdpEnabled $false ; Get-EC2NetworkInterface -NetworkInterfaceId eni-0123f4567890a1b23 | ` Select-Object ` NetworkInterfaceId, @{Name = 'EnaSrdEnabled'; Expression = { $_.Attachment.EnaSrdSpecification.EnaSrdEnabled }}, @{Name = 'EnaSrdUdpEnabled'; Expression = { $_.Attachment.EnaSrdSpecification.EnaSrdUdpSpecification.EnaSrdUdpEnabled }} | ` Format-List ``` ------ ## Configure ENA Express at launch You can use one of the following methods to configure ENA Express directly when you launch an instance. The specified links refer you to the AWS Management Console instructions for these methods. + **Launch instance wizard** – You can configure ENA Express at launch with the launch instance wizard. For more information, see **Advanced network configuration** in the [Network settings](ec2-instance-launch-parameters.md#liw-network-settings) for the launch instance wizard. + **Launch template** – You can configure ENA Express at launch when you use a launch template. For more information, see the [Create an Amazon EC2 launch template](create-launch-template.md) page, then expand the **Network settings** section and review the **Advanced network configuration**. # Enhanced networking with the Intel 82599 VF interface Intel 82599 VF For [Xen-based instances](instance-types.md#instance-hypervisor-type), the Intel 82599 Virtual Function (VF) interface provides enhanced networking capabilities. The interface uses the Intel `ixgbevf` driver. The following tabs show how to verify the network adapter driver that's installed for your instance operating system. ------ #### [ Linux ] **Linux network interface driver** Use the following command to verify that the module is being used on a particular interface, substituting the interface name that you want to check. If you are using a single interface (default), this is `eth0`. If the operating system supports [predictable network names](#predictable-network-names-sriov), this could be a name like `ens5`. In the following example, the `ixgbevf` module is not loaded, because the listed driver is `vif`. ``` [ec2-user ~]$ ethtool -i eth0 driver: vif version: firmware-version: bus-info: vif-0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ``` In this example, the `ixgbevf` module is loaded. This instance has enhanced networking properly configured. ``` [ec2-user ~]$ ethtool -i eth0 driver: ixgbevf version: 4.0.3 firmware-version: N/A bus-info: 0000:00:03.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: yes supports-priv-flags: no ``` ------ #### [ Windows ] **Windows network adapter** To verify that the driver is installed, connect to your instance and open Device Manager. You should see `Intel(R) 82599 Virtual Function` listed under **Network adapters**. ------ **Topics** + [ ## Prepare your instance for enhanced networking ](#ixgbevf-requirements) + [ ## Test whether enhanced networking is enabled ](#test-enhanced-networking) + [ ## Enable enhanced networking on your instance ](#enable-enhanced-networking) + [ ## Troubleshoot connectivity issues ](#enhanced-networking-troubleshooting) ## Prepare your instance for enhanced networking To prepare for enhanced networking using the Intel 82599 VF interface, set up your instance as follows: + Verify that the instance type is one of the following: C3, C4, D2, I2, M4 (excluding `m4.16xlarge`), and R3. + Ensure that the instance has internet connectivity. + If you have important data on the instance that you want to preserve, you should back that data up now by creating an AMI from your instance. Updating kernels and kernel modules, as well as enabling the `sriovNetSupport` attribute, might render incompatible instances or operating systems unreachable. If you have a recent backup, your data will still be retained if this happens. + **Linux instances** – Launch the instance from an HVM AMI using Linux kernel version of 2.6.32 or later. The latest Amazon Linux HVM AMIs have the modules required for enhanced networking installed and have the required attributes set. Therefore, if you launch an Amazon EBS–backed, enhanced networking–supported instance using a current Amazon Linux HVM AMI, enhanced networking is already enabled for your instance. **Warning** Enhanced networking is supported only for HVM instances. Enabling enhanced networking with a PV instance can make it unreachable. Setting this attribute without the proper module or module version can also make your instance unreachable. + **Windows instances** – Launch the instance from a 64-bit HVM AMI. You can't enable enhanced networking on Windows Server 2008. Enhanced networking is already enabled for Windows Server 2012 R2 and Windows Server 2016 and later AMIs. Windows Server 2012 R2 includes Intel driver 1.0.15.3 and we recommend that you upgrade that driver to the latest version using the Pnputil.exe utility. + Use [AWS CloudShell](https://console.aws.amazon.com/cloudshell) from the AWS Management Console, or install and configure the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) or the [AWS Tools for Windows PowerShell](https://docs.aws.amazon.com/powershell/latest/userguide/) on any computer you choose, preferably your local desktop or laptop. For more information, see [Access Amazon EC2](concepts.md#access-ec2) or the [AWS CloudShell User Guide](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html). Enhanced networking cannot be managed from the Amazon EC2 console. ## Test whether enhanced networking is enabled Verify that the `sriovNetSupport` attribute is set on the instance or the image. ------ #### [ AWS CLI ] **To check the instance attribute (sriovNetSupport)** Use the following [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-attribute.html) command. If the attribute is set, the value is `simple`. ``` aws ec2 describe-instance-attribute \ --instance-id i-1234567890abcdef0 \ --attribute sriovNetSupport ``` **To check the image attribute (sriovNetSupport)** Use the following [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html) command. If the attribute is set, the value is `simple`. ``` aws ec2 describe-images \ --image-id ami-0abcdef1234567890 \ --query "Images[].SriovNetSupport" ``` ------ #### [ PowerShell ] **To check the instance attribute (sriovNetSupport)** Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceAttribute.html) cmdlet. If the attribute is set, the value is `simple`. ``` Get-EC2InstanceAttribute ` -InstanceId i-1234567890abcdef0 ` -Attribute sriovNetSupport ``` **To check the image attribute (sriovNetSupport)** Use the following [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html) command. If the attribute is set, the value is `simple`. ``` (Get-EC2Image -ImageId ami-0abcdef1234567890).SriovNetSupport ``` ------ ## Enable enhanced networking on your instance The procedure that you use depends on the operating system of the instance. **Warning** There is no way to disable the enhanced networking attribute after you've enabled it. ### Amazon Linux The latest Amazon Linux HVM AMIs have the `ixgbevf` module required for enhanced networking installed and have the required `sriovNetSupport` attribute set. Therefore, if you launch an instance type using a current Amazon Linux HVM AMI, enhanced networking is already enabled for your instance. For more information, see [Test whether enhanced networking is enabled](#test-enhanced-networking). If you launched your instance using an older Amazon Linux AMI and it does not have enhanced networking enabled already, use the following procedure to enable enhanced networking. **To enable enhanced networking** 1. Connect to your instance. 1. From the instance, run the following command to update your instance with the newest kernel and kernel modules, including `ixgbevf`: ``` [ec2-user ~]$ sudo yum update ``` 1. From your local computer, reboot your instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/reboot-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/reboot-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Restart-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Restart-EC2Instance.html) (AWS Tools for Windows PowerShell). 1. Connect to your instance again and verify that the `ixgbevf` module is installed and at the minimum recommended version using the **modinfo ixgbevf** command from [Test whether enhanced networking is enabled](#test-enhanced-networking). 1. [EBS-backed instance] From your local computer, stop the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html) (AWS Tools for Windows PowerShell). [Instance store-backed instance] You can't stop the instance to modify the attribute. Instead, skip to the next procedure. 1. From your local computer, enable the enhanced networking attribute using one of the following commands: ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html) command as follows. ``` aws ec2 modify-instance-attribute \ --instance-id i-1234567890abcdef0 \ --sriov-net-support simple ``` ------ #### [ PowerShell ] Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html) cmdlet as follows. ``` Edit-EC2InstanceAttribute ` -InstanceId i-1234567890abcdef0 ` -SriovNetSupport "simple" ``` ------ 1. (Optional) Create an AMI from the instance, as described in [Create an Amazon EBS-backed AMI](creating-an-ami-ebs.md). The AMI inherits the enhanced networking attribute from the instance. Therefore, you can use this AMI to launch another instance with enhanced networking enabled by default. 1. From your local computer, start the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html) (AWS Tools for Windows PowerShell). 1. Connect to your instance and verify that the `ixgbevf` module is installed and loaded on your network interface using the **ethtool -i eth*n*** command from [Test whether enhanced networking is enabled](#test-enhanced-networking). **To enable enhanced networking (instance store-backed instances)** Follow the previous procedure until the step where you stop the instance. Create a new AMI as described in [Create an Amazon S3-backed AMI](creating-an-ami-instance-store.md), making sure to enable the enhanced networking attribute when you register the AMI. ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/register-image.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/register-image.html) command as follows. ``` aws ec2 register-image --sriov-net-support simple ... ``` ------ #### [ PowerShell ] Use [https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Image.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Image.html) as follows. ``` Register-EC2Image -SriovNetSupport "simple" ... ``` ------ ### Ubuntu Before you begin, [check if enhanced networking is already enabled](#test-enhanced-networking) on your instance. The Quick Start Ubuntu HVM AMIs include the necessary drivers for enhanced networking. If you have a version of `ixgbevf` earlier than 2.16.4, you can install the `linux-aws` kernel package to get the latest enhanced networking drivers. The following procedure provides the general steps for compiling the `ixgbevf` module on an Ubuntu instance. **To install the `linux-aws` kernel package** 1. Connect to your instance. 1. Update the package cache and packages. ``` ubuntu:~$ sudo apt-get update && sudo apt-get upgrade -y linux-aws ``` **Important** If during the update process, you are prompted to install `grub`, use `/dev/xvda` to install `grub`, and then choose to keep the current version of `/boot/grub/menu.lst`. ### Other Linux distributions Before you begin, [check if enhanced networking is already enabled](#test-enhanced-networking) on your instance. The latest Quick Start HVM AMIs include the necessary drivers for enhanced networking, therefore you do not need to perform additional steps. The following procedure provides the general steps if you need to enable enhanced networking with the Intel 82599 VF interface on a Linux distribution other than Amazon Linux or Ubuntu. For more information, such as detailed syntax for commands, file locations, or package and tool support, see the specific documentation for your Linux distribution. **To enable enhanced networking on Linux** 1. Connect to your instance. 1. Download the source for the `ixgbevf` module on your instance from Sourceforge at [https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/](https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/). Versions of `ixgbevf` earlier than 2.16.4, including version 2.14.2, do not build properly on some Linux distributions, including certain versions of Ubuntu. 1. Compile and install the `ixgbevf` module on your instance. **Warning** If you compile the `ixgbevf` module for your current kernel and then upgrade your kernel without rebuilding the driver for the new kernel, your system might revert to the distribution-specific `ixgbevf` module at the next reboot. This could make your system unreachable if the distribution-specific version is incompatible with enhanced networking. 1. Run the **sudo depmod** command to update module dependencies. 1. Update `initramfs` on your instance to ensure that the new module loads at boot time. 1. Determine if your system uses predictable network interface names by default. Systems that use **systemd** or **udev** versions 197 or greater can rename Ethernet devices and they do not guarantee that a single network interface will be named `eth0`. This behavior can cause problems connecting to your instance. For more information and to see other configuration options, see [Predictable Network Interface Names](https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/) on the freedesktop.org website. 1. You can check the **systemd** or **udev** versions on RPM-based systems with the following command: ``` [ec2-user ~]$ rpm -qa | grep -e '^systemd-[0-9]\+\|^udev-[0-9]\+' systemd-208-11.el7_0.2.x86_64 ``` In the above Red Hat Enterprise Linux 7 example, the **systemd** version is 208, so predictable network interface names must be disabled. 1. Disable predictable network interface names by adding the `net.ifnames=0` option to the `GRUB_CMDLINE_LINUX` line in `/etc/default/grub`. ``` [ec2-user ~]$ sudo sed -i '/^GRUB\_CMDLINE\_LINUX/s/\"$/\ net\.ifnames\=0\"/' /etc/default/grub ``` 1. Rebuild the grub configuration file. ``` [ec2-user ~]$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg ``` 1. [EBS-backed instance] From your local computer, stop the instance using the Amazon EC2 console or one of the following commands: [stop-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/stop-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Stop-EC2Instance.html) (AWS Tools for Windows PowerShell). [Instance store-backed instance] You can't stop the instance to modify the attribute. Instead, skip to the next procedure. 1. From your local computer, enable the enhanced networking attribute using one of the following commands: ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html) command as follows. ``` aws ec2 modify-instance-attribute \ --instance-id i-1234567890abcdef0 -\ -sriov-net-support simple ``` ------ #### [ PowerShell ] Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html) cmdlet as follows. ``` Edit-EC2InstanceAttribute ` -InstanceId i-1234567890abcdef0 ` -SriovNetSupport "simple" ``` ------ 1. (Optional) Create an AMI from the instance, as described in [Create an Amazon EBS-backed AMI](creating-an-ami-ebs.md). The AMI inherits the enhanced networking attribute from the instance. Therefore, you can use this AMI to launch another instance with enhanced networking enabled by default. If your instance operating system contains an `/etc/udev/rules.d/70-persistent-net.rules` file, you must delete it before creating the AMI. This file contains the MAC address for the Ethernet adapter of the original instance. If another instance boots with this file, the operating system will be unable to find the device and `eth0` might fail, causing boot issues. This file is regenerated at the next boot cycle, and any instances launched from the AMI create their own version of the file. 1. From your local computer, start the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html) (AWS Tools for Windows PowerShell). 1. (Optional) Connect to your instance and verify that the module is installed. **To enable enhanced networking (instance store–backed instances)** Follow the previous procedure until the step where you stop the instance. Create a new AMI as described in [Create an Amazon S3-backed AMI](creating-an-ami-instance-store.md), making sure to enable the enhanced networking attribute when you register the AMI. ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/register-image.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/register-image.html) command as follows. ``` aws ec2 register-image --sriov-net-support simple ... ``` ------ #### [ PowerShell ] Use [https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Image.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Register-EC2Image.html) as follows. ``` Register-EC2Image -SriovNetSupport "simple" ... ``` ------ ### Windows If you launched your instance and it does not have enhanced networking enabled already, you must download and install the required network adapter driver on your instance, and then set the `sriovNetSupport` instance attribute to activate enhanced networking. You can only enable this attribute on supported instance types. For more information, see [Enhanced networking on Amazon EC2 instances](enhanced-networking.md). **Important** To view the latest driver updates in the Windows AMIs, see [Windows AMI version history](https://docs.aws.amazon.com/ec2/latest/windows-ami-reference/ec2-windows-ami-version-history.html) in the *AWS Windows AMI Reference*. **To enable enhanced networking** 1. Connect to your instance and log in as the local administrator. 1. [Windows Server 2016 and later] Run the following EC2 Launch PowerShell script to configure the instance after the driver is installed. ``` PS C:\> C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule ``` **Important** The administrator password will reset when you enable the initialize instance EC2 Launch script. You can modify the configuration file to disable the administrator password reset by specifying it in the settings for the initialization tasks. 1. From the instance, download the Intel network adapter driver for your operating system: + **Windows Server 2022** Visit the [ download page](https://www.intel.com/content/www/us/en/download/706171/intel-network-adapter-driver-for-windows-server-2022.html) and download `Wired_driver_version_x64.zip`. + **Windows Server 2019** including for Server version 1809 and later\$1 Visit the [ download page](https://www.intel.com/content/www/us/en/download/19372/intel-network-adapter-driver-for-windows-server-2019.html) and download `Wired_driver_version_x64.zip`. + **Windows Server 2016** including for Server version 1803 and earlier\$1 Visit the [ download page](https://www.intel.com/content/www/us/en/download/18737/intel-network-adapter-driver-for-windows-server-2016.html) and download `Wired_driver_version_x64.zip`. + **Windows Server 2012 R2** Visit the [ download page](https://www.intel.com/content/www/us/en/download/17480/intel-network-adapter-driver-for-windows-server-2012-r2.html) and download `Wired_driver_version_x64.zip`. + **Windows Server 2012** Visit the [ download page](https://www.intel.com/content/www/us/en/download/16789/intel-network-adapter-driver-for-windows-server-2012.html) and download `Wired_driver_version_x64.zip`. + **Windows Server 2008 R2** Visit the [ download page](https://www.intel.com/content/www/us/en/download/15590/intel-network-adapter-driver-for-windows-7-final-release.html) and download `PROWinx64Legacy.exe`. \$1Server versions 1803 and earlier as well as 1809 and later are not specifically addressed on the Intel Drivers and Software pages. 1. Install the Intel network adapter driver for your operating system. + **Windows Server 2008 R2** 1. In the **Downloads** folder, locate the `PROWinx64Legacy.exe` file and rename it to `PROWinx64Legacy.zip`. 1. Extract the contents of the `PROWinx64Legacy.zip` file. 1. Open the command line, navigate to the extracted folder, and run the following command to use the `pnputil` utility to add and install the INF file in the driver store. ``` C:\> pnputil -a PROXGB\Winx64\NDIS62\vxn62x64.inf ``` + **Windows Server 2022, Windows Server 2019, Windows Server 2016, Windows Server 2012 R2, and Windows Server 2012** 1. In the **Downloads** folder, extract the contents of the `Wired_driver_version_x64.zip` file. 1. Open the command line, navigate to the extracted folder, and run one of the following commands to use the `pnputil` utility to add and install the INF file in the driver store. + Windows Server 2022 ``` pnputil -i -a PROXGB\Winx64\NDIS68\vxn68x64.inf ``` + Windows Server 2019 ``` pnputil -i -a PROXGB\Winx64\NDIS68\vxn68x64.inf ``` + Windows Server 2016 ``` pnputil -i -a PROXGB\Winx64\NDIS65\vxn65x64.inf ``` + Windows Server 2012 R2 ``` pnputil -i -a PROXGB\Winx64\NDIS64\vxn64x64.inf ``` + Windows Server 2012 ``` pnputil -i -a PROXGB\Winx64\NDIS63\vxn63x64.inf ``` 1. From your local computer, enable the enhanced networking attribute using one of the following commands: ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html) command as follows. ``` aws ec2 modify-instance-attribute \ --instance-id i-1234567890abcdef0 \ --sriov-net-support simple ``` ------ #### [ PowerShell ] Use the [https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstanceAttribute.html) cmdlet as follows. ``` Edit-EC2InstanceAttribute ` -InstanceId i-1234567890abcdef0 ` -SriovNetSupport "simple" ``` ------ 1. (Optional) Create an AMI from the instance, as described in [Create an Amazon EBS-backed AMI](creating-an-ami-ebs.md). The AMI inherits the enhanced networking attribute from the instance. Therefore, you can use this AMI to launch another instance with enhanced networking enabled by default. 1. From your local computer, start the instance using the Amazon EC2 console or one of the following commands: [https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/start-instances.html) (AWS CLI) or [https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html](https://docs.aws.amazon.com/powershell/latest/reference/items/Start-EC2Instance.html) (AWS Tools for Windows PowerShell). ## Troubleshoot connectivity issues If you lose connectivity while enabling enhanced networking, the `ixgbevf` module might be incompatible with the kernel. Try installing the version of the `ixgbevf` module included with the distribution of Linux for your instance. If you enable enhanced networking for a PV instance or AMI, this can make your instance unreachable. For more information, see [How do I turn on and configure enhanced networking on my EC2 instances?](https://repost.aws/knowledge-center/enable-configure-enhanced-networking) # Monitor network performance for ENA settings on your EC2 instance Monitor network performance The Elastic Network Adapter (ENA) driver publishes network performance metrics from the instances where they are enabled. You can use these metrics to troubleshoot instance performance issues, choose the right instance size for a workload, plan scaling activities proactively, and benchmark applications to determine whether they maximize the performance available on an instance. Amazon EC2 defines network maximums at the instance level to ensure a high-quality networking experience, including consistent network performance across instance sizes. AWS provides maximums for the following for each instance: + **Bandwidth capability** – Each EC2 instance has a maximum bandwidth for aggregate inbound and outbound traffic, based on instance type and size. Some instances use a network I/O credit mechanism to allocate network bandwidth based on average bandwidth utilization. Amazon EC2 also has maximum bandwidth for traffic to Direct Connect and the internet. For more information, see [Amazon EC2 instance network bandwidth](ec2-instance-network-bandwidth.md). + **Packet-per-second (PPS) performance** – Each EC2 instance has a maximum PPS performance, based on instance type and size. + **Connections tracked** – The security group tracks each connection established to ensure that return packets are delivered as expected. There is a maximum number of connections that can be tracked per instance. For more information, see [Amazon EC2 security group connection tracking](security-group-connection-tracking.md) + **Link-local service access** – Amazon EC2 provides a maximum PPS per network interface for traffic to local proxy services such as the Amazon DNS service, the Instance Metadata Service, and the Amazon Time Sync Service. When the network traffic for an instance exceeds a maximum, AWS shapes the traffic that exceeds the maximum by queueing and then dropping network packets. You can monitor when traffic exceeds a maximum using the network performance metrics. These metrics inform you, in real time, of impact to network traffic and possible network performance issues. **Topics** + [ ## Requirements ](#network-performance-metrics-requirements) + [ ## Metrics for the ENA driver ](#network-performance-metrics) + [ ## View the network performance metrics for your instance ](#view-network-performance-metrics) + [ ## Metrics for ENA Express ](#network-performance-metrics-ena-express) + [ ## Network performance metrics with the DPDK driver for ENA ](#network-performance-metrics-dpdk) + [ ## Metrics on instances running FreeBSD ](#network-performance-metrics-freebsd) ## Requirements **Linux instances** + Install ENA driver version 2.2.10 or later. To verify the installed version, use the **ethtool** command. In the following example, the version meets the minimum requirement. ``` [ec2-user ~]$ ethtool -i eth0 | grep version version: 2.2.10 ``` To upgrade your ENA driver, see [Enhanced networking](enhanced-networking-ena.md). + To import these metrics to Amazon CloudWatch, install the CloudWatch agent. For more information, see [Collect network performance metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-network-performance.html) in the *Amazon CloudWatch User Guide*. + To support the `conntrack_allowance_available` metric, install ENA driver version 2.8.1 or later. + To override the egress fragment PPS limit of 1024, install ENA driver version 2.13.3 or later. **Windows instances** + Install ENA driver version 2.2.2 or later. To verify the installed version, use Device Manager as follows. 1. Open Device Manager by running `devmgmt.msc`. 1. Expand **Network Adapters**. 1. Choose **Amazon Elastic Network Adapter**, **Properties**. 1. On the **Driver** tab, locate **Driver Version**. To upgrade your ENA driver, see [Enhanced networking](enhanced-networking-ena.md). + To import these metrics to Amazon CloudWatch, install the CloudWatch agent. For more information, see [Collect advanced network metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-network-performance.html) in the *Amazon CloudWatch User Guide*. ## Metrics for the ENA driver The ENA driver delivers the following metrics to the instance in real time. They provide the cumulative number of packets queued or dropped on each network interface since the last driver reset. | Metric | Description | Supported on | | --- | --- | --- | | bw\$1in\$1allowance\$1exceeded | The number of packets queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance. | All instance types | | bw\$1out\$1allowance\$1exceeded | The number of packets queued or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance. | All instance types | | conntrack\$1allowance\$1exceeded | The number of packets dropped because connection tracking exceeded the maximum for the instance and new connections could not be established. This can result in packet loss for traffic to or from the instance. | All instance types | | conntrack\$1allowance\$1available | The number of tracked connections that can be established by the instance before hitting the Connections Tracked allowance of that instance type. | [Nitro-based instances](instance-types.md#instance-hypervisor-type) only | | linklocal\$1allowance\$1exceeded | The number of packets dropped because the PPS of the traffic to local proxy services exceeded the maximum for the network interface. This impacts traffic to the Amazon DNS service, the Instance Metadata Service, and the Amazon Time Sync Service, but does not impact traffic to custom DNS resolvers. | All instance types | | pps\$1allowance\$1exceeded | The number of packets queued or dropped because the bidirectional PPS exceeded the maximum for the instance. \$1 | All instance types | \$1 Depending on the fragment proxy mode setting for ENA Linux driver v2.13.3 or later, this limit can also include egress fragment drops that exceed 1024 PPS for the network interface. If fragment proxy mode is enabled for the Linux driver, egress fragment drops bypass the 1024 PPS limit that usually applies and are counted within standard PPS allowances. Fragment proxy mode is disabled by default. ## View the network performance metrics for your instance The procedure that you use depends on the operating system of the instance. ### Linux instances You can publish metrics to your favorite tools to visualize the metric data. For example, you can publish the metrics to Amazon CloudWatch using the CloudWatch agent. The agent enables you to select individual metrics and control publication. You can also use the **ethtool** to retrieve the metrics for each network interface, such as eth0, as follows. ``` [ec2-user ~]$ ethtool -S eth0 bw_in_allowance_exceeded: 0 bw_out_allowance_exceeded: 0 pps_allowance_exceeded: 0 conntrack_allowance_exceeded: 0 linklocal_allowance_exceeded: 0 conntrack_allowance_available: 136812 ``` ### Windows instances You can view the metrics using any consumer of Windows performance counters. The data can be parsed according to the EnaPerfCounters manifest. This is an XML file that defines the performance counter provider and its countersets. **To install the manifest** If you launched the instance using an AMI that contains ENA driver 2.2.2 or later, or used the install script in the driver package for ENA driver 2.2.2, the manifest is already installed. To install the manifest manually, use the following steps: 1. Remove the existing manifest using the following command: ``` unlodctr /m:EnaPerfCounters.man ``` 1. Copy the manifest file `EnaPerfCounters.man` from the driver installation package to `%SystemRoot%\System32\drivers`. 1. Install the new manifest using the following command: ``` lodctr /m:EnaPerfCounters.man ``` **To view metrics using Performance Monitor** 1. Open Performance Monitor. 1. Press Ctrl\$1N to add new counters. 1. Choose **ENA Packets Shaping** from the list. 1. Select the instances to monitor and choose **Add**. 1. Choose **OK**. ## Metrics for ENA Express ENA Express is powered by AWS Scalable Reliable Datagram (SRD) technology. SRD is a high performance network transport protocol that uses dynamic routing to increase throughput and minimize tail latency. If you've enabled ENA Express for the network interface attachments on both the sending instance and receiving instance, you can use ENA Express metrics to help ensure that your instances take full advantage of the performance improvements that SRD technology provides. For example: + Evaluate your resources to ensure that they have sufficient capacity to establish more SRD connections. + Identify where there are potential issues that prevent eligible outgoing packets from using SRD. + Calculate the percentage of outgoing traffic that uses SRD for the instance. + Calculate the percentage of incoming traffic that uses SRD for the instance. **Note** To produce metrics, use driver version 2.8 or higher. To see a list of metrics for your Linux instance that's filtered for ENA Express, run the following **ethtool** command for your network interface (shown here as `eth0`). Take note of the value of the `ena_srd_mode` metric. ``` [ec2-user ~]$ ethtool -S eth0 | grep ena_srd NIC statistics: ena_srd_mode: 1 ena_srd_tx_pkts: 0 ena_srd_eligible_tx_pkts: 0 ena_srd_rx_pkts: 0 ena_srd_resource_utilization: 0 ``` The following metrics are available for all instances that have ENA Express enabled. **ena\$1srd\$1mode** Describes which ENA Express features are enabled. Values are as follows: + `0` = ENA Express off, UDP off + `1` = ENA Express on, UDP off + `2` = ENA Express off, UDP on **Note** This only happens when ENA Express was originally enabled, and UDP was configured to use it. The prior value is retained for UDP traffic. + `3` = ENA Express on, UDP on **ena\$1srd\$1eligible\$1tx\$1pkts** The number of network as follows: + Both sending and receiving instance types are supported. See the [Supported instance types for ENA Express](ena-express.md#ena-express-supported-instance-types) table for more information. + Both sending and receiving instances must have ENA Express configured. + The sending and receiving instances must run in the same Availability Zone. + The network path between the instances must not include middleware boxes. ENA Express doesn't currently support middleware boxes. The ENA Express eligibility metric covers source and destination requirements, and the network between the two endpoints. Eligible packets can still be disqualified after they’ve already been counted. For example, if an eligible packet is over the maximum transmission unit (MTU) limit, it falls back to standard ENA transmission, though the packet is still reflected as eligible in the counter. **ena\$1srd\$1tx\$1pkts** The number of SRD packets transmitted within a given time period. **ena\$1srd\$1rx\$1pkts** The number of SRD packets received within a given time period. **ena\$1srd\$1resource\$1utilization** The percentage of the maximum allowed memory utilization for concurrent SRD connections that the instance has consumed. To confirm if packet transmission is using SRD, you can compare the number of eligible packets (`ena_srd_eligible_tx_pkts` metric) to the number of SRD packets transmitted (`ena_srd_tx_pkts` metric) during a given time period. **Egress traffic (outgoing packets)** To ensure that your egress traffic uses SRD as expected, compare the number of SRD eligible packets (`ena_srd_eligible_tx_pkts`) with the number of SRD packets sent (`ena_srd_tx_pkts`) over a given time period. Significant differences between the number of eligible packets and the number of SRD packets sent are often caused by resource utilization issues. When the network card attached to the instance has used up its maximum resources, or if packets are over the MTU limit, eligible packets are not able to transmit via SRD, and must fall back to standard ENA transmission. Packets can also fall into this gap during live migrations or live server updates. Additional troubleshooting is required to determine the root cause. **Note** You can ignore occasional minor differences between the number of eligible packets and the number of SRD packets. This can happen when your instance establishes a connection to another instance for SRD traffic, for example. To find out what percentage of your total egress traffic over a given time period uses SRD, compare the number of SRD packets sent (`ena_srd_tx_pkts`) to the total number of packets sent for the instance (`NetworkPacketOut`) during that time. **Ingress traffic (incoming packets)** To find out what percentage of your ingress traffic uses SRD, compare the number of SRD packets received (`ena_srd_rx_pkts`) over a given time period to the total number of packets received for the instance (`NetworkPacketIn`) during that time. **Resource utilization** Resource utilization is based on the number of concurrent SRD connections a single instance can hold at a given time. The resource utilization metric (`ena_srd_resource_utilization`) keeps track of your current utilization for the instance. As utilization approaches 100%, you can expect to see performance issues. ENA Express falls back from SRD to standard ENA transmission, and the possibility of dropped packets increases. High resource utilization is a sign that it’s time to scale the instance out to improve network performance. **Note** When the network traffic for an instance exceeds a maximum, AWS shapes the traffic that exceeds the maximum by queueing and then dropping network packets. **Persistence** Egress and ingress metrics accrue while ENA Express is enabled for the instance. Metrics stop accruing if ENA Express is deactivated, but persist as long as the instance is still running. Metrics reset if the instance reboots or is terminated, or if the network interface is detached from the instance. ## Network performance metrics with the DPDK driver for ENA The ENA driver version 2.2.0 and later supports network metrics reporting. DPDK 20.11 includes the ENA driver 2.2.0 and is the first DPDK version to support this feature. DPDK driver v25.03 or later supports fragment proxy mode. If fragment proxy mode is enabled for the DPDK driver, egress fragment drops bypass the 1024 PPS limit that usually applies and are counted within standard PPS allowances. Fragment proxy mode is disabled by default. You can use an example application to view DPDK statistics. To start an interactive version of the example application, run the following command. ``` ./app/dpdk-testpmd -- -i ``` Within this interactive session, you can enter a command to retrieve extended statistics for a port. The following example command retrieves the statistics for port 0. ``` show port xstats 0 ``` The following is an example of an interactive session with the DPDK example application. ``` [root@ip-192.0.2.0 build]# ./app/dpdk-testpmd -- -i EAL: Detected 4 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: Probing VFIO support... EAL: Invalid NUMA socket, default to 0 EAL: Invalid NUMA socket, default to 0 EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket 0) EAL: No legacy callbacks, legacy socket not created Interactive-mode selected Port 0: link state change event testpmd: create a new mbuf pool : n=171456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. Configuring Port 0 (socket 0) Port 0: 02:C7:17:A2:60:B1 Checking link statuses... Done Error during enabling promiscuous mode for port 0: Operation not supported - ignore testpmd> show port xstats 0 ###### NIC extended statistics for port 0 rx_good_packets: 0 tx_good_packets: 0 rx_good_bytes: 0 tx_good_bytes: 0 rx_missed_errors: 0 rx_errors: 0 tx_errors: 0 rx_mbuf_allocation_errors: 0 rx_q0_packets: 0 rx_q0_bytes: 0 rx_q0_errors: 0 tx_q0_packets: 0 tx_q0_bytes: 0 wd_expired: 0 dev_start: 1 dev_stop: 0 tx_drops: 0 bw_in_allowance_exceeded: 0 bw_out_allowance_exceeded: 0 pps_allowance_exceeded: 0 conntrack_allowance_exceeded: 0 linklocal_allowance_exceeded: 0 rx_q0_cnt: 0 rx_q0_bytes: 0 rx_q0_refill_partial: 0 rx_q0_bad_csum: 0 rx_q0_mbuf_alloc_fail: 0 rx_q0_bad_desc_num: 0 rx_q0_bad_req_id: 0 tx_q0_cnt: 0 tx_q0_bytes: 0 tx_q0_prepare_ctx_err: 0 tx_q0_linearize: 0 tx_q0_linearize_failed: 0 tx_q0_tx_poll: 0 tx_q0_doorbells: 0 tx_q0_bad_req_id: 0 tx_q0_available_desc: 1023 testpmd> ``` For more information about the example application and using it to retrieve extended statistics. see [Testpmd Application User Guide](https://doc.dpdk.org/guides/testpmd_app_ug/) in the DPDK documentation. ## Metrics on instances running FreeBSD Starting with version 2.3.0, the ENA FreeBSD driver supports collecting network performance metrics on instances running FreeBSD. To enable the collection of FreeBSD metrics, enter the following command and set *interval* to a value between 1 and 3600. This specifies how often, in seconds, to collect FreeBSD metrics. ``` sysctl dev.ena.network_interface.eni_metrics.sample_interval=interval ``` For example, the following command sets the driver to collect FreeBSD metrics on network interface 1 every 10 seconds: ``` sysctl dev.ena.1.eni_metrics.sample_interval=10 ``` To turn off the collection of FreeBSD metrics, you can run the preceding command and specify `0` as the *interval*. After you enable collecting FreeBSD metrics, you can retrieve the latest set of collected metrics by running the following command. ``` sysctl dev.ena.network_interface.eni_metrics ``` # Improve network latency for Linux based EC2 instances Improve network latency on Linux Network latency is the amount of time it takes for a packet of data to travel from its source to its destination. Applications that send data across the network depend on timely responses to provide a positive user experience. High network latency can lead to various issues, such as the following: + Slow load times for web pages + Video stream lag + Difficulty accessing online resources This section outlines steps that you can take to improve the network latency on Amazon EC2 instances that run on Linux. To achieve optimal latency, follow these steps to configure your instance, kernel, and ENA driver settings. For additional configuration guidance, see the [ENA Linux Driver Best Practices and Performance Optimization Guide](https://github.com/amzn/amzn-drivers/blob/master/kernel/linux/ena/ENA_Linux_Best_Practices.rst) on GitHub. **Note** Steps and settings may vary slightly, depending on your specific network hardware, the AMI that you launched your instance from, and your application use case. Before you make any changes, thoroughly test and monitor your network performance to ensure that you're getting the desired results. ## Reduce the number of network hops for data packets Each hop that a data packet takes as it moves from router to router increases network latency. Typically, traffic must take multiple hops to reach your destination. There are two ways to reduce network hops for your Amazon EC2 instances, as follows: + **Cluster placement group** – When you specify a [cluster placement group](placement-strategies.md#placement-groups-cluster), Amazon EC2 launches instances that are in close proximity to each other, physically within the same Availability Zone (AZ) with tighter packing. The physical proximity of the instances in the group allows them to take advantage of high-speed connectivity, resulting in low latency and high single flow throughput. + **Dedicated Host** – A [Dedicated Host](dedicated-hosts-overview.md) is a physical server that's dedicated for your use. With a Dedicated Host, you can launch your instances to run on the same physical server. Communication between instances that run on the same Dedicated Host can happen without extra network hops. ## How Linux kernel configuration affects latency Linux kernel configuration can increase or decrease network latency. To achieve your latency optimization goals, it's important to fine-tune the Linux kernel configuration according to the specific requirements of your workload. There are many configuration options for the Linux kernel that might help decrease network latency. The most impactful options are as follows. + **Enable busy poll mode** – Busy poll mode reduces latency on the network receive path. When you enable busy poll mode, the socket layer code can directly poll the receive queue of a network device. The downside of busy polling is higher CPU usage in the host that comes from polling for new data in a tight loop. There are two global settings that control the number of microseconds to wait for packets for all interfaces.   `busy_read` A low latency busy poll timeout for socket reads. This controls the number of microseconds to wait for the socket layer to read packets on the device queue. To enable the feature globally with the **sysctl** command, the Linux Kernel organization recommends a value of 50 microseconds. For more information, see [busy\$1read](https://www.kernel.org/doc/html/v5.19/admin-guide/sysctl/net.html?highlight=busy_read) in the *Linux kernel user's and administrator's guide*. ``` [ec2-user ~]$ sudo sysctl -w net.core.busy_read=50 ``` `busy_poll` A low latency busy poll timeout for poll and select. This controls the number of microseconds to wait for events. The recommended value is between 50-100 microseconds, depending on the number of sockets you're polling. The more sockets you add, the higher the number should be. ``` [ec2-user ~]$ sudo sysctl -w net.core.busy_poll=50 ``` + **Configure CPU power states (C-states)** – C-states control the sleep levels that a core may enter when it's inactive. You might want to control C-states to tune your system for latency versus performance. In deeper C-states, the CPU is essentially "asleep" and can't respond to requests until it wakes up and transitions back to an active state. Putting cores to sleep takes time, and although a sleeping core allows more headroom for another core to boost to a higher frequency, it takes time for that sleeping core to wake back up and perform work. For example, if a core that's assigned to handle network packet interrupts is asleep, there might be a delay in servicing that interrupt. You can configure the system so that it doesn't use deeper C-states. However, while this configuration reduces the processor reaction latency, it also reduces the headroom available to other cores for Turbo Boost. To reduce the processor reaction latency, you can limit deeper C-states. For more information, see [High performance and low latency by limiting deeper C-states](https://docs.aws.amazon.com/linux/al2/ug/processor_state_control.html#c-states) in the *Amazon Linux 2 User Guide*. ## Interrupt moderation The ENA network driver enables communication between an instance and a network. The driver processes network packets and passes them on to the network stack or to the Nitro card. When a network packet comes in, the Nitro card generates an interrupt for the CPU to notify the software of an event. Interrupt An interrupt is a signal that a device or application sends to the processor. The interrupt tells the processor that an event has occurred or a condition has been met that requires immediate attention. Interrupts can handle time-sensitive tasks such as receiving data from a network interface, handling hardware events, or servicing requests from other devices. Interrupt moderation Interrupt moderation is a technique that reduces the number of interrupts a device generates by aggregating or delaying them. The purpose of interrupt moderation is to improve system performance by reducing the overhead associated with handling a large number of interrupts. Too many interrupts increase CPU usage, impacting the throughput adversely, while too few interrupts increase the latency. Dynamic interrupt moderation Dynamic interrupt moderation is an enhanced form of interrupt moderation that dynamically adjusts the interrupt rate based on the current system load and traffic patterns. It aims to strike a balance between reducing interrupt overhead and packets per second, or bandwidth. Dynamic interrupt moderation is enabled by default in some AMIs (but can be enabled or disabled in all AMIs). To minimize network latency, it might be necessary to disable interrupt moderation. However, this can also increase the overhead of interrupt processing. It's important to find the right balance between reducing latency and minimizing overhead. `ethtool` commands can help you configure interrupt moderation. By default, `rx-usecs` is set to `20`, and `tx-usecs` is set to `64`. To get the current interrupt modification configuration, use the following command. ``` [ec2-user ~]$ ethtool -c interface | egrep "rx-usecs:|tx-usecs:|Adaptive RX" Adaptive RX: on TX: off rx-usecs: 20 tx-usecs: 64 ``` To disable interrupt modification and dynamic interrupt moderation, use the following command. ``` [ec2-user ~]$ sudo ethtool -C interface adaptive-rx off rx-usecs 0 tx-usecs 0 ``` # Nitro system considerations for performance tuning Nitro performance considerations The Nitro System is a collection of hardware and software components built by AWS that enable high performance, high availability, and high security. The Nitro System provides bare metal-like capabilities that eliminate virtualization overhead and support workloads that require full access to host hardware. For more detailed information, see [AWS Nitro System](https://aws.amazon.com/ec2/nitro/). All current generation EC2 instance types perform network packet processing on EC2 Nitro Cards. This topic covers high level packet handling on the Nitro card, common aspects of network architecture and configuration that impact packet handling performance, and what actions you can take to achieve peak performance for your Nitro based instances. Nitro Cards handle all input and output (I/O) interfaces, such as those needed for Virtual Private Clouds (VPCs). For all of the components that send or receive information over the network, the Nitro cards act as a self-contained computing device for I/O traffic that's physically separate from the system main board on which customer workloads run. ## Network packet flow on Nitro cards EC2 instances built on the Nitro system have hardware acceleration capabilities that enable faster packet processing, as measured by packets per second (PPS) throughput rates. When a Nitro card performs the initial evaluation for a new flow, it saves information that's the same for all packets in the flow, such as security groups, access control lists, and route table entries. When it processes additional packets for the same flow, it can use the saved information to reduce overhead for those packets. Your connection rate is measured by the connections per second (CPS) metric. Each new connection requires additional processing overhead that must be factored into workload capability estimates. It's important to consider both the CPS and PPS metrics when you design your workloads. **How a connection is established** When a connection is established between a Nitro based instance and another endpoint, the Nitro card evaluates the full flow for the first packet that's sent or received between the two endpoints. For subsequent packets of the same flow, full reevaluation is usually not necessary. However, there are exceptions. For more information about the exceptions, see [Packets that don't use hardware acceleration](#ena-nitro-perf-exceptions). The following properties define the two endpoints and the packet flow between them. These five properties together are known as a 5-tuple flow. + Source IP + Source port + Destination IP + Destination port + Communication protocol The direction of the packet flow is known as *ingress* (inbound) and *egress* (outbound). The following high level descriptions summarize end to end network packet flow. + **Ingress** – When a Nitro card handles an inbound network packet, it evaluates the packet against stateful firewall rules and access control lists. It tracks the connection, meters it, and performs other actions as applicable. Then it forwards the packet to its destination on the host CPU. + **Egress** – When a Nitro card handles an outbound network packet, it looks up the remote interface destination, evaluates various VPC functions, applies rate limits, and performs other actions that apply. Then it forwards the packet to its next hop destination on the network. ## Design your network for optimal performance To take advantage of your Nitro system's performance capabilities, you must understand what your network processing needs are and how those needs affect the workload for your Nitro resources. Then you can design for optimal performance for your network landscape. Your infrastructure settings and application workload design and configuration can impact both the packet processing and connection rates. For example, if your application has a high rate of connection establishment, such as a DNS service, firewall, or virtual router, it will have less opportunity to take advantage of the hardware acceleration that only occurs after the connection is established. You can configure application and infrastructure settings to streamline workloads and improve network performance. However, not all packets are eligible for acceleration. The Nitro system uses the full network flow for new connections and for packets that aren't eligible for acceleration. The remainder of this section will focus on application and infrastructure design considerations to help ensure that packets flow within the accelerated path as much as possible. ### Network design considerations for the Nitro system When you configure network traffic for your instance, there are many aspects to consider that can affect PPS performance. After a flow is established, the majority of packets that regularly come in or go out are eligible for acceleration. However, exceptions exist to ensure that infrastructure designs and packet flows continue to meet protocol standards. To get the best performance from your Nitro card, you should carefully consider the pros and cons of the following configuration details for your infrastructure and applications. #### Infrastructure considerations Your infrastructure configuration can affect your packet flow and processing efficiency. The following list includes some important considerations. **Network interface configuration with asymmetry** Security groups use connection tracking to track information about traffic that flows to and from the instance. Asymmetric routing, where traffic comes into an instance through one network interface and leaves through a different network interface, can reduce the peak performance that an instance can achieve if flows are tracked. For more information about security group connection tracking, untracked connections, and automatically tracked connections, see [Amazon EC2 security group connection tracking](security-group-connection-tracking.md). **Network drivers** Network drivers are updated and released on a regular basis. If your drivers are out of date, that can significantly impair performance. Keep your drivers up to date to ensure that you have the latest patches and can take advantage of performance improvements, such as the accelerated path feature that's only available for the latest generation of drivers. Earlier drivers don't support the accelerated path feature. To take advantage of the accelerated path feature, we recommend that you install the latest ENA driver on your instances. **Linux instances** – ENA Linux driver 2.2.9 or later. To install or update the ENA Linux driver from the Amazon Drivers GitHub repository, see the [Driver compilation](https://github.com/amzn/amzn-drivers/tree/master/kernel/linux/ena#driver-compilation) section of the readme file. **Windows instances** – ENA Windows driver 2.0.0 or later. To install or update the ENA Windows driver, see [Install the ENA driver on EC2 Windows instances](ena-adapter-driver-install-upgrade-win.md). **Distance between endpoints** A connection between two instances in the same Availability Zone can process more packets per second than a connection across Regions as a result of TCP windowing at the application layer, which determines how much data can be in flight at any given time. Long distances between instances increase latency and decrease the number of packets that the endpoints can process. **Byte queue limit (BQL)** BQL is a feature that limits the number of bytes passed to the Nitro card to reduce queuing. BQL is disabled by default in ENA drivers, in Amazon Linux operating systems, and in most Linux distributions. If BQL and the fragment proxy override are both enabled, it can result in performance limitations by restricting the number of bytes passed to Nitro before all fragments are processed. #### Application design considerations There are aspects of application design and configuration that can affect your processing efficiency. The following list includes some important considerations. **Packet size** Larger packet sizes can increase throughput for the data that an instance can send and receive on the network. Amazon EC2 supports jumbo frames of 9001 bytes, however other services may enforce different limits. Smaller packet sizes can increase the packet process rate, but this can reduce the maximum achieved bandwidth when the number of packets exceed PPS allowances. If the size of a packet exceeds the Maximum Transmission Unit (MTU) of a network hop, a router along the path might fragment it. The resulting packet fragments are considered exceptions, and are normally processed at the standard rate (not accelerated). This can cause variations in your performance. However, you can override the standard behavior for outbound fragmented packets with the fragment proxy mode setting. For more information, see [Maximize network performance on your Nitro system](#ena-nitro-perf-maximize). We recommended that you evaluate your topology when you configure MTU. **Protocol trade-offs** Reliable protocols like TCP have more overhead than unreliable protocols like UDP. The lower overhead and simplified network processing for the UDP transport protocol can result in a higher PPS rate, but at the expense of reliable packet delivery. If reliable packet delivery isn’t critical for your application, UDP might be a good option. **Micro-bursting** Micro-bursting occurs when traffic exceeds allowances during brief periods of time rather than being evenly distributed. This typically happens on a microsecond scale. For example, say that you have an instance that can send up to 10 Gbps, and your application sends the full 10 Gb in half a second. This micro-burst exceeds the allowance during the first half second and leaves nothing for the remainder of the second. Even though you sent 10Gb in the 1 second timeframe, allowances in the first half second can result in packets being queued or dropped. You can use a network scheduler such as Linux Traffic Control to help pace your throughput and avoid causing queued or dropped packets as a result of micro-bursting. **Number of flows** A single flow is limited to 5 Gbps unless it's inside of a cluster placement group that supports up to 10 Gbps, or if it uses ENA Express, which supports up to 25 Gbps. Similarly, a Nitro card can process more packets across multiple flows as opposed to using a single flow. To achieve the peak packet processing rate per instance, we recommend at least 100 flows on instances with 100 Gbps or higher aggregate bandwidth. As aggregate bandwidth capabilities increase, the number of flows needed to achieve peak processing rates also increases. Benchmarking will help you determine what configuration you need to achieve peak rates on your network. **Elastic Network Adapter (ENA) queues** ENA (Elastic Network Adapter) uses multiple receive (Rx) and transmit (Tx) queues (ENA queues) to improve network performance and scalability on EC2 instances. These queues efficiently manage network traffic by load-balancing sent and received data across available queues. For more information, see [ENA queues](ena-queues.md). **Feature process overhead** Features like Traffic Mirroring and ENA Express can add more processing overhead, which can reduce absolute packet processing performance. You can limit feature use or disable features to increase packet processing rates. **Connection tracking to maintain state** Your security groups use connection tracking to store information about traffic to and from the instance. Connection tracking applies rules against each individual flow of network traffic to determine if the traffic is allowed or denied. The Nitro card uses flow tracking to maintain state for the flow. As more security group rules are applied, more work is required to evaluate the flow. Not all network traffic flows are tracked. If a security group rule is configured with [Untracked connections](security-group-connection-tracking.md#untracked-connections), no additional work is required except for connections that are automatically tracked to ensure symmetric routing when there are multiple valid reply paths. #### Packets that don't use hardware acceleration Not all packets can take advantage of hardware acceleration. Handling these exceptions involves some processing overhead which is necessary to ensure the health of your network flows. Network flows must reliably meet protocol standards, conform to changes in the VPC design, and route packets only to allowed destinations. However, the overhead reduces your performance. **Packet fragments** As mentioned under **Application considerations**, packet fragments that result from packets that exceed network MTU are normally handled as exceptions, and can't take advantage of hardware acceleration. However, you can bypass egress fragment limitations with the fragment proxy mode, depending on your driver version. For more information, see actions you can take in the [Maximize network performance on your Nitro system](#ena-nitro-perf-maximize) section. **Idle connections** When a connection has no activity for a while, even if the connection hasn't reached its timeout limit, the system can de-prioritize it. Then, if data comes in after the connection is de-prioritized, the system needs to handle it as an exception in order to reconnect. To manage your connections, you can use connection tracking timeouts to close idle connections. You can also use TCP keepalives to keep idle connections open. For more information, see [Idle connection tracking timeout](security-group-connection-tracking.md#connection-tracking-timeouts). **VPC mutation** Updates to security groups, route tables, and access control lists all need to be reevaluated in the processing path to ensure that route entries and security group rules still apply as expected. **ICMP flows** Internet Control Message Protocol (ICMP) is a network layer protocol that network devices use to diagnose network communication issues. These packets always use the full flow. **Asymmetric L2 flows** NitroV3 and earlier platforms do not use hardware acceleration for traffic between two ENIs in the same subnet where one ENI is using the default gateway router and the other is not. NitroV4 and later platforms utilize hardware acceleration in this scenario. For better performance on NitroV3 or earlier platforms, ensure that either the default gateway router used matches between both ENIs, or those ENIs are in different subnets. ## Maximize network performance on your Nitro system You can maximize your network performance on Nitro system by adjusting network settings. **Topics** + [ ### Considerations ](#considerations) + [ ### Tune PPS performance ](#tuning) + [ ### Configure ENA queue allocation ](#max-perf-ena-queues) + [ ### Monitor performance on Linux instances ](#monitoring) ### Considerations Before you make any design decisions or adjust any network settings on your instance, we recommend that you take the following steps to help ensure that you have the best outcome: 1. Understand the pros and cons of the actions that you can take to improve performance by reviewing [Network design considerations for the Nitro system](#ena-nitro-perf-considerations). For more considerations and best practices for your instance configuration on Linux, see [ENA Linux Driver Best Practices and Performance Optimization Guide](https://github.com/amzn/amzn-drivers/blob/master/kernel/linux/ena/ENA_Linux_Best_Practices.rst) on GitHub. 1. Benchmark your workloads with peak active flow count to determine a baseline for your application performance. With a performance baseline, you can test variations in your settings or application design to understand which considerations will have the most impact, especially if you plan to scale up or scale out. ### Tune PPS performance The following list contains actions that you can take to tune your PPS performance, depending on your system needs. + Reduce the physical distance between two instances. When sending and receiving instances are located in same Availability Zone or use cluster placement groups, you can reduce the number of hops a packet needs to take to travel from one endpoint to another. + Use [Untracked connections](security-group-connection-tracking.md#untracked-connections). + Use the UDP protocol for network traffic. + For EC2 instances with aggregate bandwidth of 100 Gbps or more, distribute the workload over 100 or more individual flows to spread the work evenly across the Nitro card. + To overcome the egress fragment PPS limit on EC2 instances, you can enable fragment proxy mode (depending on your driver version). This setting allows fragmented packets to be evaluated in the processing path, thereby overcoming the egress PPS limit of 1024. When loading the driver, run one of the following commands to enable or disable fragment proxy mode: **Enable fragment proxy mode** ``` sudo insmod ena.ko enable_frag_bypass=1 ``` **Disable fragment proxy mode** ``` sudo insmod ena.ko enable_frag_bypass=0 ``` ### Configure ENA queue allocation On supported instance types, you can dynamically allocate these queues across Elastic Network Interfaces (ENIs). Flexible ENA queue allocation optimizes resource distribution, enabling maximum vCPU utilization. High network performance workloads typically require multiple ENA queues. For more information, see [ENA queues](ena-queues.md). ### Monitor performance on Linux instances You can use Ethtool metrics on Linux instances to monitor instance networking performance indicators such as bandwidth, packet rate, and connection tracking. For more information, see [Monitor network performance for ENA settings on your EC2 instance](monitoring-network-performance-ena.md). # Optimize network performance on EC2 Windows instances Optimize network performance on Windows To achieve the maximum network performance on your Windows instances with enhanced networking, you might need to modify the default operating system configuration. We recommend the following configuration changes for applications that require high network performance. Other optimizations (such as turning on checksum offloading and enabling RSS, for example) are already configured on official Windows AMIs. **Note** TCP chimney offloading should be disabled in most use cases, and has been deprecated as of Windows Server 2016. In addition to these operating system optimizations, you should also consider the maximum transmission unit (MTU) of your network traffic, and adjust according to your workload and network architecture. For more information, see [Network maximum transmission unit (MTU) for your EC2 instance](network_mtu.md). AWS regularly measures average round trip latencies between instances launched in a cluster placement group of 50us and tail latencies of 200us at the 99.9 percentile. If your applications require consistently low latencies, we recommend using the latest version of the ENA drivers on fixed performance instances built on the Nitro System. ## Configure Receive side scaling CPU affinity Receive side scaling (RSS) is used to distribute network traffic CPU load across multiple processors. By default, the official Amazon Windows AMIs are configured with RSS enabled. ENA elastic network interfaces provide up to eight RSS queues. By defining CPU affinity for RSS queues, as well as for other system processes, it is possible to spread the CPU load out over multi-core systems, enabling more network traffic to be processed. On instance types with more than 16 vCPUs, we recommend that you use the `Set-NetAdapterRSS` PowerShell cmdlet, which manually excludes the boot processor (logical processor 0 and 1 when hyper-threading is enabled) from the RSS configuration for all elastic network interfaces, in order to prevent contention with various system components. Windows is hyper-thread aware and ensures that the RSS queues of a single network interface card (NIC) are always placed on different physical cores. Therefore, unless hyper-threading is disabled, in order to completely prevent contention with other NICs, spread the RSS configuration of each NIC among a range of 16 logical processors. The `Set-NetAdapterRss` cmdlet allows you to define the per-NIC range of valid logical processors by defining the values of BaseProcessorGroup, BaseProcessorNumber, MaxProcessingGroup, MaxProcessorNumber, and NumaNode (optional). If there are not enough physical cores to completely eliminate inter-NIC contention, minimize the overlapping ranges or reduce the number of logical processors in the elastic network interface ranges depending on the expected workload of the interface (in other words, a low volume administrative network interface may not need as many RSS queues assigned). Also, as previously noted, various components must run on CPU 0, and therefore we recommend excluding it from all RSS configurations when sufficient vCPUs are available. For example, when there are three elastic network interfaces on a 72 vCPU instance with 2 NUMA nodes with hyper-threading enabled, the following commands spread the network load between the two CPUs without overlap and prevent the use of core 0 completely. ``` Set-NetAdapterRss -Name NIC1 -BaseProcessorGroup 0 -BaseProcessorNumber 2 -MaxProcessorNumber 16 Set-NetAdapterRss -Name NIC2 -BaseProcessorGroup 1 -BaseProcessorNumber 0 -MaxProcessorNumber 14 Set-NetAdapterRss -Name NIC3 -BaseProcessorGroup 1 -BaseProcessorNumber 16 -MaxProcessorNumber 30 ``` Note that these settings are persistent for each network adapter. If an instance is resized to one with a different number of vCPUs, you should reevaluate the RSS configuration for each enabled elastic network interface. The complete Microsoft documentation for the cmdlet can be found here: [Set-NetAdapterRss](https://learn.microsoft.com/en-us/powershell/module/netadapter/set-netadapterrss). Special note for SQL workloads: We also recommend that you review your I/O thread affinity settings along with your elastic network interface RSS configuration to minimize I/O and network contention for the same CPUs. See [Server configuration: affinity mask](https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/affinity-mask-server-configuration-option). # Elastic Fabric Adapter for AI/ML and HPC workloads on Amazon EC2 Elastic Fabric Adapter An Elastic Fabric Adapter (EFA) is a network device that you can attach to your Amazon EC2 instance to accelerate Artificial Intelligence (AI), Machine Learning (ML), and High Performance Computing (HPC) applications. EFA enables you to achieve the application performance of an on-premises AI/ML or HPC cluster, with the scalability, flexibility, and elasticity provided by the AWS Cloud. EFA provides lower and more consistent latency and higher throughput than the TCP transport traditionally used in cloud-based HPC systems. It enhances the performance of inter-instance communication that is critical for scaling AI/ML and HPC applications. It is optimized to work on the existing AWS network infrastructure and it can scale depending on application requirements. EFA integrates with Libfabric, and it supports Nvidia Collective Communications Library (NCCL) and NVIDIA Inference Xfer Library (NIXL) for AI and ML applications, and Open MPI 4.1 and later and Intel MPI 2019 Update 5 and later for HPC applications. NCCL and MPI integrate with Libfabric 1.7.0 and later. NIXL integrates with Libfabric 1.21.0 and later. EFA supports RDMA (Remote Direct Memory Access) write on most supported instance types that have Nitro version 4 and later. RDMA read is supported on all instances with Nitro version 4 and later. For more information, see [Supported instance types](#efa-instance-types). **Topics** + [ ## EFA basics ](#efa-basics) + [ ## Supported interfaces and libraries ](#efa-mpi) + [ ## Supported instance types ](#efa-instance-types) + [ ## Supported operating systems ](#efa-os) + [ ## EFA limitations ](#efa-limits) + [ ## EFA pricing ](#efa-pricing) + [Get started with EFA and MPI](efa-start.md) + [Get started with EFA and NCCL](efa-start-nccl.md) + [Get started with EFA and NIXL](efa-start-nixl.md) + [Maximize network bandwidth](efa-acc-inst-types.md) + [Create and attach an EFA](create-efa.md) + [Detach and delete an EFA](detach-efa.md) + [Monitor an EFA](efa-working-monitor.md) + [Verify the EFA installer](efa-verify.md) + [Release notes](efa-changelog.md) ## EFA basics An EFA device can be attached to an EC2 instance in two ways: 1. Using a traditional EFA interface, also called EFA with ENA, which creates both an EFA device and an ENA device. 1. Using an EFA-only interface, which creates just the EFA device. The EFA device provides capabilities like built-in OS-bypass and congestion control through the Scalable Reliable Datagram (SRD) protocol. The EFA device features enable low-latency, reliable transport functionality that allows EFA interface to provide better application performance for HPC and ML applications on Amazon EC2. While the ENA device offers traditional IP networking. ![\[Contrasting a traditional HPC software stack with one that uses an EFA.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/efa_stack.png) Traditionally, AI/ML applications use NCCL and NIXL (for disaggregated inference). HPC applications use the Message Passing Interface (MPI) to interface with the system's network transport. In the AWS cloud, this has meant that applications interface with NCCL, NIXL, or MPI, which then uses the operating system's TCP/IP stack and the ENA device driver to enable network communication between instances. With a traditional EFA (EFA with ENA) or EFA-only interface, AI/ML applications use NCCL and NIXL (for disaggregated inference). HPC applications use MPI, to interface directly with the Libfabric API. The Libfabric API bypasses the operating system kernel and communicates directly with the EFA device to put packets on the network. This reduces overhead and enables AI/ML and HPC applications to run more efficiently. **Note** Libfabric is a core component of the OpenFabrics Interfaces (OFI) framework, which defines and exports the user-space API of OFI. For more information, see the [Libfabric OpenFabrics](https://ofiwg.github.io/libfabric/) website. ### Differences between ENA, EFA, and EFA-only network interfaces Amazon EC2 provides two types of network interfaces: + **ENA** interfaces provide all of the traditional IP networking and routing features that are required to support IP networking for a VPC. For more information, see [Enable enhanced networking with ENA on your EC2 instances](enhanced-networking-ena.md). + **EFA** (EFA with ENA) interfaces provide both the ENA device for IP networking and the EFA device for low-latency, high-throughput communication. + **EFA-only** interfaces support only the EFA device capabilities, without the ENA device for traditional IP networking. The following table provides a comparison of ENA, EFA, and EFA-only network interfaces. | | ENA | EFA (EFA with ENA) | EFA-only | | --- | --- | --- | --- | | Supports IP networking functionality | Yes | Yes | No | | Can be assigned IPv4 or IPv6 addresses | Yes | Yes | No | | Can be used as primary network interface for instance | Yes | Yes | No | | Counts towards ENI attachment limit for instance | Yes | Yes | Yes | | Instance type support | Supported on all Nitro-based instances types | [Supported instance types](#efa-instance-types) | [Supported instance types](#efa-instance-types) | | Parameter naming in EC2 APIs | interface | efa | efa-only | | Field naming in EC2 console | No selection | EFA with ENA | EFA-only | ## Supported interfaces and libraries EFAs support the following interfaces and libraries: + Open MPI 4.1 and later + Intel MPI 2019 Update 5 and later + NVIDIA Collective Communications Library (NCCL) 2.4.2 and later + NVIDIA Inference Xfer Library (NIXL) 1.0.0 and later + AWS Neuron SDK version 2.3 and later ## Supported instance types All of the following instance types support EFA. Additionally, the tables indicate RDMA read and RDMA write support for the instance types. ------ #### [ Nitro v6 ] | Instance type | RDMA read support | RDMA write support | | --- |--- |--- | | General Purpose | | --- | | m8a.48xlarge | Yes | Yes | | m8a.metal-48xl | Yes | Yes | | m8azn.24xlarge | Yes | Yes | | m8azn.metal-24xl | Yes | Yes | | m8gb.16xlarge | Yes | Yes | | m8gb.24xlarge | Yes | Yes | | m8gb.48xlarge | Yes | Yes | | m8gb.metal-24xl | Yes | Yes | | m8gb.metal-48xl | Yes | Yes | | m8gn.16xlarge | Yes | Yes | | m8gn.24xlarge | Yes | Yes | | m8gn.48xlarge | Yes | Yes | | m8gn.metal-24xl | Yes | Yes | | m8gn.metal-48xl | Yes | Yes | | m8i.48xlarge | Yes | Yes | | m8i.96xlarge | Yes | Yes | | m8i.metal-48xl | Yes | Yes | | m8i.metal-96xl | Yes | Yes | | m8id.48xlarge | Yes | Yes | | m8id.96xlarge | Yes | Yes | | m8id.metal-48xl | Yes | Yes | | m8id.metal-96xl | Yes | Yes | | Compute Optimized | | --- | | c8a.48xlarge | Yes | Yes | | c8a.metal-48xl | Yes | Yes | | c8gb.16xlarge | Yes | Yes | | c8gb.24xlarge | Yes | Yes | | c8gb.48xlarge | Yes | Yes | | c8gb.metal-24xl | Yes | Yes | | c8gb.metal-48xl | Yes | Yes | | c8gn.16xlarge | Yes | Yes | | c8gn.24xlarge | Yes | Yes | | c8gn.48xlarge | Yes | Yes | | c8gn.metal-24xl | Yes | Yes | | c8gn.metal-48xl | Yes | Yes | | c8i.48xlarge | Yes | Yes | | c8i.96xlarge | Yes | Yes | | c8i.metal-48xl | Yes | Yes | | c8i.metal-96xl | Yes | Yes | | c8id.48xlarge | Yes | Yes | | c8id.96xlarge | Yes | Yes | | c8id.metal-48xl | Yes | Yes | | c8id.metal-96xl | Yes | Yes | | Memory Optimized | | --- | | r8a.48xlarge | Yes | Yes | | r8a.metal-48xl | Yes | Yes | | r8gb.16xlarge | Yes | Yes | | r8gb.24xlarge | Yes | Yes | | r8gb.48xlarge | Yes | Yes | | r8gb.metal-24xl | Yes | Yes | | r8gb.metal-48xl | Yes | Yes | | r8gn.16xlarge | Yes | Yes | | r8gn.24xlarge | Yes | Yes | | r8gn.48xlarge | Yes | Yes | | r8gn.metal-24xl | Yes | Yes | | r8gn.metal-48xl | Yes | Yes | | r8i.48xlarge | Yes | Yes | | r8i.96xlarge | Yes | Yes | | r8i.metal-48xl | Yes | Yes | | r8i.metal-96xl | Yes | Yes | | r8id.48xlarge | Yes | Yes | | r8id.96xlarge | Yes | Yes | | r8id.metal-48xl | Yes | Yes | | r8id.metal-96xl | Yes | Yes | | x8aedz.24xlarge | Yes | Yes | | x8aedz.metal-24xl | Yes | Yes | | x8i.48xlarge | Yes | Yes | | x8i.64xlarge | Yes | Yes | | x8i.96xlarge | Yes | Yes | | x8i.metal-48xl | Yes | Yes | | x8i.metal-96xl | Yes | Yes | | Storage Optimized | | --- | | i8ge.48xlarge | Yes | No | | i8ge.metal-48xl | Yes | No | | Accelerated Computing | | --- | | g7e.8xlarge | Yes | Yes | | g7e.12xlarge | Yes | Yes | | g7e.24xlarge | Yes | Yes | | g7e.48xlarge | Yes | Yes | | p6-b200.48xlarge | Yes | Yes | | p6-b300.48xlarge | Yes | Yes | | High Performance Computing | | --- | | hpc8a.96xlarge | Yes | Yes | ------ #### [ Nitro v5 ] | Instance type | RDMA read support | RDMA write support | | --- |--- |--- | | General Purpose | | --- | | m8g.24xlarge | Yes | No | | m8g.48xlarge | Yes | No | | m8g.metal-24xl | Yes | No | | m8g.metal-48xl | Yes | No | | m8gd.24xlarge | No | No | | m8gd.48xlarge | No | No | | m8gd.metal-24xl | No | No | | m8gd.metal-48xl | No | No | | Compute Optimized | | --- | | c7gn.16xlarge | Yes | No | | c7gn.metal | Yes | No | | c8g.24xlarge | Yes | No | | c8g.48xlarge | Yes | No | | c8g.metal-24xl | Yes | No | | c8g.metal-48xl | Yes | No | | c8gd.24xlarge | No | No | | c8gd.48xlarge | No | No | | c8gd.metal-24xl | No | No | | c8gd.metal-48xl | No | No | | Memory Optimized | | --- | | r8g.24xlarge | No | No | | r8g.48xlarge | No | No | | r8g.metal-24xl | No | No | | r8g.metal-48xl | No | No | | r8gd.24xlarge | No | No | | r8gd.48xlarge | No | No | | r8gd.metal-24xl | No | No | | r8gd.metal-48xl | No | No | | x8g.24xlarge | No | No | | x8g.48xlarge | No | No | | x8g.metal-24xl | No | No | | x8g.metal-48xl | No | No | | Storage Optimized | | --- | | i7ie.48xlarge | Yes | No | | i7ie.metal-48xl | Yes | No | | i8g.48xlarge | No | No | | i8g.metal-48xl | No | No | | Accelerated Computing | | --- | | p5en.48xlarge | Yes | Yes | | p6e-gb200.36xlarge | Yes | Yes | | trn2.3xlarge | Yes | Yes | | trn2.48xlarge | Yes | Yes | | trn2u.48xlarge | Yes | Yes | | High Performance Computing | | --- | | hpc7g.4xlarge | Yes | No | | hpc7g.8xlarge | Yes | No | | hpc7g.16xlarge | Yes | No | ------ #### [ Nitro v4 ] | Instance type | RDMA read support | RDMA write support | | --- |--- |--- | | General Purpose | | --- | | m6a.48xlarge | Yes | Yes | | m6a.metal | Yes | Yes | | m6i.32xlarge | Yes | Yes | | m6i.metal | Yes | Yes | | m6id.32xlarge | Yes | Yes | | m6id.metal | Yes | Yes | | m6idn.32xlarge | Yes | Yes | | m6idn.metal | Yes | Yes | | m6in.32xlarge | Yes | Yes | | m6in.metal | Yes | Yes | | m7a.48xlarge | Yes | No | | m7a.metal-48xl | Yes | No | | m7g.16xlarge | Yes | No | | m7g.metal | Yes | No | | m7gd.16xlarge | Yes | No | | m7gd.metal | Yes | No | | m7i.48xlarge | Yes | No | | m7i.metal-48xl | Yes | No | | Compute Optimized | | --- | | c6a.48xlarge | Yes | Yes | | c6a.metal | Yes | Yes | | c6gn.16xlarge | Yes | Yes | | c6i.32xlarge | Yes | Yes | | c6i.metal | Yes | Yes | | c6id.32xlarge | Yes | Yes | | c6id.metal | Yes | Yes | | c6in.32xlarge | Yes | Yes | | c6in.metal | Yes | Yes | | c7a.48xlarge | Yes | No | | c7a.metal-48xl | Yes | No | | c7g.16xlarge | Yes | Yes | | c7g.metal | Yes | Yes | | c7gd.16xlarge | Yes | No | | c7gd.metal | Yes | No | | c7i.48xlarge | Yes | No | | c7i.metal-48xl | Yes | No | | Memory Optimized | | --- | | r6a.48xlarge | Yes | Yes | | r6a.metal | Yes | Yes | | r6i.32xlarge | Yes | Yes | | r6i.metal | Yes | Yes | | r6id.32xlarge | Yes | Yes | | r6id.metal | Yes | Yes | | r6idn.32xlarge | Yes | Yes | | r6idn.metal | Yes | Yes | | r6in.32xlarge | Yes | Yes | | r6in.metal | Yes | Yes | | r7a.48xlarge | No | No | | r7a.metal-48xl | No | No | | r7g.16xlarge | No | No | | r7g.metal | No | No | | r7gd.16xlarge | No | No | | r7gd.metal | No | No | | r7i.48xlarge | No | No | | r7i.metal-48xl | No | No | | r7iz.32xlarge | No | No | | r7iz.metal-32xl | No | No | | u7i-6tb.112xlarge | Yes | Yes | | u7i-8tb.112xlarge | Yes | Yes | | u7i-12tb.224xlarge | Yes | Yes | | u7in-16tb.224xlarge | Yes | Yes | | u7in-24tb.224xlarge | Yes | Yes | | u7in-32tb.224xlarge | Yes | Yes | | u7inh-32tb.480xlarge | Yes | Yes | | x2idn.32xlarge | Yes | Yes | | x2idn.metal | Yes | Yes | | x2iedn.32xlarge | Yes | Yes | | x2iedn.metal | Yes | Yes | | Storage Optimized | | --- | | i4g.16xlarge | Yes | Yes | | i4i.32xlarge | Yes | Yes | | i4i.metal | Yes | Yes | | i7i.24xlarge | Yes | No | | i7i.48xlarge | Yes | No | | i7i.metal-48xl | Yes | No | | im4gn.16xlarge | Yes | Yes | | Accelerated Computing | | --- | | f2.48xlarge | Yes | Yes | | g6.8xlarge | Yes | Yes | | g6.12xlarge | Yes | Yes | | g6.16xlarge | Yes | Yes | | g6.24xlarge | Yes | Yes | | g6.48xlarge | Yes | Yes | | g6e.8xlarge | Yes | Yes | | g6e.12xlarge | Yes | Yes | | g6e.16xlarge | Yes | Yes | | g6e.24xlarge | Yes | Yes | | g6e.48xlarge | Yes | Yes | | gr6.8xlarge | Yes | Yes | | p5.4xlarge | Yes | Yes | | p5.48xlarge | Yes | Yes | | p5e.48xlarge | Yes | Yes | | trn1.32xlarge | Yes | Yes | | trn1n.32xlarge | Yes | Yes | | High Performance Computing | | --- | | hpc6a.48xlarge | Yes | Yes | | hpc6id.32xlarge | Yes | Yes | | hpc7a.12xlarge | Yes | No | | hpc7a.24xlarge | Yes | No | | hpc7a.48xlarge | Yes | No | | hpc7a.96xlarge | Yes | No | ------ #### [ Nitro v3 ] | Instance type | RDMA read support | RDMA write support | | --- |--- |--- | | General Purpose | | --- | | m5dn.24xlarge | No | No | | m5dn.metal | No | No | | m5n.24xlarge | No | No | | m5n.metal | No | No | | m5zn.12xlarge | No | No | | m5zn.metal | No | No | | Compute Optimized | | --- | | c5n.9xlarge | No | No | | c5n.18xlarge | No | No | | c5n.metal | No | No | | Memory Optimized | | --- | | r5dn.24xlarge | No | No | | r5dn.metal | No | No | | r5n.24xlarge | No | No | | r5n.metal | No | No | | x2iezn.12xlarge | No | No | | x2iezn.metal | No | No | | Storage Optimized | | --- | | i3en.12xlarge | No | No | | i3en.24xlarge | No | No | | i3en.metal | No | No | | Accelerated Computing | | --- | | dl1.24xlarge | Yes | No | | dl2q.24xlarge | No | No | | g4dn.8xlarge | No | No | | g4dn.12xlarge | No | No | | g4dn.16xlarge | No | No | | g4dn.metal | No | No | | g5.8xlarge | No | No | | g5.12xlarge | No | No | | g5.16xlarge | No | No | | g5.24xlarge | No | No | | g5.48xlarge | No | No | | inf1.24xlarge | No | No | | p3dn.24xlarge | No | No | | p4d.24xlarge | Yes | No | | p4de.24xlarge | Yes | No | | vt1.24xlarge | No | No | | Previous Generation | | --- | | p3dn.24xlarge | No | No | ------ **To see the available instance types that support EFAs in a specific Region** The available instance types vary by Region. To see the available instance types that support EFAs in a Region, use the [describe-instance-types](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-types.html) command with the `--region` parameter. Include the `--filters` parameter to scope the results to the instance types that support EFA and the `--query` parameter to scope the output to the value of `InstanceType`. ``` aws ec2 describe-instance-types \ --region us-east-1 \ --filters Name=network-info.efa-supported,Values=true \ --query "InstanceTypes[*].[InstanceType]" \ --output text | sort ``` ## Supported operating systems Operating system support differs depending on the processor type. The following table shows the supported operating systems. | Operating system | Intel/AMD (`x86_64`) instance types | AWS Graviton (`arm64`) instance types | | --- | --- | --- | | Amazon Linux 2023 | ✓ | ✓ | | Amazon Linux 2 | ✓ | ✓ | | RHEL 8 and 9 | ✓ | ✓ | | Debian 11, 12, and 13 | ✓ | ✓ | | Rocky Linux 8 and 9 | ✓ | ✓ | | Ubuntu 22.04 and 24.04 | ✓ | ✓ | | SUSE Linux Enterprise 15 SP2 and later | ✓ | ✓ | | OpenSUSE Leap 15.5 and later | ✓ | | **Note** Some of the listed operating systems might not be supported with Intel MPI. If you are using Intel MPI, refer to the [ Intel MPI documentation](https://www.intel.com/content/www/us/en/developer/articles/system-requirements/mpi-library-system-requirements.html) to verify support for your operating system. ## EFA limitations EFAs have the following limitations: + RDMA write is not supported with all instance types. For more information, see [Supported instance types](#efa-instance-types). + EFA traffic1 between P4d/P4de/DL1 instances and other instance types is currently not supported. + [Instance types that support multiple network cards](using-eni.md#network-cards) can be configured with one EFA per network card. All other supported instance types support only one EFA per instance. + `c7g.16xlarge`, `m7g.16xlarge`, and `r7g.16xlarge` Dedicated Instances and Dedicated Hosts are not supported when an EFA is attached. + EFA traffic1 can't cross Availability Zones or VPCs. This does not apply to normal IP traffic from the ENA device of an EFA interface. + EFA traffic1 is not routable. Normal IP traffic from the ENA device of an EFA interface remains routable. + EFA is not supported on AWS Outposts. + The EFA device of an EFA (EFA with ENA) interface is supported on Windows instances only for AWS Cloud Digital Interface Software Development Kit (AWS CDI SDK) based applications. If you attach an EFA (EFA with ENA) interface to a Windows instance for non-CDI SDK based applications, it functions as an ENA interface, without the added EFA device capabilities. The EFA-only interface is not supported by AWS CDI based applications on Windows or Linux. For more information, see the [AWS Cloud Digital Interface Software Development Kit (AWS CDI SDK) User Guide](https://docs.aws.amazon.com/CDI-SDK/latest/ug/what-is.html). 1*EFA traffic* refers to the traffic transmitted through the EFA device of either an EFA (EFA with ENA) or EFA-only interface. ## EFA pricing EFA is available as an optional Amazon EC2 networking feature that you can enable on any supported instance at no additional cost. # Get started with EFA and MPI for HPC workloads on Amazon EC2 Get started with EFA and MPI This tutorial helps you to launch an EFA and MPI-enabled instance cluster for HPC workloads. **Note** The `u7i-12tb.224xlarge`, `u7in-16tb.224xlarge`, `u7in-24tb.224xlarge`, and `u7in-32tb.224xlarge` instances can run up to 128 parallel MPI processes with Open MPI or up to 256 parallel MPI processes with Intel MPI. **Topics** + [ ## Step 1: Prepare an EFA-enabled security group ](#efa-start-security) + [ ## Step 2: Launch a temporary instance ](#efa-start-tempinstance) + [ ## Step 3: Install the EFA software ](#efa-start-enable) + [ ## Step 4: (*Optional*) Enable Open MPI 5 ](#efa-start-ompi5) + [ ## Step 5: (*Optional*) Install Intel MPI ](#efa-start-impi) + [ ## Step 6: Disable ptrace protection ](#efa-start-ptrace) + [ ## Step 7. Confirm installation ](#efa-start-test) + [ ## Step 8: Install your HPC application ](#efa-start-hpc-app) + [ ## Step 9: Create an EFA-enabled AMI ](#efa-start-ami) + [ ## Step 10: Launch EFA-enabled instances into a cluster placement group ](#efa-start-instances) + [ ## Step 11: Terminate the temporary instance ](#efa-start-terminate) + [ ## Step 12: Enable passwordless SSH ](#efa-start-passwordless) ## Step 1: Prepare an EFA-enabled security group An EFA requires a security group that allows all inbound and outbound traffic to and from the security group itself. The following procedure creates a security group that allows all inbound and outbound traffic to and from itself, and that allows inbound SSH traffic from any IPv4 address for SSH connectivity. **Important** This security group is intended for testing purposes only. For your production environments, we recommend that you create an inbound SSH rule that allows traffic only from the IP address from which you are connecting, such as the IP address of your computer, or a range of IP addresses in your local network. For other scenarios, see [Security group rules for different use cases](security-group-rules-reference.md). **To create an EFA-enabled security group** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Security Groups** and then choose **Create security group**. 1. In the **Create security group** window, do the following: 1. For **Security group name**, enter a descriptive name for the security group, such as `EFA-enabled security group`. 1. (Optional) For **Description**, enter a brief description of the security group. 1. For **VPC**, select the VPC into which you intend to launch your EFA-enabled instances. 1. Choose **Create security group**. 1. Select the security group that you created, and on the **Details** tab, copy the **Security group ID**. 1. With the security group still selected, choose **Actions**, **Edit inbound rules**, and then do the following: 1. Choose **Add rule**. 1. For **Type**, choose **All traffic**. 1. For **Source type**, choose **Custom** and paste the security group ID that you copied into the field. 1. Choose **Add rule**. 1. For **Type**, choose **SSH**. 1. For **Source type**, choose **Anywhere-IPv4**. 1. Choose **Save rules**. 1. With the security group still selected, choose **Actions**, **Edit outbound rules**, and then do the following: 1. Choose **Add rule**. 1. For **Type**, choose **All traffic**. 1. For **Destination type**, choose **Custom** and paste the security group ID that you copied into the field. 1. Choose **Save rules**. ## Step 2: Launch a temporary instance Launch a temporary instance that you can use to install and configure the EFA software components. You use this instance to create an EFA-enabled AMI from which you can launch your EFA-enabled instances. **To launch a temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**, and then choose **Launch Instances** to open the new launch instance wizard. 1. (*Optional*) In the **Name and tags** section, provide a name for the instance, such as `EFA-instance`. The name is assigned to the instance as a resource tag (`Name=EFA-instance`). 1. In the **Application and OS Images** section, select an AMI for one of the [supported operating systems](efa.md#efa-os). 1. In the **Instance type** section, select a [supported instance type](efa.md#efa-instance-types). 1. In the **Key pair** section, select the key pair to use for the instance. 1. In the **Network settings** section, choose **Edit**, and then do the following: 1. For **Subnet**, choose the subnet in which to launch the instance. **Important** You must select a subnet. If you do not select a subnet, you can't enable the instance for EFA. 1. For **Firewall (security groups)**, choose **Select existing security group**, and then select the security group that you created in the previous step. 1. Expand the **Advanced network configuration** section. For **Network interface 1**, select **Network card index = 0**, **Device index = 0**, and **Interface type = EFA with ENA**. (Optional) If you are using a multi-card instance type, such as `p4d.24xlarge` or `p5.48xlarge`, for each additional network interface required, choose **Add network interface**, for **Network card index** select the next unused index, and then select **Device index = 1** and **Interface type = EFA with ENA** or **EFA-only**. 1. In the **Storage** section, configure the volumes as needed. 1. In the **Summary** panel on the right, choose **Launch instance**. **Note** Consider requiring the use of IMDSv2 for the temporary instance as well as the AMI that you will create in [Step 9](#efa-start-ami) unless you have already [set IMDSv2 as the default for the account](configuring-IMDS-new-instances.md#set-imdsv2-account-defaults). For more information about IMDSv2 configuration steps, see [Configure instance metadata options for new instances](configuring-IMDS-new-instances.md). ## Step 3: Install the EFA software Install the EFA-enabled kernel, EFA drivers, Libfabric, and Open MPI stack that is required to support EFA on your temporary instance. The steps differ depending on whether you intend to use EFA with Open MPI, with Intel MPI, or with Open MPI and Intel MPI. **Note** Some operating systems might not be supported with Intel MPI. If you are using Intel MPI, refer to the [ Intel MPI documentation](https://www.intel.com/content/www/us/en/developer/articles/system-requirements/mpi-library-system-requirements.html) to verify support for your operating system. **To install the EFA software** 1. Connect to the instance you launched. For more information, see [Connect to your Linux instance using SSH](connect-to-linux-instance.md). 1. To ensure that all of your software packages are up to date, perform a quick software update on your instance. This process may take a few minutes. + Amazon Linux 2023, Amazon Linux 2, RHEL 8/9, Rocky Linux 8/9 ``` $ sudo yum update -y ``` + Ubuntu and Debian ``` $ sudo apt-get update && sudo apt-get upgrade -y ``` + SUSE Linux Enterprise ``` $ sudo zypper update -y ``` 1. Reboot the instance and reconnect to it. 1. Download the EFA software installation files. The software installation files are packaged into a compressed tarball (`.tar.gz`) file. To download the latest *stable* version, use the following command. You can also get the latest version by replacing the version number with `latest` in the preceding command. ``` $ curl -O https://efa-installer.amazonaws.com/aws-efa-installer-1.47.0.tar.gz ``` 1. (*Optional*) Verify the authenticity and integrity of the EFA tarball (`.tar.gz`) file. We recommend that you do this to verify the identity of the software publisher and to check that the file has not been altered or corrupted since it was published. If you do not want to verify the tarball file, skip this step. **Note** Alternatively, if you prefer to verify the tarball file by using an MD5 or SHA256 checksum instead, see [Verify the EFA installer using a checksum](efa-verify.md). 1. Download the public GPG key and import it into your keyring. ``` $ wget https://efa-installer.amazonaws.com/aws-efa-installer.key && gpg --import aws-efa-installer.key ``` The command should return a key value. Make a note of the key value, because you need it in the next step. 1. Verify the GPG key's fingerprint. Run the following command and specify the key value from the previous step. ``` $ gpg --fingerprint key_value ``` The command should return a fingerprint that is identical to `4E90 91BC BB97 A96B 26B1 5E59 A054 80B1 DD2D 3CCC`. If the fingerprint does not match, don't run the EFA installation script, and contact Support. 1. Download the signature file and verify the signature of the EFA tarball file. ``` $ wget https://efa-installer.amazonaws.com/aws-efa-installer-1.47.0.tar.gz.sig && gpg --verify ./aws-efa-installer-1.47.0.tar.gz.sig ``` The following shows example output. ``` gpg: Signature made Wed 29 Jul 2020 12:50:13 AM UTC using RSA key ID DD2D3CCC gpg: Good signature from "Amazon EC2 EFA " gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 4E90 91BC BB97 A96B 26B1 5E59 A054 80B1 DD2D 3CCC ``` If the result includes `Good signature`, and the fingerprint matches the fingerprint returned in the previous step, proceed to the next step. If not, don't run the EFA installation script, and contact Support. 1. Extract the files from the compressed `.tar.gz` file and navigate into the extracted directory. ``` $ tar -xf aws-efa-installer-1.47.0.tar.gz && cd aws-efa-installer ``` 1. Install the EFA software. Do one of the following depending on your use case. **Note** **EFA does not support NVIDIA GPUDirect with SUSE Linux**. If you are using SUSE Linux, you must additionally specify the `--skip-kmod` option to prevent kmod installation. By default, SUSE Linux does not allow out-of-tree kernel modules. ------ #### [ Open MPI and Intel MPI ] If you intend to use EFA with Open MPI and Intel MPI, you must install the EFA software with Libfabric and Open MPI, and you ** must complete Step 5: Install Intel MPI**. To install the EFA software with Libfabric and Open MPI, run the following command. **Note** From EFA 1.30.0, both Open MPI 4.1 and Open MPI 5 are installed by default. You can optionally specify the version of Open MPI that you want to install. To install only Open MPI 4.1, include `--mpi=openmpi4`. To install only Open MPI 5, include `--mpi=openmpi5`. To install both, omit the `--mpi` option. ``` $ sudo ./efa_installer.sh -y ``` Libfabric is installed to `/opt/amazon/efa`. Open MPI 4.1 is installed to `/opt/amazon/openmpi`. Open MPI 5 is installed to `/opt/amazon/openmpi5`. ------ #### [ Open MPI only ] If you intend to use EFA with Open MPI only, you must install the EFA software with Libfabric and Open MPI, and you can **skip Step 5: Install Intel MPI**. To install the EFA software with Libfabric and Open MPI, run the following command. **Note** From EFA 1.30.0, both Open MPI 4.1 and Open MPI 5 are installed by default. You can optionally specify the version of Open MPI that you want to install. To install only Open MPI 4.1, include `--mpi=openmpi4`. To install only Open MPI 5, include `--mpi=openmpi5`. To install both, omit the `--mpi` option. ``` $ sudo ./efa_installer.sh -y ``` Libfabric is installed to `/opt/amazon/efa`. Open MPI 4.1 is installed to `/opt/amazon/openmpi`. Open MPI 5 is installed to `/opt/amazon/openmpi5`. ------ #### [ Intel MPI only ] If you intend to use EFA with Intel MPI only, you can install the EFA software without Libfabric and Open MPI. In this case, Intel MPI uses its embedded Libfabric. If you choose to do this, you **must complete Step 5: Install Intel MPI**. To install the EFA software without Libfabric and Open MPI, run the following command. ``` $ sudo ./efa_installer.sh -y --minimal ``` ------ 1. If the EFA installer prompts you to reboot the instance, do so and then reconnect to the instance. Otherwise, log out of the instance and then log back in to complete the installation. 1. Delete the uncompressed tarball and the tarball itself. Otherwise, these will be included in the EFA-enabled AMI that you create, increasing its size. ## Step 4: (*Optional*) Enable Open MPI 5 **Note** Perform this step only if you intend to use Open MPI 5. From EFA 1.30.0, both Open MPI 4.1 and Open MPI 5 are installed by default. Alternatively, you can choose to install only Open MPI 4.1 or Open MPI 5. If you chose to install Open MPI 5 in **Step 3: Install the EFA software**, and you intend to use it, you must perform the following steps to enable it. **To enable Open MPI 5** 1. Add Open MPI 5 to the PATH environment variable. ``` $ module load openmpi5 ``` 1. Verify that Open MPI 5 is enabled for use. ``` $ which mpicc ``` The command should return the Open MPI 5 installation directory - `/opt/amazon/openmpi5`. 1. (*Optional*) To ensure that Open MPI 5 is added to PATH environment variable each time the instance starts, do the following: ------ #### [ bash shell ] Add `module load openmpi5` to `/home/username/.bashrc` and `/home/username/.bash_profile`. ------ #### [ csh and tcsh shells ] Add `module load openmpi5` to `/home/username/.cshrc`. ------ If you need to remove Open MPI 5 from the PATH environment variable, run the following command and remove the command from the shell startup scripts. ``` $ module unload openmpi5 ``` ## Step 5: (*Optional*) Install Intel MPI **Important** Perform this step only if you intend to use Intel MPI. If you intend to only use Open MPI, skip this step. Intel MPI requires an additional installation and environment variable configuration. **Prerequisite** Ensure that the user performing the following steps has sudo permissions. **To install Intel MPI** 1. To download the Intel MPI installation script, do the following 1. Visit the [Intel website](https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html#mpi). 1. In the **Intel MPI Library** section of the webpage, choose the link for the **Intel MPI Library for Linux** **Offline** installer. 1. Run the installation script that you downloaded in the previous step. ``` $ sudo bash installation_script_name.sh ``` 1. In the installer, choose **Accept & install**. 1. Read the Intel Improvement Program, choose the appropriate option, and then choose **Begin Installation**. 1. When the installation completes, choose **Close**. 1. By default, Intel MPI uses its embedded (internal) Libfabric. You can configure Intel MPI to use the Libfabric that ships with the EFA installer instead. Typically, the EFA installer ships with a later version of Libfabric than Intel MPI. In some cases, the Libfabric that ships with the EFA installer is more performant than that of Intel MPI. To configure Intel MPI to use the Libfabric that ships with the EFA installer, do one of the following depending on your shell. ------ #### [ bash shells ] Add the following statement to `/home/username/.bashrc` and `/home/username/.bash_profile`. ``` export I_MPI_OFI_LIBRARY_INTERNAL=0 ``` ------ #### [ csh and tcsh shells ] Add the following statement to `/home/username/.cshrc`. ``` setenv I_MPI_OFI_LIBRARY_INTERNAL 0 ``` ------ 1. Add the following **source** command to your shell script to source the `vars.sh` script from the installation directory to set up the compiler environment each time the instance starts. Do one of the following depending on your shell. ------ #### [ bash shells ] Add the following statement to `/home/username/.bashrc` and `/home/username/.bash_profile`. ``` source /opt/intel/oneapi/mpi/latest/env/vars.sh ``` ------ #### [ csh and tcsh shells ] Add the following statement to `/home/username/.cshrc`. ``` source /opt/intel/oneapi/mpi/latest/env/vars.csh ``` ------ 1. By default, if EFA is not available due to a misconfiguration, Intel MPI defaults to the TCP/IP network stack, which might result in slower application performance. You can prevent this by setting `I_MPI_OFI_PROVIDER` to `efa`. This causes Intel MPI to fail with the following error if EFA is not available: ``` Abort (XXXXXX) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: OtherMPI error, MPIR_Init_thread (XXX)........: MPID_Init (XXXX)..............: MPIDI_OFI_mpi_init_hook (XXXX): open_fabric (XXXX)............: find_provider (XXXX)..........: OFI fi_getinfo() failed (ofi_init.c:2684:find_provider: ``` Do one of the following depending on your shell. ------ #### [ bash shells ] Add the following statement to `/home/username/.bashrc` and `/home/username/.bash_profile`. ``` export I_MPI_OFI_PROVIDER=efa ``` ------ #### [ csh and tcsh shells ] Add the following statement to `/home/username/.cshrc`. ``` setenv I_MPI_OFI_PROVIDER efa ``` ------ 1. By default, Intel MPI doesn't print debugging information. You can specify different verbosity levels to control the debugging information. Possible values (in order of the amount of detail they provide) are: `0` (default), `1`, `2`, `3`, `4`, `5`. Level `1` and higher prints the `libfabric version` and `libfabric provider`. Use `libfabric version` to check whether Intel MPI is using the internal Libfabric or the Libfabric that ships with the EFA installer. If it's using the internal Libfabric, the version is suffixed with `impi`. Use `libfabric provider` to check with Intel MPI is using EFA or the TCP/IP network. If it's using EFA, the value is `efa`. If it's using TCP/IP, the value is `tcp;ofi_rxm`. To enable debugging information, do one of the following depending on your shell. ------ #### [ bash shells ] Add the following statement to `/home/username/.bashrc` and `/home/username/.bash_profile`. ``` export I_MPI_DEBUG=value ``` ------ #### [ csh and tcsh shells ] Add the following statement to `/home/username/.cshrc`. ``` setenv I_MPI_DEBUG value ``` ------ 1. By default, Intel MPI uses the operating system’s shared memory (`shm`) for intra-node communication, and it uses Libfabric (`ofi`) only for inter-node communication. Generally, this configuration provides the best performance. However, in some cases the Intel MPI shm fabric can cause certain applications to hang indefinitely. To resolve this issue, you can force Intel MPI to use Libfabric for both intra-node and inter-node communication. To do this, do one of the following depending on your shell. ------ #### [ bash shells ] Add the following statement to `/home/username/.bashrc` and `/home/username/.bash_profile`. ``` export I_MPI_FABRICS=ofi ``` ------ #### [ csh and tcsh shells ] Add the following statement to `/home/username/.cshrc`. ``` setenv I_MPI_FABRICS ofi ``` ------ **Note** The EFA Libfabric provider uses the operating system's shared memory for intra-node communication. This means that setting `I_MPI_FABRICS` to `ofi` yields similar performance to the default `shm:ofi` configuration. 1. Log out of the instance and then log back in. If you no longer want to use Intel MPI, remove the environment variables from the shell startup scripts. ## Step 6: Disable ptrace protection To improve your HPC application's performance, Libfabric uses the instance's local memory for interprocess communications when the processes are running on the same instance. The shared memory feature uses Cross Memory Attach (CMA), which is not supported with *ptrace protection*. If you are using a Linux distribution that has ptrace protection enabled by default, such as Ubuntu, you must disable it. If your Linux distribution does not have ptrace protection enabled by default, skip this step. **To disable ptrace protection** Do one of the following: + To temporarily disable ptrace protection for testing purposes, run the following command. ``` $ sudo sysctl -w kernel.yama.ptrace_scope=0 ``` + To permanently disable ptrace protection, add `kernel.yama.ptrace_scope = 0` to `/etc/sysctl.d/10-ptrace.conf` and reboot the instance. ## Step 7. Confirm installation **To confirm successful installation** 1. To confirm that MPI was successfully installed, run the following command: ``` $ which mpicc ``` + For Open MPI, the returned path should include `/opt/amazon/` + For Intel MPI, the returned path should include `/opt/intel/`. If you do not get the expected output, ensure that you have sourced the Intel MPI `vars.sh` script. 1. To confirm that the EFA software components and Libfabric were successfully installed, run the following command. ``` $ fi_info -p efa -t FI_EP_RDM ``` The command should return information about the Libfabric EFA interfaces. The following example shows the command output. ``` provider: efa fabric: EFA-fe80::94:3dff:fe89:1b70 domain: efa_0-rdm version: 2.0 type: FI_EP_RDM protocol: FI_PROTO_EFA ``` ## Step 8: Install your HPC application Install the HPC application on the temporary instance. The installation procedure varies depending on the specific HPC application. For more information, see [Manage software on your AL2 instance](https://docs.aws.amazon.com/linux/al2/ug/managing-software.html) in the *Amazon Linux 2 User Guide*. **Note** Refer to your HPC application’s documentation for installation instructions. ## Step 9: Create an EFA-enabled AMI After you have installed the required software components, you create an AMI that you can reuse to launch your EFA-enabled instances. **To create an AMI from your temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the temporary instance that you created and choose **Actions**, **Image**, **Create image**. 1. For **Create image**, do the following: 1. For **Image name**, enter a descriptive name for the AMI. 1. (Optional) For **Image description**, enter a brief description of the purpose of the AMI. 1. Choose **Create image**. 1. In the navigation pane, choose **AMIs**. 1. Locate the AMI tht you created in the list. Wait for the status to change from `pending` to `available` before continuing to the next step. ## Step 10: Launch EFA-enabled instances into a cluster placement group Launch your EFA-enabled instances into a cluster placement group using the EFA-enabled AMI that you created in **Step 7**, and the EFA-enabled security group that you created in **Step 1**. **Note** It is not an absolute requirement to launch your EFA-enabled instances into a cluster placement group. However, we do recommend running your EFA-enabled instances in a cluster placement group as it launches the instances into a low-latency group in a single Availability Zone. To ensure that capacity is available as you scale your cluster’s instances, you can create a Capacity Reservation for your cluster placement group. For more information, see [Use Capacity Reservations with cluster placement groups](cr-cpg.md). **To launch an instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**, and then choose **Launch Instances** to open the new launch instance wizard. 1. (*Optional*) In the **Name and tags** section, provide a name for the instance, such as `EFA-instance`. The name is assigned to the instance as a resource tag (`Name=EFA-instance`). 1. In the **Application and OS Images** section, choose **My AMIs**, and then select the AMI that you created in the previous step. 1. In the **Instance type** section, select a [supported instance type](efa.md#efa-instance-types). 1. In the **Key pair** section, select the key pair to use for the instance. 1. In the **Network settings** section, choose **Edit**, and then do the following: 1. For **Subnet**, choose the subnet in which to launch the instance. **Important** You must select a subnet. If you do not select a subnet, you can't enable the instance for EFA. 1. For **Firewall (security groups)**, choose **Select existing security group**, and then select the security group that you created in the previous step. 1. Expand the **Advanced network configuration** section. For **Network interface 1**, select **Network card index = 0**, **Device index = 0**, and **Interface type = EFA with ENA**. (*Optional*) If you are using a multi-card instance type, such as `p4d.24xlarge` or `p5.48xlarge`, for each additional network interface required, choose **Add network interface**, for **Network card index** select the next unused index, and then select **Device index = 1** and **Interface type = EFA with ENA** or **EFA-only**. 1. (*Optional*) In the **Storage** section, configure the volumes as needed. 1. In the **Advanced details** section, for **Placement group name**, select the cluster placement group into which to launch the instances. If you need to create a new cluster placement group, choose **Create new placement group**. 1. In the **Summary** panel on the right, for **Number of instances**, enter the number of EFA-enabled instances that you want to launch, and then choose **Launch instance**. ## Step 11: Terminate the temporary instance At this point, you no longer need the instance that you launched in [Step 2](#efa-start-tempinstance). You can terminate the instance to stop incurring charges for it. **To terminate the temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the temporary instance that you created and then choose **Actions**, **Instance state**, **Terminate (delete) instance**. 1. When prompted for confirmation, choose **Terminate (delete)**. ## Step 12: Enable passwordless SSH To enable your applications to run across all of the instances in your cluster, you must enable passwordless SSH access from the leader node to the member nodes. The leader node is the instance from which you run your applications. The remaining instances in the cluster are the member nodes. **To enable passwordless SSH between the instances in the cluster** 1. Select one instance in the cluster as the leader node, and connect to it. 1. Disable `strictHostKeyChecking` and enable `ForwardAgent` on the leader node. Open `~/.ssh/config` using your preferred text editor and add the following. ``` Host * ForwardAgent yes Host * StrictHostKeyChecking no ``` 1. Generate an RSA key pair. ``` $ ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa ``` The key pair is created in the `$HOME/.ssh/` directory. 1. Change the permissions of the private key on the leader node. ``` $ chmod 600 ~/.ssh/id_rsa chmod 600 ~/.ssh/config ``` 1. Open `~/.ssh/id_rsa.pub` using your preferred text editor and copy the key. 1. For each member node in the cluster, do the following: 1. Connect to the instance. 1. Open `~/.ssh/authorized_keys` using your preferred text editor and add the public key that you copied earlier. 1. To test that the passwordless SSH is functioning as expected, connect to your leader node and run the following command. ``` $ ssh member_node_private_ip ``` You should connect to the member node without being prompted for a key or password. # Get started with EFA and NCCL for ML workloads on Amazon EC2 Get started with EFA and NCCL The NVIDIA Collective Communications Library (NCCL) is a library of standard collective communication routines for multiple GPUs across a single node or multiple nodes. NCCL can be used together with EFA, Libfabric, and MPI to support various machine learning workloads. For more information, see the [NCCL](https://developer.nvidia.com/nccl) website. **Requirements** + Only accelerated computing P series instance types are supported. For more information, see [ Amazon EC2 accelerated computing instances](https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html#ac-sizes). + Only Amazon Linux 2023, Amazon Linux 2, Ubuntu 24.04, and Ubuntu 22.04 base AMIs are supported. + Only NCCL 2.4.2 and later is supported with EFA. For more information about running machine learning workloads with EFA and NCCL using an AWS Deep Learning AMIs, see [ Using EFA on the DLAMI](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-efa-using.html) in the *AWS Deep Learning AMIs Developer Guide*. **Topics** + [ ## Step 1: Prepare an EFA-enabled security group ](#nccl-start-base-setup) + [ ## Step 2: Launch a temporary instance ](#nccl-start-base-temp) + [ ## Step 3: Install Nvidia GPU drivers, Nvidia CUDA toolkit, and cuDNN ](#nccl-start-base-drivers) + [ ## Step 4: Install GDRCopy ](#nccl-start-base-gdrcopy) + [ ## Step 5: Install the EFA software ](#nccl-start-base-enable) + [ ## Step 6: Install NCCL ](#nccl-start-base-nccl) + [ ## Step 7: Install the NCCL tests ](#nccl-start-base-tests) + [ ## Step 8: Test your EFA and NCCL configuration ](#nccl-start-base-test) + [ ## Step 9: Install your machine learning applications ](#nccl-start-base-app) + [ ## Step 10: Create an EFA and NCCL-enabled AMI ](#nccl-start-base-ami) + [ ## Step 11: Terminate the temporary instance ](#nccl-start-base-terminate) + [ ## Step 12: Launch EFA and NCCL-enabled instances into a cluster placement group ](#nccl-start-base-cluster) + [ ## Step 13: Enable passwordless SSH ](#nccl-start-base-passwordless) ## Step 1: Prepare an EFA-enabled security group An EFA requires a security group that allows all inbound and outbound traffic to and from the security group itself. The following procedure creates a security group that allows all inbound and outbound traffic to and from itself, and that allows inbound SSH traffic from any IPv4 address for SSH connectivity. **Important** This security group is intended for testing purposes only. For your production environments, we recommend that you create an inbound SSH rule that allows traffic only from the IP address from which you are connecting, such as the IP address of your computer, or a range of IP addresses in your local network. For other scenarios, see [Security group rules for different use cases](security-group-rules-reference.md). **To create an EFA-enabled security group** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Security Groups** and then choose **Create security group**. 1. In the **Create security group** window, do the following: 1. For **Security group name**, enter a descriptive name for the security group, such as `EFA-enabled security group`. 1. (Optional) For **Description**, enter a brief description of the security group. 1. For **VPC**, select the VPC into which you intend to launch your EFA-enabled instances. 1. Choose **Create security group**. 1. Select the security group that you created, and on the **Details** tab, copy the **Security group ID**. 1. With the security group still selected, choose **Actions**, **Edit inbound rules**, and then do the following: 1. Choose **Add rule**. 1. For **Type**, choose **All traffic**. 1. For **Source type**, choose **Custom** and paste the security group ID that you copied into the field. 1. Choose **Add rule**. 1. For **Type**, choose **SSH**. 1. For **Source type**, choose **Anywhere-IPv4**. 1. Choose **Save rules**. 1. With the security group still selected, choose **Actions**, **Edit outbound rules**, and then do the following: 1. Choose **Add rule**. 1. For **Type**, choose **All traffic**. 1. For **Destination type**, choose **Custom** and paste the security group ID that you copied into the field. 1. Choose **Save rules**. ## Step 2: Launch a temporary instance Launch a temporary instance that you can use to install and configure the EFA software components. You use this instance to create an EFA-enabled AMI from which you can launch your EFA-enabled instances. **To launch a temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**, and then choose **Launch Instances** to open the new launch instance wizard. 1. (*Optional*) In the **Name and tags** section, provide a name for the instance, such as `EFA-instance`. The name is assigned to the instance as a resource tag (`Name=EFA-instance`). 1. In the **Application and OS Images** section, select an AMI for one of the supported operating systems. 1. In the **Instance type** section, select a supported instance type. 1. In the **Key pair** section, select the key pair to use for the instance. 1. In the **Network settings** section, choose **Edit**, and then do the following: 1. For **Subnet**, choose the subnet in which to launch the instance. **Important** You must select a subnet. If you do not select a subnet, you can't enable the instance for EFA. 1. For **Firewall (security groups)**, choose **Select existing security group**, and then select the security group that you created in the previous step. 1. Expand the **Advanced network configuration** section. For **Network interface 1**, select **Network card index = 0**, **Device index = 0**, and **Interface type = EFA with ENA**. (*Optional*) If you are using a multi-card instance type, such as `p4d.24xlarge` or `p5.48xlarge`, for each additional network interface required, choose **Add network interface**, for **Network card index** select the next unused index, and then select **Device index = 1** and **Interface type = EFA with ENA** or **EFA-only**. 1. In the **Storage** section, configure the volumes as needed. **Note** You must provision an additional 10 to 20 GiB of storage for the Nvidia CUDA Toolkit. If you do not provision enough storage, you will receive an `insufficient disk space` error when attempting to install the Nvidia drivers and CUDA toolkit. 1. In the **Summary** panel on the right, choose **Launch instance**. ## Step 3: Install Nvidia GPU drivers, Nvidia CUDA toolkit, and cuDNN ------ #### [ Amazon Linux 2023 and Amazon Linux 2 ] **To install the Nvidia GPU drivers, Nvidia CUDA toolkit, and cuDNN** 1. To ensure that all of your software packages are up to date, perform a quick software update on your instance. ``` $ sudo yum upgrade -y && sudo reboot ``` After the instance has rebooted, reconnect to it. 1. Install the utilities that are needed to install the Nvidia GPU drivers and the Nvidia CUDA toolkit. ``` $ sudo yum groupinstall 'Development Tools' -y ``` 1. Disable the `nouveau` open source drivers. 1. Install the required utilities and the kernel headers package for the version of the kernel that you are currently running. ``` $ sudo yum install -y wget kernel-devel-$(uname -r) kernel-headers-$(uname -r) ``` 1. Add `nouveau` to the `/etc/modprobe.d/blacklist.conf `deny list file. ``` $ cat << EOF | sudo tee --append /etc/modprobe.d/blacklist.conf blacklist vga16fb blacklist nouveau blacklist rivafb blacklist nvidiafb blacklist rivatv EOF ``` 1. Append `GRUB_CMDLINE_LINUX="rdblacklist=nouveau"` to the `grub` file and rebuild the Grub configuration. ``` $ echo 'GRUB_CMDLINE_LINUX="rdblacklist=nouveau"' | sudo tee -a /etc/default/grub \ && sudo grub2-mkconfig -o /boot/grub2/grub.cfg ``` 1. Reboot the instance and reconnect to it. 1. Prepare the required repositories 1. Enable the EPEL repository and set the distribution to `rhel7`. ``` $ sudo amazon-linux-extras install epel \ && distribution='rhel7' ``` 1. Set up the CUDA network repository and update the repository cache. ``` $ ARCH=$( /bin/arch ) \ && sudo yum-config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/$distribution/${ARCH}/cuda-$distribution.repo \ && sudo yum clean expire-cache ``` 1. (*Kernel version 5.10 only*) Perform these steps only if you are using Amazon Linux 2 with kernel version 5.10. If you are using Amazon Linux 2 with kernel version 4.12, skip these steps. To check your kernel version, run **uname -r**. 1. Create the Nvidia driver configuration file named `/etc/dkms/nvidia.conf`. ``` $ sudo mkdir -p /etc/dkms \ && echo "MAKE[0]=\"'make' -j2 module SYSSRC=\${kernel_source_dir} IGNORE_XEN_PRESENCE=1 IGNORE_PREEMPT_RT_PRESENCE=1 IGNORE_CC_MISMATCH=1 CC=/usr/bin/gcc10-gcc\"" | sudo tee /etc/dkms/nvidia.conf ``` 1. (`p4d.24xlarge` and `p5.48xlarge` only) Copy the Nvidia driver configuration file. ``` $ sudo cp /etc/dkms/nvidia.conf /etc/dkms/nvidia-open.conf ``` 1. Install the Nvidia GPU drivers, NVIDIA CUDA toolkit, and cuDNN. ``` $ sudo yum clean all \ && sudo yum -y install nvidia-driver-latest-dkms \ && sudo yum -y install cuda-drivers-fabricmanager cuda libcudnn8-devel ``` 1. Reboot the instance and reconnect to it. 1. (`p4d.24xlarge` and `p5.48xlarge` only) Start the Nvidia Fabric Manager service, and ensure that it starts automatically when the instance starts. Nvidia Fabric Manager is required for NV Switch Management. ``` $ sudo systemctl enable nvidia-fabricmanager && sudo systemctl start nvidia-fabricmanager ``` 1. Ensure that the CUDA paths are set each time that the instance starts. + For *bash* shells, add the following statements to `/home/username/.bashrc` and `/home/username/.bash_profile`. ``` export PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH ``` + For *tcsh* shells, add the following statements to `/home/username/.cshrc`. ``` setenv PATH=/usr/local/cuda/bin:$PATH setenv LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH ``` 1. To confirm that the Nvidia GPU drivers are functional, run the following command. ``` $ nvidia-smi -q | head ``` The command should return information about the Nvidia GPUs, Nvidia GPU drivers, and Nvidia CUDA toolkit. ------ #### [ Ubuntu 24.04 and Ubuntu 22.04 ] **To install the Nvidia GPU drivers, Nvidia CUDA toolkit, and cuDNN** 1. To ensure that all of your software packages are up to date, perform a quick software update on your instance. ``` $ sudo apt-get update && sudo apt-get upgrade -y ``` 1. Install the utilities that are needed to install the Nvidia GPU drivers and the Nvidia CUDA toolkit. ``` $ sudo apt-get update && sudo apt-get install build-essential -y ``` 1. To use the Nvidia GPU driver, you must first disable the `nouveau` open source drivers. 1. Install the required utilities and the kernel headers package for the version of the kernel that you are currently running. ``` $ sudo apt-get install -y gcc make linux-headers-$(uname -r) ``` 1. Add `nouveau` to the `/etc/modprobe.d/blacklist.conf `deny list file. ``` $ cat << EOF | sudo tee --append /etc/modprobe.d/blacklist.conf blacklist vga16fb blacklist nouveau blacklist rivafb blacklist nvidiafb blacklist rivatv EOF ``` 1. Open `/etc/default/grub` using your preferred text editor and add the following. ``` GRUB_CMDLINE_LINUX="rdblacklist=nouveau" ``` 1. Rebuild the Grub configuration. ``` $ sudo update-grub ``` 1. Reboot the instance and reconnect to it. 1. Add the CUDA repository and install the Nvidia GPU drivers, NVIDIA CUDA toolkit, and cuDNN. + `p3dn.24xlarge` ``` $ sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub \ && wget -O /tmp/deeplearning.deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb \ && sudo dpkg -i /tmp/deeplearning.deb \ && wget -O /tmp/cuda.pin https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin \ && sudo mv /tmp/cuda.pin /etc/apt/preferences.d/cuda-repository-pin-600 \ && sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub \ && sudo add-apt-repository 'deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /' \ && sudo apt update \ && sudo apt install nvidia-dkms-535 \ && sudo apt install -o Dpkg::Options::='--force-overwrite' cuda-drivers-535 cuda-toolkit-12-3 libcudnn8 libcudnn8-dev -y ``` + `p4d.24xlarge` and `p5.48xlarge` ``` $ sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub \ && wget -O /tmp/deeplearning.deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb \ && sudo dpkg -i /tmp/deeplearning.deb \ && wget -O /tmp/cuda.pin https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin \ && sudo mv /tmp/cuda.pin /etc/apt/preferences.d/cuda-repository-pin-600 \ && sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub \ && sudo add-apt-repository 'deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /' \ && sudo apt update \ && sudo apt install nvidia-kernel-open-535 \ && sudo apt install -o Dpkg::Options::='--force-overwrite' cuda-drivers-535 cuda-toolkit-12-3 libcudnn8 libcudnn8-dev -y ``` 1. Reboot the instance and reconnect to it. 1. (`p4d.24xlarge` and `p5.48xlarge` only) Install the Nvidia Fabric Manager. 1. You must install the version of the Nvidia Fabric Manager that matches the version of the Nvidia kernel module that you installed in the previous step. Run the following command to determine the version of the Nvidia kernel module. ``` $ cat /proc/driver/nvidia/version | grep "Kernel Module" ``` The following is example output. ``` NVRM version: NVIDIA UNIX x86_64 Kernel Module 450.42.01 Tue Jun 15 21:26:37 UTC 2021 ``` In the example above, major version `450` of the kernel module was installed. This means that you need to install Nvidia Fabric Manager version `450`. 1. Install the Nvidia Fabric Manager. Run the following command and specify the major version identified in the previous step. ``` $ sudo apt install -o Dpkg::Options::='--force-overwrite' nvidia-fabricmanager-major_version_number ``` For example, if major version `450` of the kernel module was installed, use the following command to install the matching version of Nvidia Fabric Manager. ``` $ sudo apt install -o Dpkg::Options::='--force-overwrite' nvidia-fabricmanager-450 ``` 1. Start the service, and ensure that it starts automatically when the instance starts. Nvidia Fabric Manager is required for NV Switch Management. ``` $ sudo systemctl start nvidia-fabricmanager && sudo systemctl enable nvidia-fabricmanager ``` 1. Ensure that the CUDA paths are set each time that the instance starts. + For *bash* shells, add the following statements to `/home/username/.bashrc` and `/home/username/.bash_profile`. ``` export PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH ``` + For *tcsh* shells, add the following statements to `/home/username/.cshrc`. ``` setenv PATH=/usr/local/cuda/bin:$PATH setenv LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH ``` 1. To confirm that the Nvidia GPU drivers are functional, run the following command. ``` $ nvidia-smi -q | head ``` The command should return information about the Nvidia GPUs, Nvidia GPU drivers, and Nvidia CUDA toolkit. ------ ## Step 4: Install GDRCopy Install GDRCopy to improve the performance of Libfabric. For more information about GDRCopy, see the [GDRCopy repository](https://github.com/NVIDIA/gdrcopy). ------ #### [ Amazon Linux 2023 and Amazon Linux 2 ] **To install GDRCopy** 1. Install the required dependencies. ``` $ sudo yum -y install dkms rpm-build make check check-devel subunit subunit-devel ``` 1. Download and extract the GDRCopy package. ``` $ wget https://github.com/NVIDIA/gdrcopy/archive/refs/tags/v2.4.tar.gz \ && tar xf v2.4.tar.gz ; cd gdrcopy-2.4/packages ``` 1. Build the GDRCopy RPM package. ``` $ CUDA=/usr/local/cuda ./build-rpm-packages.sh ``` 1. Install the GDRCopy RPM package. ``` $ sudo rpm -Uvh gdrcopy-kmod-2.4-1dkms.noarch*.rpm \ && sudo rpm -Uvh gdrcopy-2.4-1.x86_64*.rpm \ && sudo rpm -Uvh gdrcopy-devel-2.4-1.noarch*.rpm ``` ------ #### [ Ubuntu 24.04 and Ubuntu 22.04 ] **To install GDRCopy** 1. Install the required dependencies. ``` $ sudo apt -y install build-essential devscripts debhelper check libsubunit-dev fakeroot pkg-config dkms ``` 1. Download and extract the GDRCopy package. ``` $ wget https://github.com/NVIDIA/gdrcopy/archive/refs/tags/v2.4.tar.gz \ && tar xf v2.4.tar.gz \ && cd gdrcopy-2.4/packages ``` 1. Build the GDRCopy RPM package. ``` $ CUDA=/usr/local/cuda ./build-deb-packages.sh ``` 1. Install the GDRCopy RPM package. ``` $ sudo dpkg -i gdrdrv-dkms_2.4-1_amd64.*.deb \ && sudo dpkg -i libgdrapi_2.4-1_amd64.*.deb \ && sudo dpkg -i gdrcopy-tests_2.4-1_amd64.*.deb \ && sudo dpkg -i gdrcopy_2.4-1_amd64.*.deb ``` ------ ## Step 5: Install the EFA software Install the EFA-enabled kernel, EFA drivers, Libfabric, aws-ofi-nccl plugin, and Open MPI stack that is required to support EFA on your instance. **To install the EFA software** 1. Connect to the instance you launched. For more information, see [Connect to your Linux instance using SSH](connect-to-linux-instance.md). 1. Download the EFA software installation files. The software installation files are packaged into a compressed tarball (`.tar.gz`) file. To download the latest *stable* version, use the following command. You can also get the latest version by replacing the version number with `latest` in the preceding command. ``` $ curl -O https://efa-installer.amazonaws.com/aws-efa-installer-1.47.0.tar.gz ``` 1. (*Optional*) Verify the authenticity and integrity of the EFA tarball (`.tar.gz`) file. We recommend that you do this to verify the identity of the software publisher and to check that the file has not been altered or corrupted since it was published. If you do not want to verify the tarball file, skip this step. **Note** Alternatively, if you prefer to verify the tarball file by using an MD5 or SHA256 checksum instead, see [Verify the EFA installer using a checksum](efa-verify.md). 1. Download the public GPG key and import it into your keyring. ``` $ wget https://efa-installer.amazonaws.com/aws-efa-installer.key && gpg --import aws-efa-installer.key ``` The command should return a key value. Make a note of the key value, because you need it in the next step. 1. Verify the GPG key's fingerprint. Run the following command and specify the key value from the previous step. ``` $ gpg --fingerprint key_value ``` The command should return a fingerprint that is identical to `4E90 91BC BB97 A96B 26B1 5E59 A054 80B1 DD2D 3CCC`. If the fingerprint does not match, don't run the EFA installation script, and contact Support. 1. Download the signature file and verify the signature of the EFA tarball file. ``` $ wget https://efa-installer.amazonaws.com/aws-efa-installer-1.47.0.tar.gz.sig && gpg --verify ./aws-efa-installer-1.47.0.tar.gz.sig ``` The following shows example output. ``` gpg: Signature made Wed 29 Jul 2020 12:50:13 AM UTC using RSA key ID DD2D3CCC gpg: Good signature from "Amazon EC2 EFA " gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 4E90 91BC BB97 A96B 26B1 5E59 A054 80B1 DD2D 3CCC ``` If the result includes `Good signature`, and the fingerprint matches the fingerprint returned in the previous step, proceed to the next step. If not, don't run the EFA installation script, and contact Support. 1. Extract the files from the compressed `.tar.gz` file and navigate into the extracted directory. ``` $ tar -xf aws-efa-installer-1.47.0.tar.gz && cd aws-efa-installer ``` 1. Run the EFA software installation script. **Note** From EFA 1.30.0, both Open MPI 4.1 and Open MPI 5 are installed by default. Unless you need Open MPI 5, we recommend that you install only Open MPI 4.1. The following command installs Open MPI 4.1 only. If you want to install Open MPI 4.1 and Open MPI 5, remove `--mpi=openmpi4`. ``` $ sudo ./efa_installer.sh -y --mpi=openmpi4 ``` **Libfabric** is installed in the `/opt/amazon/efa` directory. The **aws-ofi-nccl plugin** is installed in the `/opt/amazon/ofi-nccl` directory. **Open MPI** is installed in the `/opt/amazon/openmpi` directory. 1. If the EFA installer prompts you to reboot the instance, do so and then reconnect to the instance. Otherwise, log out of the instance and then log back in to complete the installation. 1. Confirm that the EFA software components were successfully installed. ``` $ fi_info -p efa -t FI_EP_RDM ``` The command should return information about the Libfabric EFA interfaces. The following example shows the command output. + `p3dn.24xlarge` with single network interface ``` provider: efa fabric: EFA-fe80::94:3dff:fe89:1b70 domain: efa_0-rdm version: 2.0 type: FI_EP_RDM protocol: FI_PROTO_EFA ``` + `p4d.24xlarge` and `p5.48xlarge` with multiple network interfaces ``` provider: efa fabric: EFA-fe80::c6e:8fff:fef6:e7ff domain: efa_0-rdm version: 111.0 type: FI_EP_RDM protocol: FI_PROTO_EFA provider: efa fabric: EFA-fe80::c34:3eff:feb2:3c35 domain: efa_1-rdm version: 111.0 type: FI_EP_RDM protocol: FI_PROTO_EFA provider: efa fabric: EFA-fe80::c0f:7bff:fe68:a775 domain: efa_2-rdm version: 111.0 type: FI_EP_RDM protocol: FI_PROTO_EFA provider: efa fabric: EFA-fe80::ca7:b0ff:fea6:5e99 domain: efa_3-rdm version: 111.0 type: FI_EP_RDM protocol: FI_PROTO_EFA ``` ## Step 6: Install NCCL Install NCCL. For more information about NCCL, see the [NCCL repository](https://github.com/NVIDIA/nccl). **To install NCCL** 1. Navigate to the `/opt` directory. ``` $ cd /opt ``` 1. Clone the official NCCL repository to the instance and navigate into the local cloned repository. ``` $ sudo git clone https://github.com/NVIDIA/nccl.git -b v2.23.4-1 && cd nccl ``` 1. Build and install NCCL and specify the CUDA installation directory. ``` $ sudo make -j src.build CUDA_HOME=/usr/local/cuda ``` ## Step 7: Install the NCCL tests Install the NCCL tests. The NCCL tests enable you to confirm that NCCL is properly installed and that it is operating as expected. For more information about the NCCL tests, see the [nccl-tests repository](https://github.com/NVIDIA/nccl-tests). **To install the NCCL tests** 1. Navigate to your home directory. ``` $ cd $HOME ``` 1. Clone the official nccl-tests repository to the instance and navigate into the local cloned repository. ``` $ git clone https://github.com/NVIDIA/nccl-tests.git && cd nccl-tests ``` 1. Add the Libfabric directory to the `LD_LIBRARY_PATH` variable. + Amazon Linux 2023 and Amazon Linux 2 ``` $ export LD_LIBRARY_PATH=/opt/amazon/efa/lib64:$LD_LIBRARY_PATH ``` + Ubuntu 24.04 and Ubuntu 22.04 ``` $ export LD_LIBRARY_PATH=/opt/amazon/efa/lib:$LD_LIBRARY_PATH ``` 1. Install the NCCL tests and specify the MPI, NCCL, and CUDA installation directories. ``` $ make MPI=1 MPI_HOME=/opt/amazon/openmpi NCCL_HOME=/opt/nccl/build CUDA_HOME=/usr/local/cuda ``` ## Step 8: Test your EFA and NCCL configuration Run a test to ensure that your temporary instance is properly configured for EFA and NCCL. **To test your EFA and NCCL configuration** 1. Create a host file that specifies the hosts on which to run the tests. The following command creates a host file named `my-hosts` that includes a reference to the instance itself. ------ #### [ IMDSv2 ] ``` [ec2-user ~]$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \ && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/local-ipv4 >> my-hosts ``` ------ #### [ IMDSv1 ] ``` [ec2-user ~]$ curl http://169.254.169.254/latest/meta-data/local-ipv4 >> my-hosts ``` ------ 1. Run the test and specify the host file (`--hostfile`) and the number of GPUs to use (`-n`). The following command runs the `all_reduce_perf` test on 8 GPUs on the instance itself, and specifies the following environment variables. + `FI_EFA_USE_DEVICE_RDMA=1`—(`p4d.24xlarge` only) uses the device's RDMA functionality for one-sided and two-sided transfer. + `NCCL_DEBUG=INFO`—enables detailed debugging output. You can also specify `VERSION` to print only the NCCL version at the start of the test, or `WARN` to receive only error messages. For more information about the NCCL test arguments, see the [NCCL Tests README](https://github.com/NVIDIA/nccl-tests/blob/master/README.md) in the official nccl-tests repository. + `p3dn.24xlarge` ``` $ /opt/amazon/openmpi/bin/mpirun \ -x LD_LIBRARY_PATH=/opt/nccl/build/lib:/usr/local/cuda/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/amazon/ofi-nccl/lib:$LD_LIBRARY_PATH \ -x NCCL_DEBUG=INFO \ --hostfile my-hosts -n 8 -N 8 \ --mca pml ^cm --mca btl tcp,self --mca btl_tcp_if_exclude lo,docker0 --bind-to none \ $HOME/nccl-tests/build/all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100 ``` + `p4d.24xlarge` and `p5.48xlarge` ``` $ /opt/amazon/openmpi/bin/mpirun \ -x FI_EFA_USE_DEVICE_RDMA=1 \ -x LD_LIBRARY_PATH=/opt/nccl/build/lib:/usr/local/cuda/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/amazon/ofi-nccl/lib:$LD_LIBRARY_PATH \ -x NCCL_DEBUG=INFO \ --hostfile my-hosts -n 8 -N 8 \ --mca pml ^cm --mca btl tcp,self --mca btl_tcp_if_exclude lo,docker0 --bind-to none \ $HOME/nccl-tests/build/all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100 ``` 1. You can confirm that EFA is active as the underlying provider for NCCL when the `NCCL_DEBUG` log is printed. ``` ip-192-168-2-54:14:14 [0] NCCL INFO NET/OFI Selected Provider is efa* ``` The following additional information is displayed when using a `p4d.24xlarge` instance. ``` ip-192-168-2-54:14:14 [0] NCCL INFO NET/OFI Running on P4d platform, Setting NCCL_TOPO_FILE environment variable to /home/ec2-user/install/plugin/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml ``` ## Step 9: Install your machine learning applications Install the machine learning applications on the temporary instance. The installation procedure varies depending on the specific machine learning application. For more information about installing software on your Linux instance, see [Manage software on your Amazon Linux 2 instance](https://docs.aws.amazon.com/linux/al2/ug/managing-software.html). **Note** Refer to your machine learning application’s documentation for installation instructions. ## Step 10: Create an EFA and NCCL-enabled AMI After you have installed the required software components, you create an AMI that you can reuse to launch your EFA-enabled instances. **To create an AMI from your temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the temporary instance that you created and choose **Actions**, **Image**, **Create image**. 1. For **Create image**, do the following: 1. For **Image name**, enter a descriptive name for the AMI. 1. (Optional) For **Image description**, enter a brief description of the purpose of the AMI. 1. Choose **Create image**. 1. In the navigation pane, choose **AMIs**. 1. Locate the AMI tht you created in the list. Wait for the status to change from `pending` to `available` before continuing to the next step. ## Step 11: Terminate the temporary instance At this point, you no longer need the temporary instance that you launched. You can terminate the instance to stop incurring charges for it. **To terminate the temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the temporary instance that you created and then choose **Actions**, **Instance state**, **Terminate instance**. 1. When prompted for confirmation, choose **Terminate**. ## Step 12: Launch EFA and NCCL-enabled instances into a cluster placement group Launch your EFA and NCCL-enabled instances into a cluster placement group using the EFA-enabled AMI and the EFA-enabled security group that you created earlier. **Note** It is not an absolute requirement to launch your EFA-enabled instances into a cluster placement group. However, we do recommend running your EFA-enabled instances in a cluster placement group as it launches the instances into a low-latency group in a single Availability Zone. To ensure that capacity is available as you scale your cluster’s instances, you can create a Capacity Reservation for your cluster placement group. For more information, see [Use Capacity Reservations with cluster placement groups](cr-cpg.md). ------ #### [ New console ] **To launch a temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**, and then choose **Launch Instances** to open the new launch instance wizard. 1. (*Optional*) In the **Name and tags** section, provide a name for the instance, such as `EFA-instance`. The name is assigned to the instance as a resource tag (`Name=EFA-instance`). 1. In the **Application and OS Images** section, choose **My AMIs**, and then select the AMI that you created in the previous step. 1. In the **Instance type** section, select either `p3dn.24xlarge` or `p4d.24xlarge`. 1. In the **Key pair** section, select the key pair to use for the instance. 1. In the **Network settings** section, choose **Edit**, and then do the following: 1. For **Subnet**, choose the subnet in which to launch the instance. If you do not select a subnet, you can't enable the instance for EFA. 1. For **Firewall (security groups)**, choose **Select existing security group**, and then select the security group that you created in the previous step. 1. Expand the **Advanced network configuration** section. For **Network interface 1**, select **Network card index = 0**, **Device index = 0**, and **Interface type = EFA with ENA**. (Optional) If you are using a multi-card instance type, such as `p4d.24xlarge` or `p5.48xlarge`, for each additional network interface required, choose **Add network interface**, for **Network card index** select the next unused index, and then select **Device index = 1** and **Interface type = EFA eith ENA** or **EFA-only**. 1. (*Optional*) In the **Storage** section, configure the volumes as needed. 1. In the **Advanced details** section, for **Placement group name**, select the cluster placement group into which to launch the instance. If you need to create a new cluster placement group, choose **Create new placement group**. 1. In the **Summary** panel on the right, for **Number of instances**, enter the number of EFA-enabled instances that you want to launch, and then choose **Launch instance**. ------ #### [ Old console ] **To launch your EFA and NCCL-enabled instances into a cluster placement group** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. Choose **Launch Instance**. 1. On the **Choose an AMI** page, choose **My AMIs**, find the AMI that you created earlier, and then choose **Select**. 1. On the **Choose an Instance Type** page, select **p3dn.24xlarge** and then choose **Next: Configure Instance Details**. 1. On the **Configure Instance Details** page, do the following: 1. For **Number of instances**, enter the number of EFA and NCCL-enabled instances that you want to launch. 1. For **Network** and **Subnet**, select the VPC and subnet into which to launch the instances. 1. For **Placement group**, select **Add instance to placement group**. 1. For **Placement group name**, select **Add to a new placement group**, and then enter a descriptive name for the placement group. Then for **Placement group strategy**, select **cluster**. 1. For **EFA**, choose **Enable**. 1. In the **Network Interfaces** section, for device **eth0**, choose **New network interface**. You can optionally specify a primary IPv4 address and one or more secondary IPv4 addresses. If you are launching the instance into a subnet that has an associated IPv6 CIDR block, you can optionally specify a primary IPv6 address and one or more secondary IPv6 addresses. 1. Choose **Next: Add Storage**. 1. On the **Add Storage** page, specify the volumes to attach to the instances in addition to the volumes specified by the AMI (such as the root device volume). Then choose **Next: Add Tags**. 1. On the **Add Tags** page, specify tags for the instances, such as a user-friendly name, and then choose **Next: Configure Security Group**. 1. On the **Configure Security Group** page, for **Assign a security group**, select **Select an existing security group**, and then select the security group that you created earlier. 1. Choose **Review and Launch**. 1. On the **Review Instance Launch** page, review the settings, and then choose **Launch** to choose a key pair and to launch your instances. ------ ## Step 13: Enable passwordless SSH To enable your applications to run across all of the instances in your cluster, you must enable passwordless SSH access from the leader node to the member nodes. The leader node is the instance from which you run your applications. The remaining instances in the cluster are the member nodes. **To enable passwordless SSH between the instances in the cluster** 1. Select one instance in the cluster as the leader node, and connect to it. 1. Disable `strictHostKeyChecking` and enable `ForwardAgent` on the leader node. Open `~/.ssh/config` using your preferred text editor and add the following. ``` Host * ForwardAgent yes Host * StrictHostKeyChecking no ``` 1. Generate an RSA key pair. ``` $ ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa ``` The key pair is created in the `$HOME/.ssh/` directory. 1. Change the permissions of the private key on the leader node. ``` $ chmod 600 ~/.ssh/id_rsa chmod 600 ~/.ssh/config ``` 1. Open `~/.ssh/id_rsa.pub` using your preferred text editor and copy the key. 1. For each member node in the cluster, do the following: 1. Connect to the instance. 1. Open `~/.ssh/authorized_keys` using your preferred text editor and add the public key that you copied earlier. 1. To test that the passwordless SSH is functioning as expected, connect to your leader node and run the following command. ``` $ ssh member_node_private_ip ``` You should connect to the member node without being prompted for a key or password. # Get started with EFA and NIXL for inference workloads on Amazon EC2 Get started with EFA and NIXL The NVIDIA Inference Xfer Library (NIXL) is a high-throughput, low-latency communication library designed specifically for disaggregated inference workloads. NIXL can be used together with EFA and Libfabric to support KV-cache transfer between prefill and decode nodes, and it enables efficient KV-cache movement between various storage layers. For more information, see the [NIXL](https://github.com/ai-dynamo/nixl) website. **Requirements** + Only Ubuntu 24.04 and Ubuntu 22.04 base AMIs are supported. + EFA supports only NIXL 1.0.0 and later. **Topics** ## Step 1: Prepare an EFA-enabled security group An EFA requires a security group that allows all inbound and outbound traffic to and from the security group itself. The following procedure creates a security group that allows all inbound and outbound traffic to and from itself, and that allows inbound SSH traffic from any IPv4 address for SSH connectivity. **Important** This security group is intended for testing purposes only. For your production environments, we recommend that you create an inbound SSH rule that allows traffic only from the IP address from which you are connecting, such as the IP address of your computer, or a range of IP addresses in your local network. For other scenarios, see [Security group rules for different use cases](security-group-rules-reference.md). **To create an EFA-enabled security group** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Security Groups** and then choose **Create security group**. 1. In the **Create security group** window, do the following: 1. For **Security group name**, enter a descriptive name for the security group, such as `EFA-enabled security group`. 1. (Optional) For **Description**, enter a brief description of the security group. 1. For **VPC**, select the VPC into which you intend to launch your EFA-enabled instances. 1. Choose **Create security group**. 1. Select the security group that you created, and on the **Details** tab, copy the **Security group ID**. 1. With the security group still selected, choose **Actions**, **Edit inbound rules**, and then do the following: 1. Choose **Add rule**. 1. For **Type**, choose **All traffic**. 1. For **Source type**, choose **Custom** and paste the security group ID that you copied into the field. 1. Choose **Add rule**. 1. For **Type**, choose **SSH**. 1. For **Source type**, choose **Anywhere-IPv4**. 1. Choose **Save rules**. 1. With the security group still selected, choose **Actions**, **Edit outbound rules**, and then do the following: 1. Choose **Add rule**. 1. For **Type**, choose **All traffic**. 1. For **Destination type**, choose **Custom** and paste the security group ID that you copied into the field. 1. Choose **Save rules**. ## Step 2: Launch a temporary instance Launch a temporary instance that you can use to install and configure the EFA software components. You use this instance to create an EFA-enabled AMI from which you can launch your EFA-enabled instances. **To launch a temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**, and then choose **Launch Instances** to open the new launch instance wizard. 1. (*Optional*) In the **Name and tags** section, provide a name for the instance, such as `EFA-instance`. The name is assigned to the instance as a resource tag (`Name=EFA-instance`). 1. In the **Application and OS Images** section, select an AMI for one of the supported operating systems. You can also select a supported DLAMI found on the [DLAMI Release Notes Page](https://docs.aws.amazon.com/dlami/latest/devguide/appendix-ami-release-notes). 1. In the **Instance type** section, select a supported instance type. 1. In the **Key pair** section, select the key pair to use for the instance. 1. In the **Network settings** section, choose **Edit**, and then do the following: 1. For **Subnet**, choose the subnet in which to launch the instance. If you do not select a subnet, you can't enable the instance for EFA. 1. For **Firewall (security groups)**, choose **Select existing security group**, and then select the security group that you created in the previous step. 1. Expand the **Advanced network configuration** section. For **Network interface 1**, select **Network card index = 0**, **Device index = 0**, and **Interface type = EFA with ENA**. (*Optional*) If you are using a multi-card instance type, such as `p4d.24xlarge` or `p5.48xlarge`, for each additional network interface required, choose **Add network interface**, for **Network card index** select the next unused index, and then select **Device index = 1** and **Interface type = EFA with ENA** or **EFA-only**. 1. In the **Storage** section, configure the volumes as needed. **Note** You must provision an additional 10 to 20 GiB of storage for the Nvidia CUDA Toolkit. If you do not provision enough storage, you will receive an `insufficient disk space` error when attempting to install the Nvidia drivers and CUDA toolkit. 1. In the **Summary** panel on the right, choose **Launch instance**. **Important** Skip Step 3 if your AMI already includes Nvidia GPU drivers, the CUDA toolkit, and cuDNN, or if you are using a non-GPU instance. ## Step 3: Install Nvidia GPU drivers, Nvidia CUDA toolkit, and cuDNN **To install the Nvidia GPU drivers, Nvidia CUDA toolkit, and cuDNN** 1. To ensure that all of your software packages are up to date, perform a quick software update on your instance. ``` $ sudo apt-get update && sudo apt-get upgrade -y ``` 1. Install the utilities that are needed to install the Nvidia GPU drivers and the Nvidia CUDA toolkit. ``` $ sudo apt-get install build-essential -y ``` 1. To use the Nvidia GPU driver, you must first disable the `nouveau` open source drivers. 1. Install the required utilities and the kernel headers package for the version of the kernel that you are currently running. ``` $ sudo apt-get install -y gcc make linux-headers-$(uname -r) ``` 1. Add `nouveau` to the `/etc/modprobe.d/blacklist.conf `deny list file. ``` $ cat << EOF | sudo tee --append /etc/modprobe.d/blacklist.conf blacklist vga16fb blacklist nouveau blacklist rivafb blacklist nvidiafb blacklist rivatv EOF ``` 1. Open `/etc/default/grub` using your preferred text editor and add the following. ``` GRUB_CMDLINE_LINUX="rdblacklist=nouveau" ``` 1. Rebuild the Grub configuration. ``` $ sudo update-grub ``` 1. Reboot the instance and reconnect to it. 1. Add the CUDA repository and install the Nvidia GPU drivers, NVIDIA CUDA toolkit, and cuDNN. ``` $ sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub \ && wget -O /tmp/deeplearning.deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb \ && sudo dpkg -i /tmp/deeplearning.deb \ && wget -O /tmp/cuda.pin https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin \ && sudo mv /tmp/cuda.pin /etc/apt/preferences.d/cuda-repository-pin-600 \ && sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub \ && sudo add-apt-repository 'deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /' \ && sudo apt update \ && sudo apt install nvidia-dkms-535 \ && sudo apt install -o Dpkg::Options::='--force-overwrite' cuda-drivers-535 cuda-toolkit-12-3 libcudnn8 libcudnn8-dev -y ``` 1. Reboot the instance and reconnect to it. 1. (`p4d.24xlarge` and `p5.48xlarge` only) Install the Nvidia Fabric Manager. 1. You must install the version of the Nvidia Fabric Manager that matches the version of the Nvidia kernel module that you installed in the previous step. Run the following command to determine the version of the Nvidia kernel module. ``` $ cat /proc/driver/nvidia/version | grep "Kernel Module" ``` The following is example output. ``` NVRM version: NVIDIA UNIX x86_64 Kernel Module 450.42.01 Tue Jun 15 21:26:37 UTC 2021 ``` In the example above, major version `450` of the kernel module was installed. This means that you need to install Nvidia Fabric Manager version `450`. 1. Install the Nvidia Fabric Manager. Run the following command and specify the major version identified in the previous step. ``` $ sudo apt install -o Dpkg::Options::='--force-overwrite' nvidia-fabricmanager-major_version_number ``` For example, if major version `450` of the kernel module was installed, use the following command to install the matching version of Nvidia Fabric Manager. ``` $ sudo apt install -o Dpkg::Options::='--force-overwrite' nvidia-fabricmanager-450 ``` 1. Start the service, and ensure that it starts automatically when the instance starts. Nvidia Fabric Manager is required for NV Switch Management. ``` $ sudo systemctl start nvidia-fabricmanager && sudo systemctl enable nvidia-fabricmanager ``` 1. Ensure that the CUDA paths are set each time that the instance starts. + For *bash* shells, add the following statements to `/home/username/.bashrc` and `/home/username/.bash_profile`. ``` export PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH ``` + For *tcsh* shells, add the following statements to `/home/username/.cshrc`. ``` setenv PATH=/usr/local/cuda/bin:$PATH setenv LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH ``` 1. To confirm that the Nvidia GPU drivers are functional, run the following command. ``` $ nvidia-smi -q | head ``` The command should return information about the Nvidia GPUs, Nvidia GPU drivers, and Nvidia CUDA toolkit. **Important** Skip Step 4 if your AMI already includes GDRCopy, or if you are using a non-GPU instance. ## Step 4: Install GDRCopy Install GDRCopy to improve the performance of Libfabric on GPU-based platforms. For more information about GDRCopy, see the [GDRCopy repository](https://github.com/NVIDIA/gdrcopy). **To install GDRCopy** 1. Install the required dependencies. ``` $ sudo apt -y install build-essential devscripts debhelper check libsubunit-dev fakeroot pkg-config dkms ``` 1. Download and extract the GDRCopy package. ``` $ wget https://github.com/NVIDIA/gdrcopy/archive/refs/tags/v2.4.tar.gz \ && tar xf v2.4.tar.gz \ && cd gdrcopy-2.4/packages ``` 1. Build the GDRCopy DEB packages. ``` $ CUDA=/usr/local/cuda ./build-deb-packages.sh ``` 1. Install the GDRCopy DEB packages. ``` $ sudo dpkg -i gdrdrv-dkms_2.4-1_amd64.*.deb \ && sudo dpkg -i libgdrapi_2.4-1_amd64.*.deb \ && sudo dpkg -i gdrcopy-tests_2.4-1_amd64.*.deb \ && sudo dpkg -i gdrcopy_2.4-1_amd64.*.deb ``` **Important** Skip Step 5 if your AMI already includes the latest EFA installer. ## Step 5: Install the EFA software Install the EFA-enabled kernel, EFA drivers, and Libfabric stack that is required to support EFA on your instance. **To install the EFA software** 1. Connect to the instance you launched. For more information, see [Connect to your Linux instance using SSH](connect-to-linux-instance.md). 1. Download the EFA software installation files. The software installation files are packaged into a compressed tarball (`.tar.gz`) file. To download the latest *stable* version, use the following command. ``` $ curl -O https://efa-installer.amazonaws.com/aws-efa-installer-1.47.0.tar.gz ``` 1. Extract the files from the compressed `.tar.gz` file, delete the tarball, and navigate into the extracted directory. ``` $ tar -xf aws-efa-installer-1.47.0.tar.gz && rm -rf aws-efa-installer-1.47.0.tar.gz && cd aws-efa-installer ``` 1. Run the EFA software installation script. ``` $ sudo ./efa_installer.sh -y ``` **Libfabric** is installed in the `/opt/amazon/efa` directory. 1. If the EFA installer prompts you to reboot the instance, do so and then reconnect to the instance. Otherwise, log out of the instance and then log back in to complete the installation. 1. Confirm that the EFA software components were successfully installed. ``` $ fi_info -p efa -t FI_EP_RDM ``` The command should return information about the Libfabric EFA interfaces. The following example shows the command output. + `p3dn.24xlarge` with single network interface ``` provider: efa fabric: EFA-fe80::94:3dff:fe89:1b70 domain: efa_0-rdm version: 2.0 type: FI_EP_RDM protocol: FI_PROTO_EFA ``` + `p4d.24xlarge` and `p5.48xlarge` with multiple network interfaces ``` provider: efa fabric: EFA-fe80::c6e:8fff:fef6:e7ff domain: efa_0-rdm version: 111.0 type: FI_EP_RDM protocol: FI_PROTO_EFA provider: efa fabric: EFA-fe80::c34:3eff:feb2:3c35 domain: efa_1-rdm version: 111.0 type: FI_EP_RDM protocol: FI_PROTO_EFA provider: efa fabric: EFA-fe80::c0f:7bff:fe68:a775 domain: efa_2-rdm version: 111.0 type: FI_EP_RDM protocol: FI_PROTO_EFA provider: efa fabric: EFA-fe80::ca7:b0ff:fea6:5e99 domain: efa_3-rdm version: 111.0 type: FI_EP_RDM protocol: FI_PROTO_EFA ``` ## Step 6: Install NIXL Install NIXL. For more information about NIXL, see the [NIXL repository](https://github.com/ai-dynamo/nixl). ------ #### [ Pre-built distributions ] **To install NIXL using PyPI** 1. Install the required dependencies. ``` $ sudo apt install pip ``` 1. Install NIXL. ``` $ pip install nixl ``` ------ #### [ Build from source ] **To build and install NIXL from source** 1. Install the required dependencies. ``` $ sudo apt install cmake pkg-config meson pybind11-dev libaio-dev nvidia-cuda-toolkit pip libhwloc-dev \ && pip install meson ninja pybind11 ``` 1. Navigate to your home directory. ``` $ cd $HOME ``` 1. Clone the official NIXL repository to the instance and navigate into the local cloned repository. ``` $ sudo git clone https://github.com/ai-dynamo/nixl.git && cd nixl ``` 1. Build and install NIXL and specify the path to the Libfabric installation directory. ``` $ sudo meson setup . nixl --prefix=/usr/local/nixl -Dlibfabric_path=/opt/amazon/efa $ cd nixl && sudo ninja && sudo ninja install ``` ------ ## Step 7: Install NIXL Benchmark and test your EFA and NIXL configuration Install the NIXL Benchmark and run a test to ensure that your temporary instance is properly configured for EFA and NIXL. The NIXL Benchmark enables you to confirm that NIXL is properly installed and that it is operating as expected. For more information, see the [nixlbench repository](https://github.com/ai-dynamo/nixl/tree/main/benchmark/nixlbench). NIXL Benchmark (nixlbench) requires ETCD for coordination between client and server. To use ETCD with NIXL requires ETCD Server and Client, and ETCD CPP API. ------ #### [ Build from Docker ] **To install and test NIXL Benchmark using Docker** 1. Clone the official NIXL repository to the instance and navigate to the nixlbench build directory. ``` $ git clone https://github.com/ai-dynamo/nixl.git $ cd nixl/benchmark/nixlbench/contrib ``` 1. Build the container. ``` $ ./build.sh ``` For more information about Docker build options, see the [nixlbench repository](https://github.com/ai-dynamo/nixl/tree/main/benchmark/nixlbench). 1. Install Docker. ``` $ sudo apt install docker.io -y ``` 1. Start the ETCD server for coordination. ``` $ docker run -d --name etcd-server \ -p 2379:2379 -p 2380:2380 \ quay.io/coreos/etcd:v3.5.18 \ /usr/local/bin/etcd \ --data-dir=/etcd-data \ --listen-client-urls=http://0.0.0.0:2379 \ --advertise-client-urls=http://0.0.0.0:2379 \ --listen-peer-urls=http://0.0.0.0:2380 \ --initial-advertise-peer-urls=http://0.0.0.0:2380 \ --initial-cluster=default=http://0.0.0.0:2380 ``` 1. Validate that the ETCD server is running. ``` $ curl -L http://localhost:2379/health ``` Expected output: ``` {"health":"true"} ``` 1. Open two terminals for the instance. On both terminals, run the following command to verify the installation. The command uses the ETCD server on the same instance, uses Libfabric as the backend, and operates using GPU memory. ``` $ docker run -it --gpus all --network host nixlbench:latest \ nixlbench --etcd_endpoints http://localhost:2379 \ --backend LIBFABRIC \ --initiator_seg_type VRAM \ --target_seg_type VRAM ``` **Note** Use the value `DRAM` instead of `VRAM` for non-GPU instances. ------ #### [ Build from source ] **Important** Follow this tab only if you chose **Build from source** in Step 6. **To install NIXL Benchmark** 1. Install the required system dependencies. ``` $ sudo apt install libgflags-dev ``` 1. Install ETCD Server and Client. ``` $ sudo apt install -y etcd-server etcd-client ``` 1. Install the ETCD CPP API. 1. Install the required dependencies for ETCD CPP API. ``` $ sudo apt install libboost-all-dev libssl-dev libgrpc-dev libgrpc++-dev libprotobuf-dev protobuf-compiler-grpc libcpprest-dev ``` 1. Clone and install ETCD CPP API. ``` $ cd $HOME $ git clone https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3.git $ cd etcd-cpp-apiv3 $ mkdir build && cd build $ cmake .. $ sudo make -j$(nproc) && sudo make install ``` 1. Build and install nixlbench. ``` $ sudo meson setup . $HOME/nixl/benchmark/nixlbench -Dnixl_path=/usr/local/nixl/ $ sudo ninja && sudo ninja install ``` **To test your EFA and NIXL configuration** 1. Start the ETCD server on the instance. ``` $ etcd --listen-client-urls "http://0.0.0.0:2379" \ --advertise-client-urls "http://localhost:2379" & ``` 1. Validate that the ETCD server is running. ``` $ curl -L http://localhost:2379/health ``` Expected output: ``` {"health":"true"} ``` 1. Open two terminals for the instance. On both terminals, complete the following steps to run nixlbench. 1. Navigate to the directory where nixlbench is installed. ``` $ cd /usr/local/nixlbench/bin/ ``` 1. Run the test and specify the backend, address of the ETCD server, and initiator segment type. The following command uses the ETCD server on the same instance, uses Libfabric as the backend, and operates using GPU memory. The environment variables configure the following: + `NIXL_LOG_LEVEL=INFO` — Enables detailed debugging output. You can also specify `WARN` to receive only error messages. + `LD_LIBRARY_PATH` — Sets the path for the NIXL library. For more information about the NIXL Benchmark arguments, see the [NIXLbench README](https://github.com/ai-dynamo/nixl/blob/main/benchmark/nixlbench/README.md) in the official nixlbench repository. ``` $ export NIXL_LOG_LEVEL=INFO $ export LD_LIBRARY_PATH=/usr/local/nixl/lib/$(gcc -dumpmachine):$LD_LIBRARY_PATH $ nixlbench --etcd-endpoints 'http://localhost:2379' \ --backend 'LIBFABRIC' \ --initiator_seg_type 'VRAM' \ --target_seg_type 'VRAM' ``` **Note** Use the value `DRAM` instead of `VRAM` for non-GPU instances. ------ ## Step 8: Install your machine learning applications Install the machine learning applications on the temporary instance. The installation procedure varies depending on the specific machine learning application. **Note** Refer to your machine learning application's documentation for installation instructions. ## Step 9: Create an EFA and NIXL-enabled AMI After you have installed the required software components, you create an AMI that you can reuse to launch your EFA-enabled instances. **To create an AMI from your temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the temporary instance that you created and choose **Actions**, **Image**, **Create image**. 1. For **Create image**, do the following: 1. For **Image name**, enter a descriptive name for the AMI. 1. (Optional) For **Image description**, enter a brief description of the purpose of the AMI. 1. Choose **Create image**. 1. In the navigation pane, choose **AMIs**. 1. Locate the AMI tht you created in the list. Wait for the status to change from `pending` to `available` before continuing to the next step. ## Step 10: Terminate the temporary instance At this point, you no longer need the temporary instance that you launched. You can terminate the instance to stop incurring charges for it. **To terminate the temporary instance** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the temporary instance that you created and then choose **Actions**, **Instance state**, **Terminate instance**. 1. When prompted for confirmation, choose **Terminate**. ## Step 11: Launch EFA and NIXL-enabled instances Launch your EFA and NIXL-enabled instances using the EFA-enabled AMI that you created in **Step 9**, and the EFA-enabled security group that you created in **Step 1**. **To launch EFA and NIXL-enabled instances** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**, and then choose **Launch Instances** to open the new launch instance wizard. 1. (*Optional*) In the **Name and tags** section, provide a name for the instance, such as `EFA-instance`. The name is assigned to the instance as a resource tag (`Name=EFA-instance`). 1. In the **Application and OS Images** section, choose **My AMIs**, and then select the AMI that you created in the previous step. 1. In the **Instance type** section, select a supported instance type. 1. In the **Key pair** section, select the key pair to use for the instance. 1. In the **Network settings** section, choose **Edit**, and then do the following: 1. For **Subnet**, choose the subnet in which to launch the instance. If you do not select a subnet, you can't enable the instance for EFA. 1. For **Firewall (security groups)**, choose **Select existing security group**, and then select the security group that you created in **Step 1**. 1. Expand the **Advanced network configuration** section. For **Network interface 1**, select **Network card index = 0**, **Device index = 0**, and **Interface type = EFA with ENA**. (*Optional*) If you are using a multi-card instance type, such as `p4d.24xlarge` or `p5.48xlarge`, for each additional network interface required, choose **Add network interface**, for **Network card index** select the next unused index, and then select **Device index = 1** and **Interface type = EFA with ENA** or **EFA-only**. 1. (*Optional*) In the **Storage** section, configure the volumes as needed. 1. In the **Summary** panel on the right, for **Number of instances**, enter the number of EFA-enabled instances that you want to launch, and then choose **Launch instance**. ## Step 12: Enable passwordless SSH To enable your applications to run across all of the instances in your cluster, you must enable passwordless SSH access from the leader node to the member nodes. The leader node is the instance from which you run your applications. The remaining instances in the cluster are the member nodes. **To enable passwordless SSH between the instances in the cluster** 1. Select one instance in the cluster as the leader node, and connect to it. 1. Disable `strictHostKeyChecking` and enable `ForwardAgent` on the leader node. Open `~/.ssh/config` using your preferred text editor and add the following. ``` Host * ForwardAgent yes Host * StrictHostKeyChecking no ``` 1. Generate an RSA key pair. ``` $ ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa ``` The key pair is created in the `$HOME/.ssh/` directory. 1. Change the permissions of the private key on the leader node. ``` $ chmod 600 ~/.ssh/id_rsa chmod 600 ~/.ssh/config ``` 1. Open `~/.ssh/id_rsa.pub` using your preferred text editor and copy the key. 1. For each member node in the cluster, do the following: 1. Connect to the instance. 1. Open `~/.ssh/authorized_keys` using your preferred text editor and add the public key that you copied earlier. 1. To test that the passwordless SSH is functioning as expected, connect to your leader node and run the following command. ``` $ ssh member_node_private_ip ``` You should connect to the member node without being prompted for a key or password. **Important** Follow Step 13 only if you followed Step 7. ## Step 13: Test your EFA and NIXL configuration across instances Run a test to ensure that your instances are properly configured for EFA and NIXL. ------ #### [ Build from Docker ] **To test your EFA and NIXL configuration across instances using Docker** 1. Select two hosts to run the nixlbench benchmark. Use the IP address of the first host as the ETCD server IP for metadata exchange. 1. Start the ETCD server on host 1. ``` $ docker run -d --name etcd-server \ -p 2379:2379 -p 2380:2380 \ quay.io/coreos/etcd:v3.5.18 \ /usr/local/bin/etcd \ --data-dir=/etcd-data \ --listen-client-urls=http://0.0.0.0:2379 \ --advertise-client-urls=http://0.0.0.0:2379 \ --listen-peer-urls=http://0.0.0.0:2380 \ --initial-advertise-peer-urls=http://0.0.0.0:2380 \ --initial-cluster=default=http://0.0.0.0:2380 ``` 1. Validate that the ETCD server is running. ``` $ curl -L http://localhost:2379/health ``` ``` {"health":"true"} ``` 1. Run the nixlbench benchmark on host 1. ``` $ docker run -it --gpus all --network host nixlbench:latest \ nixlbench --etcd_endpoints http://localhost:2379 \ --backend LIBFABRIC \ --initiator_seg_type VRAM ``` 1. Run the nixlbench benchmark on host 2. ``` $ docker run -it --gpus all --network host nixlbench:latest \ nixlbench --etcd_endpoints http://ETCD_SERVER_IP:2379 \ --backend LIBFABRIC \ --initiator_seg_type VRAM ``` ------ #### [ Build from source ] **Important** Follow this tab only if you chose **Build from source** in Step 6. **To test your EFA and NIXL configuration across instances** 1. Select two hosts to run the nixlbench benchmark. Use the IP address of the first host as the ETCD server IP for metadata exchange. 1. Launch the ETCD server on host 1. ``` $ etcd --listen-client-urls "http://0.0.0.0:2379" \ --advertise-client-urls "http://localhost:2379" & ``` 1. Validate that the ETCD server is running. ``` $ curl -L http://localhost:2379/health ``` ``` {"health":"true"} ``` 1. Run the nixlbench benchmark on host 1. ``` $ export NIXL_LOG_LEVEL=INFO $ export LD_LIBRARY_PATH=$HOME/nixl/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH $ nixlbench \ --etcd-endpoints http://localhost:2379 \ --backend LIBFABRIC \ --initiator_seg_type VRAM ``` 1. Run the nixlbench benchmark on host 2. ``` $ export NIXL_LOG_LEVEL=INFO $ export LD_LIBRARY_PATH=$HOME/nixl/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH $ nixlbench \ --etcd-endpoints http://ETCD_SERVER_IP:2379 \ --backend LIBFABRIC \ --initiator_seg_type VRAM ``` ------ ## Step 14: Test disaggregated inference serving over vLLM (*Optional*) After NIXL is installed, you can use NIXL through LLM inference and serving frameworks such as vLLM, SGLang, and TensorRT-LLM. **To serve your inference workload using vLLM** 1. Install vLLM. ``` $ pip install vllm ``` 1. Start the vLLM server with NIXL. The following sample commands create one prefill (producer) and one decode (consumer) instance for NIXL handshake connection, KV connector, KV role, and transport backend. For detailed examples and scripts, see the [NIXLConnector Usage Guide](https://github.com/vllm-project/vllm/blob/2d977a7a9ead3179fde9ed55d69393ef7b6cec47/docs/features/nixl_connector_usage.md). To use NIXL with EFA, set the environment variables based on your setup and use case. + Producer (Prefiller) configuration ``` $ vllm serve your-application \ --port 8200 \ --enforce-eager \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_buffer_device":"cuda","kv_connector_extra_config":{"backends":["LIBFABRIC"]}}' ``` + Consumer (Decoder) configuration ``` $ vllm serve your-application \ --port 8200 \ --enforce-eager \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_buffer_device":"cuda","kv_connector_extra_config":{"backends":["LIBFABRIC"]}}' ``` The preceding sample configuration sets the following: + `kv_role` to `kv_both`, which enables symmetric functionality where the connector can act as both producer and consumer. This provides flexibility for experimental setups and scenarios where the role distinction is not predetermined. + `kv_buffer_device` to `cuda`, which enables using GPU memory. + NIXL backend to `LIBFABRIC`, which enables NIXL traffic to go over EFA. # Maximize network bandwidth on Amazon EC2 instances with multiple network cards Maximize network bandwidth Many instances types that support EFA also have multiple network cards. For more information, see [Network cards](using-eni.md#network-cards). If you plan to use EFA with one of these instance types, we recommend the following basic configuration: + For the primary network interface (network card index `0`, device index `0`), create an ENA interface. You can't use an EFA-only network interface as the primary network interface. + If the network card index 0 supports EFA, create an EFA-only network interface for network card index `0`, device index `1`. + For each additional network interface, use the next unused network card index, device index `0`, for EFA-only network interface, and/or device index `1` for ENA network interface depending on your usecase, such as ENA bandwidth requirements or IP address space. For example use cases, see [EFA configuration for a P5 and P5e instances](#efa-for-p5). **Note** P5 instances require network interfaces to be configured in a specific manner to enable maximum network bandwidth. For more information, see [EFA configuration for a P5 and P5e instances](#efa-for-p5). The following examples show how to launch an instance based on these recommendations. ------ #### [ Instance launch ] **To specify EFAs during instance launch using the launch instance wizard** 1. In the **Network settings** section, choose **Edit**. 1. Expand **Advanced network configuration**. 1. For the primary network interface (network card index `0`, device index `0`), create an ENA interface. You can't use an EFA-only network interface as the primary network interface. 1. If the network card index 0 supports EFA, create an EFA-only network interface for network card index `0`, device index `1`. 1. For each additional network interface, use the next unused network card index, device index `0`, for EFA-only network interface, and/or device index `1` for ENA network interface depending on your usecase, such as ENA bandwidth requirements or IP address space. For example use cases, see [EFA configuration for a P5 and P5e instances](#efa-for-p5). **To specify EFAs during instance launch using the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command** For `--network-interfaces`, specify the required number of network interfaces. For the primary network interface, specify `NetworkCardIndex=0`, `DeviceIndex=0`, and `InterfaceType=interface`. If the network card index 0 supports EFA, specify `NetworkCardIndex=0`, `DeviceIndex=1`, and `InterfaceType=efa-only`. For any additional network interfaces, for `NetworkCardIndex` specify the next unused index, `DeviceIndex=0` for `InterfaceType=efa-only`, and/or `DeviceIndex=1` for `InterfaceType=interface`. The following example command snippet shows a request with 32 EFA devices and one ENA device. ``` $ aws ec2 run-instances \ --instance-type p5.48xlarge \ --count 1 \ --key-name key_pair_name \ --image-id ami-0abcdef1234567890 \ --network-interfaces "NetworkCardIndex=0,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=0,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=1,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=2,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=3,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=4,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=5,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=6,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=7,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=8,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=9,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=10,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=11,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=12,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=13,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=14,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=15,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=16,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=17,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=18,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=19,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=20,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=21,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=22,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=23,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=24,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=25,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=26,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=27,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=28,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=29,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=30,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=31,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" ... ``` ------ #### [ Launch templates ] **To add EFAs to a launch template using the Amazon EC2 console** 1. In the **Network settings** section, expand **Advanced network configuration**. 1. For the primary network interface (network card index `0`, device index `0`), create an ENA interface. You can't use an EFA-only network interface as the primary network interface. 1. If the network card index 0 supports EFA, create an EFA-only network interface for network card index `0`, device index `1`. 1. For each additional network interface, use the next unused network card index, device index `0`, for EFA-only network interface, and/or device index `1` for ENA network interface depending on your usecase, such as ENA bandwidth requirements or IP address space. For example use cases, see [EFA configuration for a P5 and P5e instances](#efa-for-p5). **To add EFAs to a launch template using the [create-launch-template](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-launch-template.html) command** For `NetworkInterfaces`, specify the required number of network interfaces. For the primary network interface, specify `NetworkCardIndex=0`, `DeviceIndex=0`, and `InterfaceType=interface`. If the network card index 0 supports EFA, specify `NetworkCardIndex=0`, `DeviceIndex=1`, and `InterfaceType=efa-only`. For any additional network interfaces, for `NetworkCardIndex` specify the next unused index, `DeviceIndex=0` for `InterfaceType=efa-only`, and/or `DeviceIndex=1` for `InterfaceType=interface`. The following snippet shows an example with 3 network interfaces out of the possible 32 network interfaces. ``` "NetworkInterfaces":[ { "NetworkCardIndex":0, "DeviceIndex":0, "InterfaceType": "interface", "AssociatePublicIpAddress":false, "Groups":[ "security_group_id" ], "DeleteOnTermination":true }, { "NetworkCardIndex": 0, "DeviceIndex": 1, "InterfaceType": "efa-only", "AssociatePublicIpAddress":false, "Groups":[ "security_group_id" ], "DeleteOnTermination":true }, { "NetworkCardIndex": 1, "DeviceIndex": 0, "InterfaceType": "efa-only", "AssociatePublicIpAddress":false, "Groups":[ "security_group_id" ], "DeleteOnTermination":true }, { "NetworkCardIndex": 2, "DeviceIndex": 0, "InterfaceType": "efa-only", "AssociatePublicIpAddress":false, "Groups":[ "security_group_id" ], "DeleteOnTermination":true }, { "NetworkCardIndex": 3, "DeviceIndex": 0, "InterfaceType": "efa-only", "AssociatePublicIpAddress":false, "Groups":[ "security_group_id" ], "DeleteOnTermination":true } ... ``` ------ ## EFA configuration for a P5 and P5e instances `p5.48xlarge` and `p5e.48xlarge` instances support 32 network cards and have a total network bandwidth capacity of 3,200 Gbps, of which up to 800 Gbps can be utilized for IP network traffic. Because EFA and IP network traffic share the same underlying resources, bandwidth used by one will reduce the bandwidth that is available to the other. This means that you can distribute the network bandwidth between EFA traffic and IP traffic in any combination, as long as the total bandwidth does not exceed 3,200 Gbps and IP bandwidth does not exceed 800 Gbps. For example, if you use 400 Gbps for IP bandwidth, you can achieve up to 2,800 Gbps of EFA bandwidth at the same time. **Use case 1: Save IP addresses and avoid potential Linux IP issues** This configuration provides up to 3200 Gbps of EFA networking bandwidth and up to 100 Gbps of IP networking bandwidth with one private IP address. This configuration also helps to avoid potential Linux IP issues, such as disallowed auto-assignment of public IP addresses and IP routing challenges (hostname to IP address mapping issues and source IP address mismatches), that can arise if an instance has multiple network interfaces. + For the primary network interface (network card index 0, device index 0), use an ENA interface. + For network card index 0, device index 1, create an EFA-only network interface. + For the remaining network interfaces (network card indexes 1-31, device index 0), use EFA-only network interfaces. **Use case 2: Maximum EFA and IP network bandwidth** This configuration provides up to 3200 Gbps of EFA networking bandwidth and up to 800 Gbps of IP networking bandwidth with 8 private IP address. You can't auto-assign public IP addresses with this configuration. However, you can attach an Elastic IP address to the primary network interface (network card index 0, device index 0) after launch for internet connectivity. + For the primary network interface (network card index 0, device index 0), use an ENA network interface. + For the remaining interfaces, do the following: + Specify EFA-only network interfaces on network card index 0 device index 1, and for network card indexes 1, 2, and 3, use device index 0. + Specify one ENA network interface and four EFA-only network interfaces **in each** of the following network card index subsets, and use device index 1 for ENA network interface and device index 0 for EFA-only network interfaces: + [4,5,6,7] + [8,9,10,11] + [12,13,14,15] + [16,17,18,19] + [20,21,22,23] + [24,25,26,27] + [28,29,30,31] The following example illustrates this configuration: ``` $ aws --region $REGION ec2 run-instances \ --instance-type p5.48xlarge \ --count 1 \ --key-name key_pair_name \ --image-id ami_id \ --network-interfaces "NetworkCardIndex=0,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=0,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=1,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=2,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=3,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=4,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=4,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=5,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=6,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=7,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=8,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=8,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=9,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=10,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=11,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=12,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=12,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=13,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=14,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=15,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=16,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=16,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=17,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=18,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=19,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=20,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=20,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=21,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=22,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=23,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=24,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=24,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=25,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=26,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=27,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=28,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=28,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=29,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=30,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=31,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" ... ``` ## EFA configuration for a P6-B200 instances P6-B200 instances have a total network bandwidth capacity of 3,200 Gbps, of which up to 1600 Gbps can be utilized for ENA. They have 8 GPUs and 8 network cards, where each network card supports up to 400 Gbps EFA bandwidth and 200 Gbps ENA bandwidth. Since EFA and ENA traffic share the same underlying resources, bandwidth used by one will reduce the bandwidth that is available to the other. **Use case 1: Save IP addresses** This configuration consumes at least one private IP address per instance and supports up to 3200 Gbps of EFA bandwidth and 200 Gbps of ENA bandwidth. + For the primary network interface (network card index 0, device index 0), use an ENA interface. + For network card index 0, device index 1, create an EFA-only network interface. + For the remaining 7 network cards (network card indexes 1-7, device index 0), use EFA-only network interfaces. **Use case 2: Maximum EFA and ENA bandwidth** This configuration consumes at least 8 private IP address per instance and supports up to 3200 Gbps of EFA bandwidth and 1600 Gbps of ENA bandwidth. + For the primary network interface (network card index 0, device index 0), use an ENA interface. + For network card index 0, device index 1, create an EFA-only network interface. + For the remaining 7 network cards (network card indexes 1-7), create an EFA-only network interface on device index 0 and an ENA network interface on device index 1. ## EFA configuration for a P6e-GB200 instances P6e-GB200 instances can be configured with up to 17 network cards. The following image shows the physical network interface card (NIC) layout for P6e-GB200 instances, along with the mapping of network card indexes (NCIs). ![\[Physical network interface card (NIC) and network card index (NCI) mapping for P6e-GB200 instances.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/p6e.png) The primary NCI (index 0) supports up to 100 Gbps of ENA bandwidth. NCIs with the following indexes support EFA-only network interfaces and 400 Gbps EFA bandwidth: [1, 3, 5, 7, 9, 11, 13, 15]. NCIs with the following indexes support up to 200 Gbps ENA or EFA bandwidth: [2, 4, 6, 8, 10, 12, 14, 16]. The NCIs in the following groups share an underlying physical NIC on the host: + [1 and 2] + [3 and 4] + [5 and 6] + [7 and 8] + [9 and 10] + [11 and 12] + [13 and 14] + [15 and 16] Each physical NIC supports up 400 Gbps of bandwidth. Because the NCIs in these groups share the same underlying physical NIC, bandwidth used by one will reduce the bandwidth that is available to the other. For example, if NCI 2 uses 200 Gbps of ENA bandwidth, NCI 1 can use up to 200 Gbps of EFA bandwidth at the same time. Each underlying GPU on the host can send traffic directly over the following pairs of NCIs: + [1 and 3] + [5 and 7] + [9 and 11] + [13 and 15] Each GPU supports up to 400 Gbps of EFA bandwidth. Because the network cards in these groups share the same GPU, bandwidth used by one will reduce the bandwidth that is available to the other. For example, if NCI 1 uses 200 Gbps of EFA bandwidth, NCI 3 can use up to 200 Gbps of EFA bandwidth at the same time. Therefore, to achieve maximum EFA performance, we recommend that you do **one of the following** to achieve a total of 1,600 Gbps EFA bandwidth: + Add an EFA-only network interface to only one NCI in each group to achieve 400 Gbps per network interface (*4 EFA network interfaces x 400 Gbps*). + Add an EFA-only network interface to each NCI in each group to achieve 200 Gbps per network interface (*8 EFA network interfaces x 200 Gbps*). For example, the following configuration provides up to 1,600 Gbps of EFA bandwidth using a single EFA-only network interface in each NCI group, and up to 100 Gbps of ENA networking bandwidth using only the primary NCI (index 0). + For the primary NCI (network card index 0, device index 0), use an ENA network interface. + Add EFA-only network interfaces to the following: + NCI 1, device index 0 + NCI 5, device index 0 + NCI 9, device index 0 + NCI 13, device index 0 ## EFA configuration for a P6-B300 instances P6-B300 instances have a total network bandwidth capacity of up to 6400 Gbps for EFA traffic, and up to 3870 Gbps for ENA traffic. They have 8 GPUs and 17 network cards, where the primary network card supports only an ENA network interface with up to 350 Gbps of bandwidth. The secondary network cards support up to 400 Gbps EFA and up to 220 Gbps of ENA bandwidth. Since EFA and ENA traffic share the same underlying resources, bandwidth used by one will reduce the bandwidth that is available to the other. ![\[Physical network interface card (NIC) and network card index (NCI) mapping for P6-B300 instances.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/p6-b300.png) **Use case 1: Save IP addresses** This configuration consumes at least one private IP address per instance and supports up to 6400 Gbps of EFA bandwidth and up to 350 Gbps of ENA bandwidth. + For the primary network interface (network card index 0, device index 0), use an ENA network interface. + For the remaining network cards (network card indexes 1-16, device index 0), use EFA-only network interfaces. ``` --network-interfaces \ "NetworkCardIndex=0,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=1,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=2,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=3,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=4,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=5,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=6,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=7,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=8,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=9,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=10,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=11,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=12,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=13,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=14,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=15,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=16,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" ``` **Use case 2: Maximum EFA and ENA bandwidth** This configuration consumes at least 17 private IP address per instance and supports up to 6400 Gbps of EFA bandwidth and up to 3870 Gbps of ENA bandwidth. + For the primary network interface (network card index 0, device index 0) use an ENA network interface. + For the remaining network cards, create an EFA-only interface (network card indexes 1-16 device index 0) and an ENA interface network card indexes 1-16 device index 1). ``` --network-interfaces \ "NetworkCardIndex=0,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=1,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=2,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=3,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=4,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=5,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=6,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=7,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=8,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=9,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=10,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=11,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=12,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=13,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=14,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=15,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=16,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa-only" \ "NetworkCardIndex=1,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=2,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=3,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=4,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=5,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=6,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=7,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=8,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=9,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=10,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=11,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=12,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=13,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=14,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=15,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" \ "NetworkCardIndex=16,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=interface" ``` # Create and attach an Elastic Fabric Adapter to an Amazon EC2 instance Create and attach an EFA You can create an EFA and attach it to an Amazon EC2 instance much like any other elastic network interface in Amazon EC2. However, unlike elastic network interfaces, EFAs can't be attached to or detached from an instance in a `running` state. **Considerations** + You can change the security group that is associated with an EFA. To enable OS-bypass functionality, the EFA must be a member of a security group that allows all inbound and outbound traffic to and from the security group itself. For more information, see [Step 1: Prepare an EFA-enabled security group](efa-start.md#efa-start-security). You change the security group that is associated with an EFA in the same way that you change the security group that is associated with an elastic network interface. For more information, see [Modify network interface attributes](modify-network-interface-attributes.md). + You assign an Elastic IP (IPv4) and IPv6 address to an EFA (EFA with ENA) network interface in the same way that you assign an IP address to an elastic network interface. For more information, see [Managing IP addresses](managing-network-interface-ip-addresses.md). You can't assign an IP address to an EFA-only network interface. **Topics** + [ ## Create an EFA ](#efa-create) + [ ## Attach an EFA to a stopped instance ](#efa-attach) + [ ## Attach an EFA when launching an instance ](#efa-launch) + [ ## Add an EFA to a launch template ](#efa-launch-template) ## Create an EFA You can create an EFA in a subnet in a VPC. You can't move the EFA to another subnet after it's created, and you can only attach it to stopped instances in the same Availability Zone. ------ #### [ Console ] **To create an EFA (EFA with ENA or ENA-only) network interface** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Network Interfaces** and then choose **Create network interface**. 1. For **Description**, enter a descriptive name for the EFA. 1. For **Subnet**, select the subnet in which to create the EFA. 1. **Interface type**, choose one of the following options: + **EFA with ENA** — To create a network interface that supports both ENA and EFA devices. + **EFA-only** — To create a network interface with an EFA device only. 1. (For EFA with ENA only) Configure the IP address and prefix assignment for the network interface. The type of IP addresses and prefixes you can assign depend on the selected subnet. For IPv4-only subnets, you can assign IPv4 IP addresses and prefixes only. For IPv6-only subnets, you can assign IPv6 IP addresses and prefixes only. For dual-stack subnets, you can assign both IPv4 and IPv6 IP addresses and prefixes. **Note** You can't assign IP addresses to an EFA-only network interface. 1. For **Private IPv4 address** and/or **IPv6 address**, choose **Auto-assign** to have Amazon EC2 automatically assign an IP address from the selected subnet, or choose **Custom** to manually specify the IP address to assign. 1. If you assign an IPv6 address, you can optionally enable **Assign primary IPv6 IP**. Doing this assigns a primary IPv6 global unicast address (GUA) to the network interface. Assigning a primary IPv6 address enables you to avoid disrupting traffic to instances or ENIs. For more information, see [IPv6 addresses](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html#vpc-ipv6-addresses). 1. For **IPv4 prefix delegation** and/or **IPv6 prefix delegation**, choose **Auto-assign** to have Amazon EC2 automatically assign a prefix from the subnet's CIDR block, or choose **Custom** to manually specify a prefix from the subnet's CIDR block. If you specify a prefix, AWS verifies that it is not already assigned to another resource. For more information, see [Prefix delegation for Amazon EC2 network interfaces](ec2-prefix-eni.md) 1. (Optional) Configure the **Idle connection tracking timeout** settings. For more information, see [Idle connection tracking timeout](security-group-connection-tracking.md#connection-tracking-timeouts) + **TCP established timeout** — The timeout period, in seconds, for idle TCP connections in an established state. Min: 60 seconds. Max: 432000 seconds (5 days). Default: 432000 seconds. Recommended: Less than 432000 seconds. + **UDP timeout** — The timeout period, in seconds, for idle UDP flows that have seen traffic only in a single direction or a single request-response transaction. Min: 30 seconds. Max: 60 seconds. Default: 30 seconds. + **UDP stream timeout** — The timeout period, in seconds, for idle UDP flows classified as streams that have seen more than one request-response transaction. Min: 60 seconds. Max: 180 seconds (3 minutes). Default: 180 seconds. 1. For **Security groups**, select one or more security groups. 1. Choose **Create network interface**. ------ #### [ AWS CLI ] **To create an EFA** Use the [create-network-interface](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-network-interface.html) command. For `--interface-type`, specify `efa` for an EFA network interface or `efa-only` for an EFA-only network interface. ``` aws ec2 create-network-interface \ --subnet-id subnet-0abcdef1234567890 \ --interface-type efa \ --description "my efa" ``` ------ #### [ PowerShell ] **To create an EFA** Use the [New-EC2NetworkInterface](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2NetworkInterface.html) cmdlet. For `-InterfaceType`, specify `efa` for an EFA network interface or `efa-only` for an EFA-only network interface ``` New-EC2NetworkInterface ` -SubnetId subnet-0abcdef1234567890 ` -InterfaceType efa ` -Description "my efa" ``` ------ ## Attach an EFA to a stopped instance You can attach an EFA to any supported instance that is in the `stopped` state. You cannot attach an EFA to an instance that is in the `running` state. For more information about the supported instance types, see [Supported instance types](efa.md#efa-instance-types). You attach an EFA to an instance in the same way that you attach a network interface to an instance. For more information, see [Attach a network interface](network-interface-attachments.md#attach_eni). ## Attach an EFA when launching an instance ------ #### [ AWS CLI ] **To attach an existing EFA when launching an instance** Use the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command with the `--network-interfaces` option. For the primary network interface, specify an EFA network interface and `NetworkCardIndex=0`, `DeviceIndex=0`. To attach multiple EFA network interfaces, see [Maximize network bandwidth](efa-acc-inst-types.md). ``` --network-interfaces "NetworkCardIndex=0, \ DeviceIndex=0, \ NetworkInterfaceId=eni-1234567890abcdef0, \ Groups=sg-1234567890abcdef0, \ SubnetId=subnet-0abcdef1234567890" ``` **To attach a new EFA when launching an instance** Use the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command with the `--network-interfaces` option. For the primary network interface, use `NetworkCardIndex=0`, `DeviceIndex=0`, and `InterfaceType=efa`. If you are attaching multiple EFA network interfaces, see [Maximize network bandwidth](efa-acc-inst-types.md). ``` --network-interfaces "NetworkCardIndex=0, \ DeviceIndex=0, \ InterfaceType=efa, \ Groups=sg-1234567890abcdef0, \ SubnetId=subnet-0abcdef1234567890" ``` ------ #### [ PowerShell ] **To attach an existing EFA when launching an instance** Use the [New-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2Instance.html) cmdlet with the `-NetworkInterfaces` parameter. ``` -NetworkInterface $networkInterface ``` Define the network interface as follows. ``` $networkInterface = New-Object Amazon.EC2.Model.InstanceNetworkInterfaceSpecification $networkInterface.DeviceIndex = 0 $networkInterface.NetworkInterfaceId = "eni-1234567890abcdef0" $networkInterface.Groups = @("sg-1234567890abcdef0") $networkInterface.SubnetId = "subnet-0abcdef1234567890" ``` **To attach a new EFA when launching an instance** Use the [New-EC2Instance](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2Instance.html) cmdlet with the `-NetworkInterfaces` parameter. ``` -NetworkInterface $networkInterface ``` Define the network interface as follows. ``` $networkInterface = New-Object Amazon.EC2.Model.InstanceNetworkInterfaceSpecification $networkInterface.DeviceIndex = 0 $networkInterface.InterfaceType = "efa" $networkInterface.Groups = @("sg-1234567890abcdef0") $networkInterface.SubnetId = "subnet-0abcdef1234567890" ``` ------ ## Add an EFA to a launch template You can create a launch template that contains the configuration information needed to launch EFA-enabled instances. You can specify both EFA and EFA-only network interfaces in the launch template. To create an EFA-enabled launch template, create a new launch template and specify a supported instance type, your EFA-enabled AMI, and an EFA-enabled security group. For `NetworkInterfaces`, specify the EFA network interfaces to attach. For the primary network interface, use `NetworkCardIndex=0`, `DeviceIndex=0`, and `InterfaceType=efa`. If you are attaching multiple EFA network interfaces, see [Maximize network bandwidth on Amazon EC2 instances with multiple network cards](efa-acc-inst-types.md). You can leverage launch templates to launch EFA-enabled instances with other AWS services, such as [AWS Batch](https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html) or [AWS ParallelCluster](https://docs.aws.amazon.com/parallelcluster/latest/ug/what-is-aws-parallelcluster.html). For more information about creating launch templates, see [Create an Amazon EC2 launch template](create-launch-template.md). # Detach and delete an EFA from an Amazon EC2 instance Detach and delete an EFA You can detach an EFA from an Amazon EC2 instance and delete it in the same way as any other elastic network interface in Amazon EC2. ## Detach an EFA To detach an EFA from an instance, you must first stop the instance. You cannot detach an EFA from an instance that is in the running state. You detach an EFA from an instance in the same way that you detach an elastic network interface from an instance. For more information, see [Detach a network interface](network-interface-attachments.md#detach_eni). ## Delete an EFA To delete an EFA, you must first detach it from the instance. You cannot delete an EFA while it is attached to an instance. You delete EFAs in the same way that you delete elastic network interfaces. For more information, see [Delete a network interface](delete_eni.md). # Monitor an Elastic Fabric Adapter on Amazon EC2 Monitor an EFA You can use the following features to monitor the performance of your Elastic Fabric Adapters. **Topics** + [ ## EFA driver metrics for an Amazon EC2 instance ](#efa-driver-metrics) + [ ## Amazon VPC flow logs ](#efa-flowlog) + [ ## Amazon CloudWatch ](#efa-cloudwatch) ## EFA driver metrics for an Amazon EC2 instance The Elastic Fabric Adapter (EFA) driver publishes multiple metrics from the instances that have EFA interfaces attached, in real time. You can use these metrics to troubleshoot application performance and networking issues, choose the right cluster size for a workload, plan scaling activities proactively, and benchmark applications to determine whether they maximize the EFA performance available on an instance. **Topics** + [ ### Available EFA driver metrics ](#available-efa-metrics) + [ ### Retrieve EFA driver metrics for your instance ](#view-efa-driver-metrics) ### Available EFA driver metrics The EFA driver publishes the following metrics to the instance in real time. They provide the cumulative number of errors, connection events, and packets or bytes sent, received, retransmitted, or dropped by the attached EFA devices since instance launch or the last driver reset. | Metric | Description | Supported instance types | | --- | --- | --- | | tx\$1bytes | The number of bytes transmitted. Unit: bytes | All instance types that support EFA | | rx\$1bytes | The number of bytes received. Unit: bytes | All instance types that support EFA | | tx\$1pkts | The number of packets transmitted. Unit: count | All instance types that support EFA | | rx\$1pkts | The number of packets received. Unit: count | All instance types that support EFA | | rx\$1drops | The number of packets that were received and then dropped. Unit: count | All instance types that support EFA | | send\$1bytes | The number of bytes sent using send operations. Unit: bytes | All instance types that support EFA | | recv\$1bytes | The number of bytes received by send operations. Unit: bytes | All instance types that support EFA | | send\$1wrs | The number of packets sent using send operations. Unit: count | All instance types that support EFA | | recv\$1wrs | The number of packets received by send operations. Unit: count | All instance types that support EFA | | rdma\$1write\$1wrs | The number of completed rdma write operations. Unit: count | All instance types that support EFA | | rdma\$1read\$1wrs | The number of completed rdma read operations. Unit: count | All instance types that support EFA | | rdma\$1write\$1bytes | The number of bytes written to it by other instances using rdma write operations. Unit: bytes | All instance types that support EFA | | rdma\$1read\$1bytes | The number of bytes received using rdma read operations. Unit: bytes | All instance types that support EFA | | rdma\$1write\$1wr\$1err | The number of rdma write operations that had local or remote errors. Unit: count | All instance types that support EFA | | rdma\$1read\$1wr\$1err | The number of rdma read operations that had local or remote errors. Unit: count | All instance types that support EFA | | rdma\$1read\$1resp\$1bytes | The number of bytes sent in response to rdma read operations. Unit: bytes | All instance types that support EFA | | rdma\$1write\$1recv\$1bytes | The number of bytes received by rdma write operations. Unit: bytes | All instance types that support EFA | | retrans\$1bytes | The number of EFA SRD bytes retransmitted. Unit: count | Nitro v4 and later instance types that support EFA | | retrans\$1pkts | The number of EFA SRD packets retransmitted. Unit: bytes | Nitro v4 and later instance types that support EFA | | retrans\$1timeout\$1events | The number of times EFA SRD traffic timed out and resulted in a network path change. Unit: count | Nitro v4 and later instance types that support EFA | | impaired\$1remote\$1conn\$1events | The number of times EFA SRD connections entered an impaired state, resulting in a reduced throughput rate limit. Unit: count | Nitro v4 and later instance types that support EFA | | unresponsive\$1remote\$1events | The number of times an EFA SRD remote connection was unresponsive. Unit: count | Nitro v4 and later instance types that support EFA | For more information about the instance types that support EFA, see [Supported instance types](efa.md#efa-instance-types). ### Retrieve EFA driver metrics for your instance You can use the [rdma-tool](https://man7.org/linux/man-pages/man8/rdma.8.html) command line tool to retrieve the metrics for all EFA interfaces attached to an instance as follows: ``` $ rdma -p statistic show link rdmap0s31/1 tx_bytes 0 tx_pkts 0 rx_bytes 0 rx_pkts 0 rx_drops 0 send_bytes 0 send_wrs 0 recv_bytes 0 recv_wrs 0 rdma_read_wrs 0 rdma_read_bytes 0 rdma_read_wr_err 0 rdma_read_resp_bytes 0 rdma_write_wrs 0 rdma_write_bytes 0 rdma_write_wr_err 0 retrans_bytes 0 retrans_pkts 0 retrans_timeout_events 0 unresponsive_remote_events 0 impaired_remote_conn_events 0 ``` Alternatively, you can retrieve the metrics for each EFA interface attached to an instance from the sys files using the following command. ``` $ more /sys/class/infiniband/device_number/ports/port_number/hw_counters/* | cat ``` For example ``` $ more /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/* | cat :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/lifespan :::::::::::::: 12 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rdma_read_bytes :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rdma_read_resp_bytes :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rdma_read_wr_err :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rdma_read_wrs :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rdma_write_bytes :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rdma_write_recv_bytes :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rdma_write_wr_err :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rdma_write_wrs :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/recv_bytes :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/recv_wrs :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rx_bytes :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rx_drops :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/rx_pkts :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/send_bytes :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/send_wrs :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/tx_bytes :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/tx_pkts :::::::::::::: 0 :::::::::::::: /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/retrans_bytes :::::::::::::: 0 /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/retrans_pkts :::::::::::::: 0 /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/retrans_timeout_events :::::::::::::: 0 /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/unresponsive_remote_events :::::::::::::: 0 /sys/class/infiniband/rdmap0s31/ports/1/hw_counters/impaired_remote_conn_events :::::::::::::: 0 ``` ## Amazon VPC flow logs You can create an Amazon VPC Flow Log to capture information about the traffic going to and from an EFA. Flow log data can be published to Amazon CloudWatch Logs and Amazon S3. After you create a flow log, you can retrieve and view its data in the chosen destination. For more information, see [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) in the *Amazon VPC User Guide*. You create a flow log for an EFA in the same way that you create a flow log for an elastic network interface. For more information, see [Create a flow log](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-flow-logs.html#create-flow-log) in the *Amazon VPC User Guide*. In the flow log entries, EFA traffic is identified by the `srcAddress` and `destAddress`, which are both formatted as MAC addresses, as shown in the following example. ``` version accountId eniId srcAddress destAddress sourcePort destPort protocol packets bytes start end action log-status 2 3794735123 eni-10000001 01:23:45:67:89:ab 05:23:45:67:89:ab - - - 9 5689 1521232534 1524512343 ACCEPT OK ``` ## Amazon CloudWatch If you are using EFA in an Amazon EKS cluster, you can monitor your EFAs using CloudWatch Container Insights. Amazon CloudWatch Container Insights supports all of the [EFA driver metrics](#efa-driver-metrics), except: `retrans_bytes`, `retrans_pkts`, `retrans_timeout_events`, `unresponsive_remote_events`, and `impaired_remote_conn_events`. For more information, see [ Amazon EKS and Kubernetes Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-enhanced-EKS.html#Container-Insights-metrics-EFA) in the *Amazon CloudWatch User Guide*. # Verify the EFA installer using a checksum Verify the EFA installer You can optionally verify the EFA tarball (`.tar.gz` file) using an MD5 or SHA256 checksum. We recommend that you do this to verify the identity of the software publisher and to check that the application has not been altered or corrupted since it was published. **To verify the tarball** Use the **md5sum** utility for the MD5 checksum, or the **sha256sum** utility for the SHA256 checksum, and specify the tarball filename. You must run the command from the directory in which you saved the tarball file. + MD5 ``` $ md5sum tarball_filename.tar.gz ``` + SHA256 ``` $ sha256sum tarball_filename.tar.gz ``` The commands should return a checksum value in the following format. ``` checksum_value tarball_filename.tar.gz ``` Compare the checksum value returned by the command with the checksum value provided in the table below. If the checksums match, then it is safe to run the installation script. If the checksums do not match, do not run the installation script, and contact Support. For example, the following command verifies the EFA 1.9.4 tarball using the SHA256 checksum. ``` $ sha256sum aws-efa-installer-1.9.4.tar.gz ``` Example output: ``` 1009b5182693490d908ef0ed2c1dd4f813cc310a5d2062ce9619c4c12b5a7f14 aws-efa-installer-1.9.4.tar.gz ``` The following table lists the checksums for recent versions of EFA. | Version | Checksums | | --- | --- | | EFA 1.47.0 | **MD5: **`c81d4caf24dabc04a6e4818590620f5f` **SHA256: **`2df4201e046833c7dc8160907bee7f52b76ff80ed147376a2d0ed8a0dd66b2db` | | EFA 1.46.0 | **MD5: **a88bbd9b71624d7ca401b54bc2fc0c19`` **SHA256: **`8302bd7849afb95c903a875d7dcb6f85b3d7629e9a8b67d020031cfc6f4d0ee1` | | EFA 1.45.1 | **MD5: **91c3c87e16bbcaca1513252c38b771bb`` **SHA256: **`9aeb20c645135b6039cc08986d8f14e63280f7839e882a74df5e83627ffeaa17` | | EFA 1.45.0 | **MD5: **800aeddfa9d9c5f139a7b8f7c4fec627`` **SHA256: **`25ba26a0877fe3317390dc126aad2f23e27fc461cf0b940004f032cb342fa539` | | EFA 1.44.0 | **MD5: **d024f6bebe080db42745103b84ca7c43`` **SHA256: **`f129a5b44a49d593d247e55a59eb9bcb57121566e1c2e42b832a4e794fa83d8a` | | EFA 1.43.3 | **MD5: **`4dbc6eeecc516760253c10cbedb6319d` **SHA256: **`6c470ebce254c7165347b5048895ac2996c88567271642297f4c597738300652` | | EFA 1.43.2 | **MD5: **`7287b25a07c9747c0d4001e8fc5f59b2` **SHA256: **`de15c5bdbc83b952afbde876110830c604ad0796680e5157c05f7c1979a41069` | | EFA 1.43.1 | **MD5: **`7cfafc8debaea51dd4966fa0b2bba673` **SHA256: **54211eda0c193138ee8ed09b5fb41c41fc76fe0a77935fa4ec8d989466342740`` | | EFA 1.43.0 | **MD5: **`f2b3dd7dc8670b541f7c23fd58e5e503` **SHA256: **`786df3458c499237be33bb8e50ffd4da7c18c20e254380ffc80fb90833d8cc73` | | EFA 1.42.0 | **MD5: **`94b2b1db09da1dde08ec049db1f24370` **SHA256: **`4114fe612905ee05083ae5cb391a00a012510f3abfecc642d86c9a5ae4be9008` | | EFA 1.41.0 | **MD5: **`086181c3ee3f8da512fc6e1c795e8936` **SHA256: **`3506354cdfbe31ff552fe75f5d0d9bb7efd29cf79bd99457347d29c751c38f9f` | | EFA 1.40.0 | **MD5: **`f3ec6f73fbeaccba082327507581157c` **SHA256: **`30491b0fe7c3470d4439594538855c981b05fa69862d74f8c05eb9b97912368a` | | EFA 1.39.0 | **MD5: **`c223d5954a85a7fbcd248c942b866e43` **SHA256: **`2cbc028c03064633bb990782b47c36156637769e2f48704417a9c700a7a32101` | | EFA 1.38.1 | **MD5: **`f112569e828ab65187777f794bab542c` **SHA256: **`83923374afd388b1cfcf4b3a21a2b1ba7cf46a01a587f7b519b8386cb95e4f81` | | EFA 1.38.0 | **MD5: **`43a2a446b33a2506f40853d55059f1ea` **SHA256: **`4f436954f35ad53754b4d005fd8d0be63de3b4184de41a695b504bdce0fecb22` | | EFA 1.37.0 | **MD5: **`6328070192bae920eca45797ad4c1db1` **SHA256: **`2584fc3c8bb99f29b3285e275747ff09d67c18e162c2a652e36c976b72154bfb` | | EFA 1.36.0 | **MD5: **`1bec83180fbffb23452ab6469ca21dfa` **SHA256: **`de183f333cfb58aeb7908a67bf9106985ba3ccb7f8638b851d2a0d8dbfacaec4` | | EFA 1.35.0 | **MD5: **`252f03c978dca5f8e8d9f34e488b256e` **SHA256: **`432b6ad4368ba0cd8b902729d14a908a97be7a3dcc5239422ea994a47f35a5e1` | | EFA 1.34.0 | **MD5: **`5cd4b28d27a31677c16139b54c9acb45` **SHA256: **`bd68839e741b0afd3ec2e37d50603803cfa7a279c120f0a736cc57c2ff2d7fdc` | | EFA 1.33.0 | **MD5: **`e2f61fccbcaa11e2ccfddd3660522276` **SHA256: **`0372877b87c6a7337bb7791d255e1053b907d030489fb2c3732ba70069185fce` | | EFA 1.32.0 | **MD5: **`db8d65cc028d8d08b5a9f2d88881c1b1` **SHA256: **`5f7233760be57f6fee6de8c09acbfbf59238de848e06048dc54d156ef578fc66` | | EFA 1.31.0 | **MD5: **`856352f12bef2ccbadcd75e35aa52aaf` **SHA256: **`943325bd37902a4300ac9e5715163537d56ecb4e7b87b37827c3e547aa1897bf` | | EFA 1.30.0 | **MD5: **`31f48e1a47fe93ede8ebd273fb747358` **SHA256: **`876ab9403e07a0c3c91a1a34685a52eced890ae052df94857f6081c5f6c78a0a` | | EFA 1.29.1 | **MD5: **`e1872ca815d752c1d7c2b5c175e52a16` **SHA256: **`178b263b8c25845b63dc93b25bcdff5870df5204ec509af26f43e8d283488744` | | EFA 1.29.0 | **MD5: **`39d06a002154d94cd982ed348133f385` **SHA256: **`836655f87015547e733e7d9f7c760e4e24697f8bbc261bb5f3560abd4206bc36` | | EFA 1.28.0 | **MD5: **`9dc13b7446665822605e66febe074035` **SHA256: **`2e625d2d6d3e073b5178e8e861891273d896b66d03cb1a32244fd56789f1c435` | | EFA 1.27.0 | **MD5: **`98bfb515ea3e8d93f554020f3837fa15` **SHA256: **`1d49a97b0bf8d964d91652a79ac851f2550e33a5bf9d0cf86ec9357ff6579aa3` | | EFA 1.26.1 | **MD5: **`884e74671fdef4725501f7cd2d451d0c` **SHA256: **`c616994c924f54ebfabfab32b7fe8ac56947fae00a0ff453d975e298d174fc96` | | EFA 1.26.0 | **MD5: **`f8839f12ff2e3b9ba09ae8a82b30e663` **SHA256: **`bc1abc1f76e97d204d3755d2a9ca307fc423e51c63141f798c2f15be3715aa11` | | EFA 1.25.1 | **MD5: **`6d876b894547847a45bb8854d4431f18` **SHA256: **`d2abc553d22b89a4ce92882052c1fa6de450d3a801fe005da718b7d4b9602b06` | | EFA 1.25.0 | **MD5: **`1993836ca749596051da04694ea0d00c` **SHA256: **`98b7b26ce031a2d6a93de2297cc71b03af647194866369ca53b60d82d45ad342` | | EFA 1.24.1 | **MD5: **`211b249f39d53086f3cb0c07665f4e6f` **SHA256: **`120cfeec233af0955623ac7133b674143329f9561a9a8193e473060f596aec62` | | EFA 1.24.0 | **MD5: **`7afe0187951e2dd2c9cc4b572e62f924` **SHA256: **`878623f819a0d9099d76ecd41cf4f569d4c3aac0c9bb7ba9536347c50b6bf88e` | | EFA 1.23.1 | **MD5: **`22491e114b6ee7160a8290145dca0c28` **SHA256: **`5ca848d8e0ff4d1571cd443c36f8d27c8cdf2a0c97e9068ebf000c303fc40797` | | EFA 1.23.0 | **MD5: **`38a6d7c1861f5038dba4e441ca7683ca` **SHA256: **`555d497a60f22e3857fdeb3dfc53aa86d05926023c68c916d15d2dc3df6525bd` | | EFA 1.22.1 | **MD5: **`600c0ad7cdbc06e8e846cb763f92901b` **SHA256: **`f90f3d5f59c031b9a964466b5401e86fd0429272408f6c207c3f9048254e9665` | | EFA 1.22.0 | **MD5: **`8f100c93dc8ab519c2aeb5dab89e98f8` **SHA256: **`f329e7d54a86a03ea51da6ea9a5b68fb354fbae4a57a02f9592e21fce431dc3a` | | EFA 1.21.0 | **MD5: **`959ccc3a4347461909ec02ed3ba7c372` **SHA256: **`c64e6ca34ccfc3ebe8e82d08899ae8442b3ef552541cf5429c43d11a04333050` | | EFA 1.20.0 | **MD5: **`7ebfbb8e85f1b94709df4ab3db47913b` **SHA256: **`aeefd2681ffd5c4c631d1502867db5b831621d6eb85b61fe3ec80df983d1dcf0` | | EFA 1.19.0 | **MD5: **`2fd45324953347ec5518da7e3fefa0ec` **SHA256: **`99b77821b9e72c8dea015cc92c96193e8db307deee05b91a58094cc331f16709` | | EFA 1.18.0 | **MD5: **`fc2571a72f5d3c7b7b576ce2de38d91e` **SHA256: **`acb18a0808aedb9a5e485f1469225b9ac97f21db9af78e4cd6939700debe1cb6` | | EFA 1.17.3 | **MD5: **`0517df4a190356ab559235147174cafd` **SHA256: **`5130998b0d2883bbae189b21ab215ecbc1b01ae0231659a9b4a17b0a33ebc6ca` | | EFA 1.17.2 | **MD5: **`a329dedab53c4832df218a24449f4c9a` **SHA256: **`bca1fdde8b32b00346e175e597ffab32a09a08ee9ab136875fb38283cc4cd099` | | EFA 1.17.1 | **MD5: **`733ae2cfc9d14b52017eaf0a2ab6b0ff` **SHA256: **`f29322640a88ae9279805993cb836276ea240623820848463ca686c8ce02136f` | | EFA 1.17.0 | **MD5: **`d430fc841563c11c3805c5f82a4746b1` **SHA256: **`75ab0cee4fb6bd38889dce313183f5d3a83bd233e0a6ef6205d8352821ea901d` | | EFA 1.16.0 | **MD5: **`399548d3b0d2e812d74dd67937b696b4` **SHA256: **`cecec36495a1bc6fdc82f97761a541e4fb6c9a3cbf3cfcb145acf25ea5dbd45b` | | EFA 1.15.2 | **MD5: **`955fea580d5170b05823d51acde7ca21` **SHA256: **`84df4fbc1b3741b6c073176287789a601a589313accc8e6653434e8d4c20bd49` | | EFA 1.15.1 | **MD5: **`c4610267039f72bbe4e35d7bf53519bc` **SHA256: **`be871781a1b9a15fca342a9d169219260069942a8bda7a8ad06d4baeb5e2efd7` | | EFA 1.15.0 | **MD5: **`9861694e1cc00d884fadac07d22898be` **SHA256: **`b329862dd5729d2d098d0507fb486bf859d7c70ce18b61c302982234a3a5c88f` | | EFA 1.14.1 | **MD5: **`50ba56397d359e57872fde1f74d4168a` **SHA256: **`c7b1b48e86fe4b3eaa4299d3600930919c4fe6d88cc6e2c7e4a408a3f16452c7` | | EFA 1.14.0 | **MD5: **`40805e7fd842c36ececb9fd7f921b1ae` **SHA256: **`662d62c12de85116df33780d40e0533ef7dad92709f4f613907475a7a1b60a97` | | EFA 1.13.0 | **MD5: **`c91d16556f4fd53becadbb345828221e` **SHA256: **`ad6705eb23a3fce44af3afc0f7643091595653a723ad0374084f4f2b715192e1` | | EFA 1.12.3 | **MD5: **`818aee81f097918cfaebd724eddea678` **SHA256: **`2c225321824788b8ca3fbc118207b944cdb096b847e1e0d1d853ef2f0d727172` | | EFA 1.12.2 | **MD5: **`956bb1fc5ae0d6f0f87d2e481d49fccf` **SHA256: **`083a868a2c212a5a4fcf3e4d732b685ce39cceb3ca7e5d50d0b74e7788d06259` | | EFA 1.12.1 | **MD5: **`f5bfe52779df435188b0a2874d0633ea` **SHA256: **`5665795c2b4f09d5f3f767506d4d4c429695b36d4a17e5758b27f033aee58900` | | EFA 1.12.0 | **MD5: **`d6c6b49fafb39b770297e1cc44fe68a6` **SHA256: **`28256c57e9ecc0b0778b41c1f777a9982b4e8eae782343dfe1246079933dca59` | | EFA 1.11.2 | **MD5: **`2376cf18d1353a4551e35c33d269c404` **SHA256: **`a25786f98a3628f7f54f7f74ee2b39bc6734ea9374720507d37d3e8bf8ee1371` | | EFA 1.11.1 | **MD5: **`026b0d9a0a48780cc7406bd51997b1c0` **SHA256: **`6cb04baf5ffc58ddf319e956b5461289199c8dd805fe216f8f9ab8d102f6d02a` | | EFA 1.11.0 | **MD5: **`7d9058e010ad65bf2e14259214a36949` **SHA256: **`7891f6d45ae33e822189511c4ea1d14c9d54d000f6696f97be54e915ce2c9dfa` | | EFA 1.10.1 | **MD5: **`78521d3d668be22976f46c6fecc7b730` **SHA256: **`61564582de7320b21de319f532c3a677d26cc46785378eb3b95c636506b9bcb4` | | EFA 1.10.0 | **MD5: **`46f73f5a7afe41b4bb918c81888fefa9` **SHA256: **`136612f96f2a085a7d98296da0afb6fa807b38142e2fc0c548fa986c41186282` | | EFA 1.9.5 | **MD5: **`95edb8a209c18ba8d250409846eb6ef4` **SHA256: **`a4343308d7ea4dc943ccc21bcebed913e8868e59bfb2ac93599c61a7c87d7d25` | | EFA 1.9.4 | **MD5: **`f26dd5c350422c1a985e35947fa5aa28` **SHA256: **`1009b5182693490d908ef0ed2c1dd4f813cc310a5d2062ce9619c4c12b5a7f14` | | EFA 1.9.3 | **MD5: **`95755765a097802d3e6d5018d1a5d3d6` **SHA256: **`46ce732d6f3fcc9edf6a6e9f9df0ad136054328e24675567f7029edab90c68f1` | | EFA 1.8.4 | **MD5: **`85d594c41e831afc6c9305263140457e` **SHA256: **`0d974655a09b213d7859e658965e56dc4f23a0eee2dc44bb41b6d039cc5bab45` | # Elastic Fabric Adapter release notes Release notes The following table describes the version history and changelog for the Elastic Fabric Adapter software. | Version | Changes | Release date | | --- | --- | --- | | 1.47.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | January 29, 2026 | | 1.46.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | December 12, 2025 | | 1.45.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | November 26, 2025 | | 1.45.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | November 17, 2025 | | 1.44.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 29, 2025 | | 1.43.3 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 01, 2025 | | 1.43.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | August 15, 2025 | | 1.43.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 31, 2025 | | 1.43.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 25, 2025 | | 1.42.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | June 6, 2025 | | 1.41.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | May 16, 2025 | | 1.40.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | May 2, 2025 | | 1.39.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | April 16, 2025 | | 1.38.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | March 3, 2025 | | 1.38.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | January 8, 2025 | | 1.37.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | November 18, 2024 | | 1.36.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | November 7, 2024 | | 1.35.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 14, 2024 | | 1.34.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | August 6, 2024 | | 1.33.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | June 20, 2024 | | 1.32.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | April 18, 2024 | | 1.31.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | March 7, 2024 | | 1.30.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | December 2023 | | 1.29.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | December 2023 | | 1.29.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | November 2023 | | 1.28.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 2023 | | 1.27.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | September 2023 | | 1.26.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | September 2023 | | 1.26.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | September 2023 | | 1.25.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | September 2023 | | 1.25.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 2023 | | 1.24.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 2023 | | 1.24.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | June 2023 | | 1.23.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | June 2023 | | 1.23.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | May 2023 | | 1.22.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | March 2023 | | 1.22.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | February 2023 | | 1.21.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | December 2022 | | 1.20.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | November 2022 | | 1.19.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 2022 | | 1.18.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | August 2022 | | 1.17.3 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | August 2022 | | 1.17.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 2022 | | 1.17.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 2022 | | 1.17.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 2022 | | 1.16.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | June 2022 | | 1.15.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | May 2022 | | 1.15.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | March 2022 | | 1.15.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | Feburary 2022 | | 1.14.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 2021 | | 1.14.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 2021 | | 1.13.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | August 2021 | | 1.12.3 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 2021 | | 1.12.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | June 2021 | | 1.12.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | May 2021 | | 1.12.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | May 2021 | | 1.11.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | February 2021 | | 1.11.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | December 2020 | | 1.11.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | December 2020 | | 1.10.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | November 2020 | | 1.10.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 2020 | | 1.9.5 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | September 2020 | | 1.9.4 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 2020 | | 1.9.3 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | June 2020 | | 1.8.4 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | April 2020 | | 1.8.3 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | February 2020 | | 1.8.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | January 2020 | | 1.8.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | January 2020 | | 1.8.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | December 2019 | | 1.7.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | December 2019 | | 1.7.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | November 2019 | | 1.6.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 2019 | | 1.6.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | October 2019 | | 1.5.4 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | September 2019 | | 1.5.3 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | September 2019 | | 1.5.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | August 2019 | | 1.5.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | August 2019 | | 1.4.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 2019 | | 1.4.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-changelog.html) | July 2019 | # Amazon EC2 topology EC2 topology Amazon EC2 topology provides a hierarchical view of the relative proximity of your compute capacity. You can use this information to manage high performance computing (HPC), machine learning (ML), and generative AI compute infrastructure at scale. **Available APIs** Amazon EC2 provides two APIs for understanding your EC2 topology: + [DescribeInstanceTopology](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTopology.html) + Shows where your *running* instances are located relative to each other in the network hierarchy. + Helps optimize where to run your workloads on your existing instances. + [DescribeCapacityReservationTopology](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeCapacityReservationTopology.html) + Shows where your reserved capacity will be located relative to each other in the network hierarchy *before you launch instances*. + Helps with capacity planning by letting you know the placement of reserved capacity before launching instances. **Key benefits** EC2 topology provides the following key benefits: + Capacity management – Optimize resource utilization. + Job scheduling – Make informed decisions about workload placement. + Node ranking – Understand relative proximity for performance optimization on tightly coupled instances. **Considerations** + Topology views are only available for: + Instances in the `running` state + Capacity Reservations in the `pending` or `active` state + Each topology view is unique per AWS account. + The AWS Management Console does not support viewing topology. + While topology information helps you understand instance placement, you can't use it to launch a new instance physically close to an existing instance. To influence instance placement, you can [create Capacity Reservations in cluster placement groups](cr-cpg.md). **Pricing** There is no additional cost for describing your EC2 topology. **Topics** + [How it works](how-ec2-instance-topology-works.md) + [Prerequisites](ec2-instance-topology-prerequisites.md) + [Examples](ec2-instance-topology-examples.md) # How Amazon EC2 topology works How it works The AWS network is arranged in a hierarchy of layers. EC2 instances connect into the network at or below the third layer, depending on the instance type. An instance's topology is described by a set of nodes, with one node in each layer of the network. The node set in the [DescribeInstanceTopology](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTopology.html) or [DescribeCapacityReservationTopology](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeCapacityReservationTopology.html) API response provides a top-down view of the network hierarchy, with the bottom node connected to an instance. **Note** Some instance types comprise 4 network nodes in a node set representing 4 layers in the network, while others comprise 3 network nodes representing 3 layers in the network. For the supported instance types, see [Instance types](ec2-instance-topology-prerequisites.md#inst-net-topology-prereqs-instance-types). Depending on the type of Capacity Reservation, you might see only 1, 2, or 3 network nodes. The following diagram provides a visual representation that you can use to understand EC2 topology. The network nodes are identified as **NN1** – **NN7**. The numerals **i**, **ii**, and **iii** identify the network layers. The numbers **1**, **2**, **3**, and **4** identify the EC2 instances. Instances connect to a node in the bottom layer, identified by **iii** in the following diagram. More than one instance can connect to the same node. ![\[Graphic representation of the instance topology.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/instance-topology.png) In this example: + Instance 1 connects to network node 4 (NN4) in layer iii. NN4 connects to network node 2 (NN2) in layer ii, and NN2 connects to network node 1 (NN1) in layer i, which is the top of the network hierarchy in this example. The network node set comprises NN1, NN2, and NN4, expressed hierarchically from the upper layers to the bottom layer. + Instance 2 also connects to network node 4 (NN4). Instance 1 and instance 2 share the same network node set: NN1, NN2, and NN4. + Instance 3 connects to network node 5 (NN5). NN5 connects to NN2, and NN2 connects to NN1. The network node set for instance 3 is NN1, NN2, and NN5. + Instance 4 connects to network node 6 (NN6). Its network node set is NN1, NN3, and NN6. When considering the proximity of instances 1, 2, and 3, instances 1 and 2 are closer to each other because they connect to the same network node (NN4), while instance 3 is further away because it connects to a different network node (NN5). When considering the proximity of all the instances in this diagram, instances 1, 2, and 3 are closer to each other than they are to instance 4 because they share NN2 in their network node set. As a general rule, if the network node connected to any two instances is the same, these instances are physically close to each other, as is the case with instances 1 and 2. Furthermore, the fewer the number of hops between network nodes, the closer the instances are to each other. For example, instances 1 and 3 have fewer hops to a common network node (NN2) than they have to the network node (NN1) they have in common with instance 4, and are therefore closer to each other than they are to instance 4. There are no instances running under network node 7 (NN7) in this example, and therefore the API output won't include NN7. ## How to interpret the DescribeInstanceTopology output You can describe the instance topology using the [DescribeInstanceTopology](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTopology.html) API. The output provides a hierarchical view of the underlying network topology for an instance. The following example output corresponds to the network topology information of the four instances in the preceding diagram. Comments are included in the example output for the purposes of this example. The following information in the output is important to note: + `NetworkNodes` describes the network node set of a single instance. + In each network node set, the network nodes are listed in hierarchical order from top to bottom. + The network node that is connected to the instance is the last network node in the list (the bottom layer). + To work out which instances are close to each other, first find common network nodes in the bottom layer. If there are no common network nodes in the bottom layer, then find common network nodes in the upper layers. In the following example output, `i-1111111111example` and `i-2222222222example` are located closest to each other compared to the other instances in this example because they have the network node `nn-4444444444example` in common in the bottom layer. **Note** The response contains 3 or more network nodes. For information about the number of network nodes in the response for each supported instance type, see [Instance types](ec2-instance-topology-prerequisites.md#inst-net-topology-prereqs-instance-types). ``` { "Instances": [ { "InstanceId": "i-1111111111example", //Corresponds to instance 1 "InstanceType": "p4d.24xlarge", "GroupName": "ML-group", "NetworkNodes": [ "nn-1111111111example", //Corresponds to NN1 in layer i "nn-2222222222example", //Corresponds to NN2 in layer ii "nn-4444444444example" //Corresponds to NN4 in layer iii - bottom layer, connected to the instance ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" }, { "InstanceId": "i-2222222222example", //Corresponds to instance 2 "InstanceType": "p4d.24xlarge", "NetworkNodes": [ "nn-1111111111example", //Corresponds to NN1 - layer i "nn-2222222222example", //Corresponds to NN2 - layer ii "nn-4444444444example" //Corresponds to NN4 - layer iii - connected to instance ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" }, { "InstanceId": "i-3333333333example", //Corresponds to instance 3 "InstanceType": "trn1.32xlarge", "NetworkNodes": [ "nn-1111111111example", //Corresponds to NN1 - layer i "nn-2222222222example", //Corresponds to NN2 - layer ii "nn-5555555555example" //Corresponds to NN5 - layer iii - connected to instance ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" }, { "InstanceId": "i-444444444example", //Corresponds to instance 4 "InstanceType": "trn1.2xlarge", "NetworkNodes": [ "nn-1111111111example", //Corresponds to NN1 - layer i "nn-3333333333example", //Corresponds to NN3 - layer ii "nn-6666666666example" //Corresponds to NN6 - layer iii - connected to instance ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" } ], "NextToken": "SomeEncryptedToken" } ``` ## How to interpret the DescribeCapacityReservationTopology output You can describe the Capacity Reservation topology using the [DescribeCapacityReservationTopology](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeCapacityReservationTopology.html) API. The output provides a hierarchical view of the underlying network topology for the reserved capacity. The following example output corresponds to the network topology information in the preceding diagram. Comments are included in the example output for the purposes of this example. The following information in the output is important to note: + `NetworkNodes` describes the network node set of a single Capacity Reservation. + In each network node set, the network nodes are listed in hierarchical order from top to bottom. + The network node that is connected to the Capacity Reservation is the last network node in the list (the bottom layer). + To work out whether Capacity Reservations will be close to each other, first find common network nodes in the bottom layer in the output. If there are no common network nodes in the bottom layer, then find common network nodes in the upper layers. In the following example output, `cr-1111111111example` is located on `nn-2222222222example` and `cr-2222222222example` is located on `nn-3333333333example`. Because the Capacity Reservations are on different network nodes in `layer ii`, communication from instances in one Capacity Reservation to instances in the other Capacity Reservation will be inefficient. **Note** The response contains 1, 2, or 3 network nodes depending on the type of Capacity Reservation. ``` { "CapacityReservations": [ { "CapacityReservationId": "cr-1111111111example", "CapacityBlockId": "null", "State": "active", "InstanceType": "p4d.24xlarge", "NetworkNodes": [ "nn-1111111111example", //Corresponds to NN1 - layer i "nn-2222222222example" //Corresponds to NN2 - layer ii // Visibility of additional nodes requires an instance launch and // the DescribeInstanceTopology API ], "AvailabilityZone": "us-west-2a" }, { "CapacityReservationId": "cr-2222222222example", "CapacityBlockId": "null", "State": "active", "InstanceType": "trn1.2xlarge", "NetworkNodes": [ "nn-1111111111example", //Corresponds to NN1 - layer i "nn-3333333333example" //Corresponds to NN3 - layer ii // Visibility of additional nodes requires an instance launch and // the DescribeInstanceTopology API ], "AvailabilityZone": "us-west-2a" } ], "NextToken": "SomeEncryptedToken" } ``` ## Differences between DescribeInstanceTopology and DescribeCapacityReservationTopology The following table compares the key differences between the DescribeInstanceTopology and DescribeCapacityReservationTopology APIs: | Comparison point | DescribeInstanceTopology | DescribeCapacityReservationTopology | | --- | --- | --- | | Usage phase | Post-launch (execution mode) | Pre-launch (planning and management mode) | | Primary purpose | Optimize workloads on running instances | Capacity planning and Capacity Reservation management (merge, split, assign) before instance launch | | Number of network nodes | Shows all nodes for a running instance. If the instance is in a Capacity Reservation, the first nodes will match the corresponding Capacity Reservation topology, followed by additional nodes to connect to the instance. | Shows a partial set of nodes, which vary based on the Capacity Reservation state (`pending` or `active`) and type.\$1 | | State | Instances must be in `running` state | Capacity Reservations must be in `pending` or `active` state | | Use cases | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/how-ec2-instance-topology-works.html) | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/how-ec2-instance-topology-works.html) | \$1 For Capacity Blocks for Ultraservers, the network node set is the same when describing the topology for an `active` Capacity Reservation or its running instance. # Prerequisites for Amazon EC2 topology Prerequisites To describe your Amazon EC2 topology, ensure that your instances and Capacity Reservations meet the following prerequisites. **Topics** + [ ## AWS Regions ](#inst-net-topology-prereqs-regions) + [ ## Instance types ](#inst-net-topology-prereqs-instance-types) + [ ## State ](#inst-net-topology-prereqs-instance-state) + [ ## IAM permissions ](#ec2-instance-topology-iam-permissions) ## AWS Regions Supported AWS Regions: + US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon) + Africa (Cape Town) + Asia Pacific (Jakarta), Asia Pacific (Hong Kong), Asia Pacific (Hyderabad), Asia Pacific (Melbourne), Asia Pacific (Mumbai), Asia Pacific (Osaka), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo) + Canada (Central) + Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), Europe (Spain), Europe (Stockholm), Europe (Zurich) + Israel (Tel Aviv) + Middle East (Bahrain), Middle East (UAE) + South America (São Paulo) + AWS GovCloud (US-West) The DescribeCapacityReservationTopology API is not supported in Israel (Tel Aviv) and AWS GovCloud (US-West). ## Instance types Supported instance types: + Returns **3\$1 network nodes** in the response: + `g6e.xlarge` \$1 `g6e.2xlarge` \$1 `g6e.4xlarge` \$1 `g6e.8xlarge` \$1 `g6e.12xlarge` \$1 `g6e.16xlarge` \$1 `g6e.24xlarge` \$1 `g6e.48xlarge` \$1 `g7e.2xlarge` \$1 `g7e.4xlarge` \$1 `g7e.8xlarge` \$1 `g7e.12xlarge` \$1 `g7e.24xlarge` \$1 `g7e.48xlarge` + `hpc6a.48xlarge` \$1 `hpc6id.32xlarge` \$1 `hpc7g.4xlarge` \$1 `hpc7g.8xlarge` \$1 `hpc7g.16xlarge` \$1 `hpc7a.12xlarge` \$1 `hpc7a.24xlarge` \$1 `hpc7a.48xlarge` \$1 `hpc7a.96xlarge` \$1 `hpc8a.96xlarge` + `p3dn.24xlarge` \$1 `p4d.24xlarge` \$1 `p4de.24xlarge` \$1 `p5.48xlarge` \$1 `p5e.48xlarge` \$1 `p5en.48xlarge` \$1 `p6e-gb200.36xlarge` + `trn1.2xlarge` \$1 `trn1.32xlarge` \$1 `trn1n.32xlarge` \$1 `trn2.48xlarge` \$1 `trn2u.48xlarge` + Returns **4\$1 network nodes** in the response: + `p6-b200.48xlarge` \$1 `p6-b300.48xlarge` \$1 The number of network nodes returned is only applicable when using the DescribeInstanceTopology API. For the DescribeCapacityReservationTopology API, the number of network nodes returned will vary depending on the type and state of the Capacity Reservation. The available instance types vary by Region. For more information, see [Amazon EC2 instance types by Region](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html). ## State + For `DescribeInstanceTopology` – Instances must be in the `running` state. + For `DescribeCapacityReservationTopology` – Capacity Reservations must be in the `pending` or `active` state. You can’t get topology information for instances or Capacity Reservations in any other state. ## IAM permissions Your IAM identity (user, user group, or role) requires the following permissions: + `ec2:DescribeInstanceTopology` + `ec2:DescribeCapacityReservationTopology` # Examples for Amazon EC2 instance topology Examples You can use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command to describe the topology for your EC2 instances. And you can use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command to describe the topology of your Capacity Reservations. When you use the `describe-instance-topology` or `describe-capacity-reservation-topology` command without parameters or filters, the response includes all your instances or Capacity Reservations (depending on the command used) that match the supported instance types for this command in the specified Region. You can specify the Region by including the `--region` parameter, or by setting a default Region. For more information about setting a default Region, see [Select a Region for your Amazon EC2 resources](using-regions-availability-zones-setup.md). You can include parameters to return instances or Capacity Reservations that match specified instance or Capacity Reservation IDs or placement group names. You can also include filters to return instances or Capacity Reservations that match a specified instance type or instance family, or instances or Capacity Reservations in a specified Availability Zone or Local Zone. You can include a single parameter or filter, or a combination of parameters and filters. The output is paginated, with up to 20 instances or Capacity Reservations per page by default. You can specify up to 100 instances or Capacity Reservations per page using the `--max-results` parameter. For more information, see [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) and [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-reservation-topology-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-reservation-topology-topology.html). **Required permissions** The following permissions are required: + `ec2:DescribeInstanceTopology` – For describing instance topology + `ec2:DescribeCapacityReservationTopology` – For describing Capacity Reservation topology **Contents** + [ ## Example 1: DescribeInstanceTopology - Instance IDs ](#instance-topology-ex1) + [ ## Example 2: DescribeInstanceTopology - Placement group name parameter ](#instance-topology-ex2) + [ ## Example 3: DescribeInstanceTopology - Instance type filter ](#instance-topology-ex3) + [ ### Example 3a – Exact match filter for a specified instance type ](#instance-topology-ex3a) + [ ### Example 3b – Wild card filter for an instance family ](#instance-topology-ex3b) + [ ### Example 3c – Combined instance family and exact match filters ](#instance-topology-ex3c) + [ ## Example 4: DescribeInstanceTopology - Zone ID filter ](#instance-topology-ex4) + [ ### Example 4a – Availability Zone filter ](#instance-topology-ex4a) + [ ### Example 4b – Local Zone filter ](#instance-topology-ex4b) + [ ### Example 4c – Combined Availability Zone and Local Zone filters ](#instance-topology-ex4c) + [ ## Example 5: DescribeInstanceTopology - Instance type and zone ID filters ](#instance-topology-ex5) + [ ## Example 6: DescribeCapacityReservationTopology - Capacity Reservation IDs ](#instance-topology-ex6) + [ ## Example 7: DescribeCapacityReservationTopology - Instance type filter ](#instance-topology-ex7) ## Example 1: DescribeInstanceTopology - Instance IDs Ex 1: DescribeInstanceTopology - Instance IDs ------ #### [ AWS CLI ] **To describe the topology of specific instances** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command with the `--instance-ids` parameter. The output includes only the instances that match the specified instance IDs. ``` aws ec2 describe-instance-topology \ --region us-west-2 \ --instance-ids i-1111111111example i-2222222222example ``` The following is example output. ``` { "Instances": [ { "InstanceId": "i-1111111111example", "InstanceType": "p4d.24xlarge", "GroupName": "ML-group", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3333333333example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" }, { "InstanceId": "i-2222222222example", "InstanceType": "trn1n.32xlarge", "GroupName": "HPC-group", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3214313214example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of specific instances** Use the [Get-EC2InstanceTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceTopology.html) cmdlet. ``` Get-EC2InstanceTopology ` -InstanceId i-1111111111example, i-2222222222example ``` ------ ## Example 2: DescribeInstanceTopology - Placement group name parameter Ex 2: DescribeInstanceTopology - Placement group name parameter ------ #### [ AWS CLI ] **To describe the topology of instances in a specific placement group** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command with the `group-names` parameter. The output includes only the instances that are in either of the specified placement groups. ``` aws ec2 describe-instance-topology \ --region us-west-2 \ --group-names ML-group HPC-group ``` The following is example output. ``` { "Instances": [ { "InstanceId": "i-1111111111example", "InstanceType": "p4d.24xlarge", "GroupName": "ML-group", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3333333333example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" }, { "InstanceId": "i-2222222222example", "InstanceType": "trn1n.32xlarge", "GroupName": "HPC-group", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3214313214example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of instances in a specific placement group** Use the [Get-EC2InstanceTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceTopology.html) cmdlet. ``` Get-EC2InstanceTopology ` -GroupName ML-group, HPC-group ``` ------ ## Example 3: DescribeInstanceTopology - Instance type filter Ex 3: DescribeInstanceTopology - Instance type filter You can filter by a specified instance type (exact match) or filter by an instance family (using a wildcard). You can also combine a specified instance type filter and instance family filter. ### Example 3a – Exact match filter for a specified instance type ------ #### [ AWS CLI ] **To describe the topology of instances with a specific instance type** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command with the `instance-type` filter. The output includes only the instances with the specified instance type. ``` aws ec2 describe-instance-topology \ --region us-west-2 \ --filters Name=instance-type,Values=trn1n.32xlarge ``` The following is example output. ``` { "Instances": [ { "InstanceId": "i-2222222222example", "InstanceType": "trn1n.32xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3333333333example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of instances with a specific instance type** Use the [Get-EC2InstanceTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceTopology.html) cmdlet. ``` Get-EC2InstanceTopology ` -Filter @{Name="instance-type"; Values="trn1n.32xlarge"} ``` ------ ### Example 3b – Wild card filter for an instance family ------ #### [ AWS CLI ] **To describe the topology of instances with a specific instance family** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command with the `instance-type` filter. The output includes only the instances with the specified instance family. ``` aws ec2 describe-instance-topology \ --region us-west-2 \ --filters Name=instance-type,Values=trn1* ``` The following is example output. ``` { "Instances": [ { "InstanceId": "i-2222222222example", "InstanceType": "trn1n.32xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3333333333example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" }, { "InstanceId": "i-3333333333example", "InstanceType": "trn1.32xlarge", "NetworkNodes": [ "nn-1212121212example", "nn-1211122211example", "nn-1311133311example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az4", "AvailabilityZone": "us-west-2d" }, { "InstanceId": "i-444444444example", "InstanceType": "trn1.2xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-5434334334example", "nn-1235301234example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of instances with a specific instance family** Use the [Get-EC2InstanceTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceTopology.html) cmdlet. ``` Get-EC2InstanceTopology ` -Filter @{Name="instance-type"; Values="trn1*"} ``` ------ ### Example 3c – Combined instance family and exact match filters ------ #### [ AWS CLI ] **To describe the topology of instances with an instance family or instance type** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command with the `instance-type` filter. The output includes only the instances that meet the specified criteria. ``` aws ec2 describe-instance-topology \ --region us-west-2 \ --filters "Name=instance-type,Values=p4d*,trn1n.32xlarge" ``` The following is example output. ``` { "Instances": [ { "InstanceId": "i-1111111111example", "InstanceType": "p4d.24xlarge", "GroupName": "ML-group", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3333333333example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" }, { "InstanceId": "i-2222222222example", "InstanceType": "trn1n.32xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-4343434343example" ], "CapacityBlockId": "null", "ZoneId": "usw2-az2", "AvailabilityZone": "us-west-2a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of instances with an instance family or instance type** Use the [Get-EC2InstanceTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceTopology.html) cmdlet. ``` Get-EC2InstanceTopology ` -Filter @{Name="instance-type"; Values="p4d*", "trn1n.32xlarge"} ``` ------ ## Example 4: DescribeInstanceTopology - Zone ID filter Ex 4: DescribeInstanceTopology - Zone ID filter You can use the `zone-id` filter to filter by an Availability Zone or Local Zone. You can also combine an Availability Zone filter and Local Zone filter. ### Example 4a – Availability Zone filter ------ #### [ AWS CLI ] **To describe the topology of instances in a specific Availability Zone** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command with the `zone-id` filter. The output includes only the instances in the specified Availability Zone. ``` aws ec2 describe-instance-topology \ --region us-east-1 \ --filters Name=zone-id,Values=use1-az1 ``` The following is example output. ``` { "Instances": [ { "InstanceId": "i-2222222222example", "InstanceType": "trn1n.32xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3214313214example" ], "CapacityBlockId": "null", "ZoneId": "use1-az1", "AvailabilityZone": "us-east-1a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of instances in a specific Availability Zone** Use the [Get-EC2InstanceTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceTopology.html) cmdlet. ``` Get-EC2InstanceTopology ` -Filter @{Name="zone-id"; Values="use1-az1"} ``` ------ ### Example 4b – Local Zone filter ------ #### [ AWS CLI ] **To describe the topology of instances in a specific Local Zone** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command with the `zone-id` filter. The output includes only the instances in the specified Local Zone. ``` aws ec2 describe-instance-topology \ --region us-east-1 \ --filters Name=zone-id,Values=use1-atl2-az1 ``` The following is example output. ``` { "Instances": [ { "InstanceId": "i-1111111111example", "InstanceType": "p4d.24xlarge", "GroupName": "ML-group", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3333333333example" ], "CapacityBlockId": "null", "ZoneId": "use1-atl2-az1", "AvailabilityZone": "us-east-1-atl-2a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of instances in a specific Local Zone** Use the [Get-EC2InstanceTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceTopology.html) cmdlet. ``` Get-EC2InstanceTopology ` -Filter @{Name="zone-id"; Values="use1-atl2-az1"} ``` ------ ### Example 4c – Combined Availability Zone and Local Zone filters ------ #### [ AWS CLI ] **To describe the topology of instances in a specific zone** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command with the `zone-id` filter. The output includes only the instances that are in either of the specified zones. ``` aws ec2 describe-instance-topology \ --region us-east-1 \ --filters Name=zone-id,Values=use1-az1,use1-atl2-az1 ``` The following is example output. ``` { "Instances": [ { "InstanceId": "i-1111111111example", "InstanceType": "p4d.24xlarge", "GroupName": "ML-group", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3333333333example" ], "CapacityBlockId": "null", "ZoneId": "use1-atl2-az1", "AvailabilityZone": "us-east-1-atl-2a" }, { "InstanceId": "i-2222222222example", "InstanceType": "trn1n.32xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3214313214example" ], "CapacityBlockId": "null", "ZoneId": "use1-az1", "AvailabilityZone": "us-east-1a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of instances in a specific zone** Use the [Get-EC2InstanceTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceTopology.html) cmdlet. ``` Get-EC2InstanceTopology ` -Filter @{Name="zone-id"; Values="use1-az1", "use1-atl2-az1"} ``` ------ ## Example 5: DescribeInstanceTopology - Instance type and zone ID filters Ex 5: DescribeInstanceTopology - Instance type and zone ID filters You can combine filters in a single command. ------ #### [ AWS CLI ] **To describe the topology of instances with specific instance types, instance families, and zones** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-topology.html) command with the `instance-type` and `zone-id` filters. The response contains any instances with either of the specified instance types and are in either of the specified zones. ``` aws ec2 describe-instance-topology \ --region us-east-1 \ --filters "Name=instance-type,Values=p4d*,trn1n.32xlarge" \ "Name=zone-id,Values=use1-az1,use1-atl2-az1" ``` The following is example output. ``` { "Instances": [ { "InstanceId": "i-1111111111example", "InstanceType": "p4d.24xlarge", "GroupName": "ML-group", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3333333333example" ], "CapacityBlockId": "null", "ZoneId": "use1-atl2-az1", "AvailabilityZone": "us-east-1-atl-2a" }, { "InstanceId": "i-2222222222example", "InstanceType": "trn1n.32xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example", "nn-3214313214example" ], "CapacityBlockId": "null", "ZoneId": "use1-az1", "AvailabilityZone": "us-east-1a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of instances with specific instance types, instance families, and zones** Use the [Get-EC2InstanceTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2InstanceTopology.html) cmdlet. ``` Get-EC2InstanceTopology ` -Filter @{Name="instance-type"; Values="p4d*", "trn1n.32xlarge"} ` @{Name="zone-id"; Values="use1-az1", "use1-atl2-az1"} ``` ------ ## Example 6: DescribeCapacityReservationTopology - Capacity Reservation IDs Ex 6: DescribeCapacityReservationTopology - Capacity Reservation IDs ------ #### [ AWS CLI ] **To describe the topology of specific Capacity Reservations** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-capacity-reservation-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-capacity-reservation-topology.html) command with the `capacity-reservation-id` parameter. The output includes only the Capacity Reservations that match the specified Capacity Reservation IDs. ``` aws ec2 describe-capacity-reservation-topology \ --region us-east-1 \ --capacity-reservation-id cr-1111111111example cr-2222222222example ``` The following is example output. ``` { "CapacityReservations": [ { "CapacityReservationId": "cr-1111111111example", "CapacityBlockId": "null", "State": "active", "InstanceType": "p5.48xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example" ], "AvailabilityZone": "us-east-1a" }, { "CapacityReservationId": "cr-2222222222example", "CapacityBlockId": "null", "State": "active", "InstanceType": "p5en.48xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example" ], "AvailabilityZone": "us-east-1a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of specific Capacity Reservations** Use the [Get-EC2CapacityReservationTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2CapacityReservationTopology.html) cmdlet. ``` Get-EC2CapacityReservationTopology ` -CapacityReservationId cr-1111111111example cr-2222222222example ``` ------ ## Example 7: DescribeCapacityReservationTopology - Instance type filter Ex 7: DescribeCapacityReservationTopology - Instance type filter You can filter by a specified instance type (exact match) or filter by an instance family (using a wildcard). You can also combine a specified instance type filter and instance family filter. ------ #### [ AWS CLI ] **To describe the topology of Capacity Reservations with a specific instance type** Use the [https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-capacity-reservation-topology.html](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-capacity-reservation-topology.html) command with the `instance-type` filter. The response contains any instances with the specified instance type. ``` aws ec2 describe-capacity-reservation-topology \ --region us-east-1 \ --filters Name=instance-type,Values=p5en.48xlarge ``` The following is example output. ``` { "CapacityReservations": [ { "CapacityReservationId": "cr-2222222222example", "CapacityBlockId": "null", "State": "active", "InstanceType": "p5en.48xlarge", "NetworkNodes": [ "nn-1111111111example", "nn-2222222222example" ], "AvailabilityZone": "us-east-1a" } ], "NextToken": "SomeEncryptedToken" } ``` ------ #### [ PowerShell ] **To describe the topology of Capacity Reservations with a specific instance type** Use the [Get-EC2CapacityReservationTopology](https://docs.aws.amazon.com/powershell/latest/reference/items/Get-EC2CapacityReservationTopology.html) cmdlet. ``` Get-EC2CapacityReservationTopology ` -Filter @{Name="instance-type"; Values="p5en.48xlarge"} ``` ------ # Placement groups for your Amazon EC2 instances Placement groups To meet the needs of your workload, you can launch a group of *interdependent* EC2 instances into a *placement group* to influence their placement. Depending on the type of workload, you can create a placement group using one of the following placement strategies: + **Cluster** – Packs instances close together inside an Availability Zone. This strategy enables workloads to achieve the low-latency network performance necessary for tightly-coupled node-to-node communication that is typical of high-performance computing (HPC) applications. + **Partition** – Spreads your instances across logical partitions such that groups of instances in one partition do not share the underlying hardware with groups of instances in different partitions. This strategy is typically used by large distributed and replicated workloads, such as Hadoop, Cassandra, and Kafka. + **Spread** – Strictly places a small group of instances across distinct underlying hardware to reduce correlated failures. Placement groups are optional. If you don't launch your instances into a placement group, EC2 tries to place the instances in such a way that all of your instances are spread out across the underlying hardware to minimize correlated failures. **Pricing** There is no charge for creating a placement group. **Rules and limitations** Before you use placement groups, be aware of the following rules: + An instance can be placed in one placement group at a time; you can't place an instance in multiple placement groups. + You can't merge placement groups. + [On-Demand Capacity Reservations](ec2-capacity-reservations.md#capacity-reservations-limits) and [zonal Reserved Instances](reserved-instances-scope.md) allow you to reserve capacity for EC2 instances in Availability Zones. When you launch an instance, if the instance attributes match those specified by an On-Demand Capacity Reservation or a zonal Reserved Instance, then the reserved capacity is automatically used by the instance. This is also true if you launch the instance into a placement group. + You can't launch Dedicated Hosts in placement groups. + You can't launch a Spot Instance that is configured to stop or hibernate on interruption in a placement group. **Topics** + [Placement strategies](placement-strategies.md) + [Create a placement group](create-placement-group.md) + [Change instance placement](change-instance-placement-group.md) + [ # Delete a placement group ](delete-placement-group.md) + [ # Shared placement groups ](share-placement-group.md) + [ # Placement groups on AWS Outposts ](placement-groups-outpost.md) # Placement strategies for your placement groups Placement strategies You can create a placement group for your EC2 instances using one of the following placement strategies. **Topics** + [ ## Cluster placement groups ](#placement-groups-cluster) + [ ## Partition placement groups ](#placement-groups-partition) + [ ## Spread placement groups ](#placement-groups-spread) ## Cluster placement groups A cluster placement group is a logical grouping of instances within a single Availability Zone. Instances are not isolated to a single rack. A cluster placement group can span peered virtual private networks (VPCs) in the same Region. Instances in the same cluster placement group enjoy a higher per-flow throughput limit for TCP/IP traffic and are placed in the same high-bisection bandwidth segment of the network. The following image shows instances that are placed into a cluster placement group. ![\[A cluster placement group.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/placement-group-cluster.png) Cluster placement groups are recommended for applications that benefit from low network latency, high network throughput, or both. They are also recommended when the majority of the network traffic is between the instances in the group. To provide the lowest latency and the highest packet-per-second network performance for your placement group, choose an instance type that supports enhanced networking. For more information, see [Enhanced Networking](enhanced-networking.md). We recommend that you launch your instances in the following way: + Use a single launch request to launch the number of instances that you need in the placement group. + Use the same instance type for all instances in the placement group. If you try to add more instances to the placement group later, or if you try to launch more than one instance type in the placement group, you increase your chances of getting an insufficient capacity error. If you stop an instance in a placement group and then start it again, it still runs in the placement group. However, the start fails if there isn't enough capacity for the instance. If you receive a capacity error when launching an instance in a placement group that already has running instances, stop and start all of the instances in the placement group, and try the launch again. Starting the instances may migrate them to hardware that has capacity for all of the requested instances. **Rules and limitations** The following rules apply to cluster placement groups: + The following instance types are supported: + Current generation instances, except for [burstable performance](burstable-performance-instances.md) instances (for example, T2), [Mac1 instances](ec2-mac-instances.md), and M7i-flex instances. + The following previous generation instances: A1, C3, C4, I2, M4, R3, and R4. + A cluster placement group can't span multiple Availability Zones. + The maximum network throughput speed of traffic between two instances in a cluster placement group is limited by the slower of the two instances. For applications with high-throughput requirements, choose an instance type with network connectivity that meets your requirements. + For instances that are enabled for enhanced networking, the following rules apply: + Instances within a cluster placement group can use up to 10 Gbps for single-flow traffic. Instances that are not within a cluster placement group can use up to 5 Gbps for single-flow traffic. + Traffic to and from Amazon S3 buckets within the same Region over the public IP address space or through a VPC endpoint can use all available instance aggregate bandwidth. + You can launch multiple instance types into a cluster placement group. However, this reduces the likelihood that the required capacity will be available for your launch to succeed. We recommend using the same instance type for all instances in a cluster placement group. + We recommend that you reserve capacity explicitly in the cluster placement group by creating an [On-Demand Capacity Reservation in the cluster placement group](cr-cpg.md). Note that you can't reserve capacity using zonal Reserved Instances, as they can't reserve capacity explicitly in a placement group. + Network traffic to the internet and over an Direct Connect connection to on-premises resources is limited to 5 Gbps for cluster placement groups. ## Partition placement groups Partition placement groups help reduce the likelihood of correlated hardware failures for your application. When using partition placement groups, Amazon EC2 divides each group into logical segments called partitions. Amazon EC2 ensures that each partition within a placement group has its own set of racks. Each rack has its own network and power source. No two partitions within a placement group share the same racks, allowing you to isolate the impact of hardware failure within your application. The following image is a simple visual representation of a partition placement group in a single Availability Zone. It shows instances that are placed into a partition placement group with three partitions—**Partition 1**, **Partition 2**, and **Partition 3**. Each partition comprises multiple instances. The instances in a partition do not share racks with the instances in the other partitions, allowing you to contain the impact of a single hardware failure to only the associated partition. ![\[A partition placement group with three partitions.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/placement-group-partition.png) Partition placement groups can be used to deploy large distributed and replicated workloads, such as HDFS, HBase, and Cassandra, across distinct racks. When you launch instances into a partition placement group, Amazon EC2 tries to distribute the instances evenly across the number of partitions that you specify. You can also launch instances into a specific partition to have more control over where the instances are placed. A partition placement group can have partitions in multiple Availability Zones in the same Region. A partition placement group can have a maximum of seven partitions per Availability Zone. The number of instances that can be launched into a partition placement group is limited only by the limits of your account. In addition, partition placement groups offer visibility into the partitions — you can see which instances are in which partitions. You can share this information with topology-aware applications, such as HDFS, HBase, and Cassandra. These applications use this information to make intelligent data replication decisions for increasing data availability and durability. If you start or launch an instance in a partition placement group and there is insufficient unique hardware to fulfill the request, the request fails. Amazon EC2 makes more distinct hardware available over time, so you can try your request again later. **Rules and limitations** The following rules apply to partition placement groups: + A partition placement group supports a maximum of seven partitions per Availability Zone. The number of instances that you can launch in a partition placement group is limited only by your account limits. + When instances are launched into a partition placement group, Amazon EC2 tries to evenly distribute the instances across all partitions. Amazon EC2 doesn’t guarantee an even distribution of instances across all partitions. + A partition placement group with Dedicated Instances can have a maximum of two partitions. + Capacity Reservations do not reserve capacity in a partition placement group. ## Spread placement groups A spread placement group is a group of instances that are each placed on distinct hardware. Spread placement groups are recommended for applications that have a small number of critical instances that should be kept separate from each other. Launching instances in a spread level placement group reduces the risk of simultaneous failures that might occur when instances share the same equipment. Spread level placement groups provide access to distinct hardware, and are therefore suitable for mixing instance types or launching instances over time. If you start or launch an instance in a spread placement group and there is insufficient unique hardware to fulfill the request, the request fails. Amazon EC2 makes more distinct hardware available over time, so you can try your request again later. Placement groups can spread instances across racks or hosts. Rack level spread placement groups can be used in AWS Regions and on AWS Outposts. Host level spread placement groups can be used with AWS Outposts only. **Rack level spread placement groups** The following image shows seven instances in a single Availability Zone that are placed into a spread placement group. The seven instances are placed on seven different racks, each rack has its own network and power source. ![\[A spread placement group.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/placement-group-spread.png) A rack level spread placement group can span multiple Availability Zones in the same Region. In a Region, a rack level spread placement group can have a maximum of seven running instances per Availability Zone per group. With Outposts, a rack level spread placement group can hold as many instances as you have racks in your Outpost deployment. **Host level spread placement groups** Host level spread placement groups are only available with AWS Outposts. A host spread level placement group can hold as many instances as you have hosts in your Outpost deployment. For more information, see [Placement groups on AWS Outposts](placement-groups-outpost.md). **Rules and limitations** The following rules apply to spread placement groups: + A rack spread placement group supports a maximum of seven running instances per Availability Zone. For example, in a Region with three Availability Zones, you can run a total of 21 instances in the group, with seven instances in each Availability Zone. If you try to start an eighth instance in the same Availability Zone and in the same spread placement group, the instance will not launch. If you need more than seven instances in an Availability Zone, we recommend that you use multiple spread placement groups. Using multiple spread placement groups does not provide guarantees about the spread of instances between groups, but it does help ensure the spread for each group, thus limiting the impact from certain classes of failures. + Spread placement groups are not supported for Dedicated Instances. + Host level spread placement groups are only supported for placement groups on AWS Outposts. A host level spread placement group can hold as many instances as you have hosts in your Outpost deployment. + In a Region, a rack level spread placement group can have a maximum of seven running instances per Availability Zone per group. With AWS Outposts, a rack level spread placement group can hold as many instances as you have racks in your Outpost deployment. + Capacity Reservations do not reserve capacity in a spread placement group. # Create a placement group for your EC2 instances Create a placement group You can use a placement group to control the placement of instances relative to each other. After you create a placement group, you can launch instances in the placement group. **Limitation** You can create a maximum of 500 placement groups per Region. ------ #### [ Console ] **To create a placement group** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Placement Groups**. 1. Choose **Create placement group**. 1. Specify a name for the group. 1. Choose the placement strategy for the group: **Cluster**, **Spread**, or **Partition**. If you chose **Spread**, you must choose the spread level: **Rack** or **Host**. If you chose **Partition**, you must enter the number of partitions for the group. 1. (Optional) To add a tag, choose **Add new tag**, and then enter a key and value. 1. Choose **Create group**. ------ #### [ AWS CLI ] Use the [create-placement-group](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-placement-group.html) command. **To create a cluster placement group** The following example creates a placement group that uses the `cluster` placement strategy, and it applies a tag with a key of `purpose` and a value of `production`. ``` aws ec2 create-placement-group \ --group-name my-cluster \ --strategy cluster \ --tag-specifications 'ResourceType=placement-group,Tags={Key=purpose,Value=production}' ``` **To create a partition placement group** The following example creates a placement group that uses the `partition` placement strategy, and specifies the five partitions using the `--partition-count` parameter. ``` aws ec2 create-placement-group \ --group-name HDFS-Group-A \ --strategy partition \ --partition-count 5 ``` ------ #### [ PowerShell ] Use the [New-EC2PlacementGroup](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2PlacementGroup.html) cmdlet. **To create a cluster placement group** The following example creates a cluster placement group. ``` New-EC2PlacementGroup ` -GroupName my-placement-group ` -Strategy cluster ``` **To create a partition placement group** The following example creates a partition placement group. ``` New-EC2PlacementGroup ` -GroupName my-placement-group ` -Strategy partition ` -PartitionCount 5 ``` ------ # Change the placement for an EC2 instance Change instance placement You can change the placement group for an instance as follows: + Add an instance to a placement group + Move an instance from one placement group to another + Remove an instance from a placement group **Requirement** Before you can change the placement group for an instance, the instance must be in the `stopped` state. ------ #### [ Console ] **To change the instance placement** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the instance. 1. Choose **Actions**, **Instance settings**, **Modify instance placement**. 1. For **Placement group**, do one of the following: + To add the instance to a placement group, choose the placement group. + To move the instance from one placement group to another, choose the placement group. + To remove the instance from the placement group, choose **None**. 1. Choose **Save**. ------ #### [ AWS CLI ] **To move an instance to a placement group** Use the following [modify-instance-placement](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-placement.html) command. ``` aws ec2 modify-instance-placement \ --instance-id i-0123a456700123456 \ --group-name MySpreadGroup ``` **To remove an instance from a placement group** Use the following [modify-instance-placement](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-placement.html) command. When you specify an empty string for the placement group name, this removes the instance from its current placement group. ``` aws ec2 modify-instance-placement \ --instance-id i-0123a456700123456 \ --group-name "" ``` ------ #### [ PowerShell ] **To move an instance to a placement group** Use the [Edit-EC2InstancePlacement](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstancePlacement.html) cmdlet with the name of the placement group. ``` Edit-EC2InstancePlacement ` -InstanceId i-0123a456700123456 ` -GroupName MySpreadGroup ``` **To remove an instance from a placement group** Use the [Edit-EC2InstancePlacement](https://docs.aws.amazon.com/powershell/latest/reference/items/Edit-EC2InstancePlacement.html) cmdlet with an empty string for the name of the placement group. ``` Edit-EC2InstancePlacement ` -InstanceId i-0123a456700123456 ` -GroupName "" ``` ------ # Delete a placement group If you need to replace a placement group or no longer need one, you can delete it. Before you can delete a placement group, it must contain no instances. You can terminate the instances, move them to another placement group, or remove them from the placement group. ------ #### [ Console ] **To delete a placement group** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Placement Groups**. 1. Select the placement group and choose **Actions**, **Delete**. 1. When prompted for confirmation, enter **Delete** and then choose **Delete**. ------ #### [ AWS CLI ] **To delete a placement group** Use the [delete-placement-group](https://docs.aws.amazon.com/cli/latest/reference/ec2/delete-placement-group.html) command. ``` aws ec2 delete-placement-group --group-name my-cluster ``` ------ #### [ PowerShell ] **To delete a placement group** Use the [Remove-EC2PlacementGroup](https://docs.aws.amazon.com/powershell/latest/reference/items/Remove-EC2PlacementGroup.html) cmdlet. ``` Remove-EC2PlacementGroup -GroupName my-cluster ``` ------ # Shared placement groups Placement group sharing allows you to influence the placement of interdependent instances that are owned by separate AWS accounts. An owner can share a placement group across multiple AWS accounts or within their organization. A participant can launch instances in a placement group that is shared with their account. A placement group owner can share a placement group with: + Specific AWS accounts inside or outside of its organization + An organizational unit inside its organization + Its entire organization You can use VPC peering to connect instances owned by separate AWS accounts and get the full latency benefits offered by shared cluster placement groups. **Topics** + [ ## Rules and limitations ](#share-placement-group-limitations) + [ ## Required permissions ](#share-placement-group-permissions) + [ ## Sharing across Availability Zones ](#share-placement-group-sharing-azs) + [ ## Placement group sharing ](#share-placement-group-share) + [ ## Placement group unsharing ](#share-placement-group-unshare) ## Rules and limitations The following rules and limitations apply when you share a placement group or when a placement group is shared with you. + To share a placement group, you must own it in your AWS account. You can't share a placement group that has been shared with you. + When you share a partition or spread placement group, the placement group limits do not change. A shared partition placement group supports a maximum of seven partitions per Availability Zone, and a shared spread placement group supports a maximum of seven running instances per Availability Zone. + To share a placement group with your organization or an organizational unit in your organization, you must enable sharing with AWS Organizations. For more information, see [Sharing your AWS resources](https://docs.aws.amazon.com/ram/latest/userguide/getting-started-sharing.html). + When using the AWS Management Console to launch an instance, you can select any placement groups that were shared with you. When using the AWS CLI to launch an instance, you must specify a shared placement group by ID, not by name. You can use the name of a placement group only if you're the owner of the shared placement group. + You are responsible for managing the instances owned by you in a shared placement group. + You can't view or modify instances and capacity reservations that are associated with a shared placement group but not owned by you. + The Amazon Resource Name (ARN) of a placement group contains the ID of the account that owns the placement group. You can use the account ID portion of a placement group ARN to identify the owner of a placement group that is shared with you. ## Required permissions To share a placement group, users must have permissions for following actions: + `ec2:PutResourcePolicy` + `ec2:DeleteResourcePolicy` ## Sharing across Availability Zones Share across AZs To ensure that resources are distributed across the Availability Zones for a Region, we independently map Availability Zones to names for each account. This could lead to Availability Zone naming differences across accounts. For example, the Availability Zone `us-east-1a` for your AWS account might not have the same location as `us-east-1a` for another AWS account. To specify the location of your Dedicated Hosts relative to your accounts, you must use the *Availability Zone ID* (AZ ID). The AZ ID is a unique and consistent identifier for an Availability Zone across all AWS accounts. For example, `use1-az1` is an Availability Zone ID for the `us-east-1` Region and it is the same location in every AWS account. For more information, see [AZ IDs](https://docs.aws.amazon.com/global-infrastructure/latest/regions/az-ids.html). ## Placement group sharing To share a placement group, you must add it to a resource share. A resource share is an AWS RAM resource that lets you share your resources across AWS accounts. A resource share specifies the resources to share, and the consumers with whom they are shared. If you are part of an organization in AWS Organizations sharing within your organization is enabled, consumers in your organization are granted access to the shared placement group. If the placement group is shared with an AWS account outside of your organization, the AWS account owner will receive an invitation to join the resource share. They can access the shared placement group after accepting the invitation. You can share a placement group across AWS accounts using AWS Resource Access Manager. For more information, see [Creating a resource share](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) in the *AWS RAM User Guide*. ## Placement group unsharing The placement group owner can unshare a shared placement group at any time. When you unshare a shared placement group, the following changes occur: + The AWS accounts with which a placement group was shared are no longer able to launch instances or reserve capacity. + Any instances running in a shared placement group are disassociated from the placement group, but they continue to run in your AWS account. + Any capacity reservations in a shared placement group are disassociated from the placement group, but remain available to you in your AWS account. For more information, see [Deleting a resource share](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-delete.html) in the *AWS RAM User Guide*. # Placement groups on AWS Outposts AWS Outposts is a fully managed service that extends AWS infrastructure, services, APIs, and tools to customer premises. By providing local access to AWS managed infrastructure, AWS Outposts enables customers to build and run applications on premises using the same programming interfaces as in AWS Regions, while using local compute and storage resources for lower latency and local data processing needs. An Outpost is a pool of AWS compute and storage capacity deployed at a customer site. AWS operates, monitors, and manages this capacity as part of an AWS Region. You can create placement groups on Outposts that you have created in your account. This allows you to spread out instances across underlying hardware on an Outpost at your site. You create and use placement groups on Outposts in the same way that you create and use placement groups in regular Availability Zones. When you create a placement group with a spread strategy on an Outpost, you can choose to have the placement group spread instances across hosts or racks. Spreading instances across hosts allows you to use a spread strategy with a single rack Outpost. **Considerations** + A rack level spread placement group can hold as many instances as you have racks in your Outpost deployment. + A host level spread placement group can hold as many instances as you have hosts in your Outpost deployment. **Prerequisite** You must have an Outpost installed at your site. For more information, see [Create an Outpost and order Outpost capacity](https://docs.aws.amazon.com/outposts/latest/userguide/order-outpost-capacity.html) in the *AWS Outposts User Guide*. **To use a placement group on an Outpost** 1. Create a subnet on the Outpost. For more information, see [Create a subnet](https://docs.aws.amazon.com/outposts/latest/userguide/launch-instance.html#create-subnet) in the *AWS Outposts User Guide*. 1. Create a placement group in the associated Region of the Outpost. If you create a placement group with a spread strategy, you can choose host or rack level spread to determine how the group will spread instances across the underlying hardware on your Outpost. For more information, see [Create a placement group for your EC2 instances](create-placement-group.md). 1. Launch an instance into the placement group. For **Subnet** choose the subnet that you created in Step 1, and for **Placement group name**, select the placement group that you created in Step 2. For more information, see [Launch an instance on the Outpost](https://docs.aws.amazon.com/outposts/latest/userguide/launch-instance.html#launch-instances) in the *AWS Outposts User Guide*. # Network maximum transmission unit (MTU) for your EC2 instance Network MTU The maximum transmission unit (MTU) of a network connection is the size, in bytes, of the largest permissible packet that can be passed over the connection. The larger the MTU of a connection, the more data that can be passed in a single packet. Ethernet frames consist of the packet, or the actual data you are sending, and the network overhead information that surrounds it. Ethernet frames can come in different formats, and the most common format is the standard Ethernet v2 frame format. It supports 1500 MTU, which is the largest Ethernet packet size supported over most of the internet. The maximum supported MTU for an instance depends on its instance type. All EC2 instance types support 1500 MTU. **Topics** + [ ## Jumbo frames (9001 MTU) ](#jumbo_frame_instances) + [ ## Path MTU Discovery ](#path_mtu_discovery) + [ # Set the MTU for your Amazon EC2 instances ](ec2-instance-mtu.md) + [ ## Troubleshoot ](#mtu-troubleshooting) ## Jumbo frames (9001 MTU) With jumbo frames, you can increase the payload size per packet, thereby increasing the percentage of the packet that is not packet overhead. With jumbo frames, fewer packets are needed to send the same amount of usable data. However, certain types of traffic are subject to the following maximum payloads: **MTU limit 1500 bytes** + Traffic over an internet gateway + Traffic over VPN connections + Traffic between AWS Regions, unless a transit gateway is used **MTU limit 8500 bytes** + Traffic over an inter-region VPC peering connection If packets are over their MTU limit, they are fragmented, or they are dropped if the `Don't Fragment` flag is set in the IP header. Jumbo frames should be used with caution for internet-bound traffic or any traffic that leaves a VPC. Packets are fragmented by intermediate systems, which slows down this traffic. To use jumbo frames inside a VPC and not slow traffic that's bound for outside the VPC, you can configure the MTU size by route, or use multiple elastic network interfaces with different MTU sizes and different routes. For instances that are collocated inside a cluster placement group, jumbo frames help to achieve the maximum network throughput possible, and they are recommended in this case. For more information, see [Placement groups for your Amazon EC2 instances](placement-groups.md). You can use jumbo frames for traffic between your VPCs and your on-premises networks over Direct Connect. For more information, and for how to verify Jumbo Frame capability, see [MTU for private virtual interfaces or transit virtual interfaces](https://docs.aws.amazon.com/directconnect/latest/UserGuide/WorkingWithVirtualInterfaces.html#set-jumbo-frames-vif.html) in the *Direct Connect User Guide*. All [current generation](https://docs.aws.amazon.com/ec2/latest/instancetypes/instance-types.html#current-gen-instances) instance types support jumbo frames. The following [previous generation](https://docs.aws.amazon.com/ec2/latest/instancetypes/instance-types.html#previous-gen-instances) instance types support jumbo frames: A1, C3, I2, M3, and R3. **Related resources** + For NAT gateways, see [NAT gateway basics ](https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-basics.html) in the *Amazon VPC User Guide*. + For transit gateways, see [Maximum transmission unit](https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-quotas.html#mtu-quotas) in the *Amazon VPC Transit Gateways User Guide*. + For Local Zones, see [Considerations](https://docs.aws.amazon.com/local-zones/latest/ug/how-local-zones-work.html#considerations) in the *AWS Local Zones User Guide*. + For AWS Wavelength, see [Maximum transmission unit](https://docs.aws.amazon.com/wavelength/latest/developerguide/how-wavelengths-work.html#mtu) in the *AWS Wavelength User Guide*. + For Outposts see [Service link maximum transmission unit requirements](https://docs.aws.amazon.com/outposts/latest/userguide/region-connectivity.html#sl-max-mtu-requirements) in the *AWS Outposts User Guide*. ## Path MTU Discovery Path MTU Discovery (PMTUD) is used to determine the path MTU between two devices. The path MTU is the maximum packet size that's supported on the path between the originating host and the receiving host. When there is a difference in the MTU size in the network between two hosts, PMTUD enables the receiving host to respond to the originating host with an ICMP message. This ICMP message instructs the originating host to use the lowest MTU size along the network path and to resend the request. Without this negotiation, packet drop can occur because the request is too large for the receiving host to accept. For IPv4, when a host sends a packet that's larger than the MTU of the receiving host or that's larger than the MTU of a device along the path, the receiving host or device drops the packet, and then returns the following ICMP message: `Destination Unreachable: Fragmentation Needed and Don't Fragment was Set` (Type 3, Code 4). This instructs the transmitting host to split the payload into multiple smaller packets, and then retransmit them. The IPv6 protocol does not support fragmentation in the network. When a host sends a packet that's larger than the MTU of the receiving host or that's larger than the MTU of a device along the path, the receiving host or device drops the packet, and then returns the following ICMP message: `ICMPv6 Packet Too Big (PTB)` (Type 2). This instructs the transmitting host to split the payload into multiple smaller packets, and then retransmit them. Connections made through some components, like NAT gateways and load balancers, are [automatically tracked](security-group-connection-tracking.md#automatic-tracking). This means that [security group tracking](security-group-connection-tracking.md) is automatically enabled for your outbound connection attempts. If connections are automatically tracked or if your security group rules allow inbound ICMP traffic, you can receive PMTUD responses. Note that ICMP traffic can be blocked even if the traffic is allowed at the security group level, such as if you have a network access control list entry that denies ICMP traffic to the subnet. **Important** Path MTU Discovery does not guarantee that jumbo frames will not be dropped by some routers. An internet gateway in your VPC will forward packets up to 1500 bytes only. 1500 MTU packets are recommended for internet traffic. For MTU rules over NAT gateways, see [Maximum transmission unit (MTU)](https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html#ngw-mtus) in the *Amazon VPC User Guide*. For MTU rules over Transit gateways, see [Maximum transmission unit (MTU)](https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-quotas.html#mtu-quotas) in the *AWS Transit Gateway User Guide*. # Set the MTU for your Amazon EC2 instances Set the MTU for your instances The maximum transmission unit (MTU) of a network connection is the size, in bytes, of the largest permissible packet that can be passed over the connection. All Amazon EC2 instances support standard frames (1500 MTU) and all current generation instance types support jumbo frames (9001 MTU). You can view the MTU for your Amazon EC2 instances, view the path MTU between your instance and another host, and configure your instances to use either standard or jumbo frames. **Topics** + [ ## Check the path MTU between two hosts ](#check_path_mtu) + [ ## Check the MTU for your instance ](#check_mtu) + [ ## Set the MTU for your instance ](#set_mtu) ## Check the path MTU between two hosts You can check the path MTU between your EC2 instance and another host. You can specify a DNS name or an IP address as the destination. If the destination is another EC2 instance, verify that its security group allows inbound UDP traffic. The procedure that you use depends on the operating system of the instance. ### Linux instances Run the **tracepath** command on your instance to check the path MTU between your EC2 instance and the specified destination. This command is part of the `iputils` package, which is available by default in many Linux distributions. This example checks the path MTU between the EC2 instance and `amazon.com`. ``` [ec2-user ~]$ tracepath amazon.com ``` In this example output, the path MTU is 1500. ``` 1?: [LOCALHOST] pmtu 9001 1: ip-172-31-16-1.us-west-1.compute.internal (172.31.16.1) 0.187ms pmtu 1500 1: no reply 2: no reply 3: no reply 4: 100.64.16.241 (100.64.16.241) 0.574ms 5: 72.21.222.221 (72.21.222.221) 84.447ms asymm 21 6: 205.251.229.97 (205.251.229.97) 79.970ms asymm 19 7: 72.21.222.194 (72.21.222.194) 96.546ms asymm 16 8: 72.21.222.239 (72.21.222.239) 79.244ms asymm 15 9: 205.251.225.73 (205.251.225.73) 91.867ms asymm 16 ... 31: no reply Too many hops: pmtu 1500 Resume: pmtu 1500 ``` ### Windows instances **To check path MTU using mturoute** 1. Download **mturoute.exe** to your EC2 instance from [https://elifulkerson.com/projects/mturoute.php](https://elifulkerson.com/projects/mturoute.php). 1. Open a Command Prompt window and change to the directory where you downloaded **mturoute.exe**. 1. Use the following command to check the path MTU between your EC2 instance and the specified destination. This example checks the path MTU between the EC2 instance and `www.elifulkerson.com`. ``` .\mturoute.exe www.elifulkerson.com ``` In this example output, the path MTU is 1500. ``` * ICMP Fragmentation is not permitted. * * Speed optimization is enabled. * * Maximum payload is 10000 bytes. * + ICMP payload of 1472 bytes succeeded. - ICMP payload of 1473 bytes is too big. Path MTU: 1500 bytes. ``` ## Check the MTU for your instance You can check the MTU value for your instance. Some instances are configured to use jumbo frames, and others are configured to use standard frame sizes. The procedure that you use depends on the operating system of the instance. ### Linux instances **To check the MTU setting on a Linux instance** Run the following **ip** command on your EC2 instance. If the primary network interface is not `eth0`, replace `eth0` with your network interface. ``` [ec2-user ~]$ ip link show eth0 ``` In this example output, *mtu 9001* indicates that the instance uses jumbo frames. ``` 2: eth0: mtu 9001 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 02:90:c0:b7:9e:d1 brd ff:ff:ff:ff:ff:ff ``` ### Windows instances The procedure that you use depends on the driver on your instance. ------ #### [ ENA driver ] **Version 2.1.0 and later** To get the MTU value, use the following **Get-NetAdapterAdvancedProperty** command on your EC2 instance. Use the wildcard (asterisk) to get all Ethernet names. Check the output for the interface name `*JumboPacket`. A value of 9015 indicates that Jumbo frames are enabled. Jumbo frames are disabled by default. ``` Get-NetAdapterAdvancedProperty -Name "Ethernet*" ``` **Version 1.5 and earlier** To get the MTU value, use the following **Get-NetAdapterAdvancedProperty** command on your EC2 instance. Check the output for the interface name `MTU`. A value of 9001 indicates that Jumbo frames are enabled. Jumbo frames are disabled by default. ``` Get-NetAdapterAdvancedProperty -Name "Ethernet" ``` ------ #### [ Intel SRIOV 82599 driver ] To get the MTU value, use the following **Get-NetAdapterAdvancedProperty** command on your EC2 instance. Check the entry for the interface name `*JumboPacket`. A value of 9014 indicates that Jumbo frames are enabled. (Note that the MTU size includes the header and the payload.) Jumbo frames are disabled by default. ``` Get-NetAdapterAdvancedProperty -Name "Ethernet" ``` ------ #### [ AWS PV driver ] To get the MTU value, use the following command on your EC2 instance. The name of the interface can vary. In the output, look for an entry with the name "Ethernet," "Ethernet 2," or "Local Area Connection". You'll need the interface name to enable or disable jumbo frames. A value of 9001 indicates that Jumbo frames are enabled. ``` netsh interface ipv4 show subinterface ``` ------ ## Set the MTU for your instance You might want to use jumbo frames for network traffic within your VPC and standard frames for internet traffic. Whatever your use case, we recommend that you verify that your instance behaves as expected. The procedure that you use depends on the operating system of the instance. ### Linux instances **To set the MTU value on a Linux instance** 1. Run the following **ip** command on your instance. It sets the desired MTU value to 1500, but you could use 9001 instead. If the primary network interface is not `eth0`, replace `eth0` with the actual network interface. ``` [ec2-user ~]$ sudo ip link set dev eth0 mtu 1500 ``` 1. (Optional) To persist your network MTU setting after a reboot, modify the following configuration files, based on your operating system type. + **Amazon Linux 2023** – Modify the `[Link]` section of the config file. The default config file is `/usr/lib/systemd/network/80-ec2.network`, or you can update any custom config file created in /run/systemd/network/, where the file name is *priority*-*interface*.network. For more information, see [Networking service](https://docs.aws.amazon.com/linux/al2023/ug/networking-service.html) in the Amazon Linux documentation. ``` MTUBytes=1500 ``` + **Amazon Linux 2** – Add the following line to the `/etc/sysconfig/network-scripts/ifcfg-eth0` file: ``` MTU=1500 ``` Add the following line to the `/etc/dhcp/dhclient.conf` file: ``` request subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-search, domain-name-servers, host-name, nis-domain, nis-servers, ntp-servers; ``` + **Other Linux distributions** – Consult their specific documentation. 1. (Optional) Reboot your instance and verify that the MTU setting is correct. ### Windows instances The procedure that you use depends on the driver on your instance. ------ #### [ ENA driver ] You can change the MTU using Device Manager or the **Set-NetAdapterAdvancedProperty** command on your instance. **Version 2.1.0 and later** Use the following command to enable jumbo frames. ``` Set-NetAdapterAdvancedProperty -Name "Ethernet" -RegistryKeyword "*JumboPacket" -RegistryValue 9015 ``` Use the following command to disable jumbo frames. ``` Set-NetAdapterAdvancedProperty -Name "Ethernet" -RegistryKeyword "*JumboPacket" -RegistryValue 1514 ``` **Version 1.5 and earlier** Use the following command to enable jumbo frames. ``` Set-NetAdapterAdvancedProperty -Name "Ethernet" -RegistryKeyword "MTU" -RegistryValue 9001 ``` Use the following command to disable jumbo frames. ``` Set-NetAdapterAdvancedProperty -Name "Ethernet" -RegistryKeyword "MTU" -RegistryValue 1500 ``` ------ #### [ Intel SRIOV 82599 driver ] You can change the MTU using Device Manager or the **Set-NetAdapterAdvancedProperty** command on your instance. Use the following command to enable jumbo frames. ``` Set-NetAdapterAdvancedProperty -Name "Ethernet" -RegistryKeyword "*JumboPacket" -RegistryValue 9014 ``` Use the following command to disable jumbo frames. ``` Set-NetAdapterAdvancedProperty -Name "Ethernet" -RegistryKeyword "*JumboPacket" -RegistryValue 1514 ``` ------ #### [ AWS PV driver ] You can change the MTU using the **netsh** command on your instance. You can't change the MTU using Device Manager. Use the following command to enable jumbo frames. ``` netsh interface ipv4 set subinterface "Ethernet" mtu=9001 ``` Use the following command to disable jumbo frames. ``` netsh interface ipv4 set subinterface "Ethernet" mtu=1500 ``` ------ ## Troubleshoot If you experience connectivity issues between your EC2 instance and an Amazon Redshift cluster when using jumbo frames, see [Queries appear to hang and sometimes fail to reach the cluster](https://docs.aws.amazon.com/redshift/latest/mgmt/troubleshooting-connections.html#connecting-drop-issues) in the *Amazon Redshift Management Guide*. # Virtual private clouds for your EC2 instances Virtual private clouds Amazon Virtual Private Cloud (Amazon VPC) enables you to define a virtual network in your own logically isolated area within the AWS cloud, known as a *virtual private cloud* or *VPC*. You can create AWS resources, such as Amazon EC2 instances, into the subnets of your VPC. Your VPC closely resembles a traditional network that you might operate in your own data center, with the benefits of using scalable infrastructure from AWS. You can configure your VPC; you can select its IP address range, create subnets, and configure route tables, network gateways, and security settings. You can connect instances in your VPC to the internet or to your own data center. **Topics** + [ ## Your default VPCs ](#default-vpcs) + [ ## Nondefault VPCs ](#create-nondefault-vpcs) + [ ## Internet access ](#access-internet-from-vpc) + [ ## Shared subnets ](#ec2-shared-VPC-subnets) + [ ## IPv6-only subnets ](#ec2-ipv6-only-subnets) ## Your default VPCs When you create your AWS account, we create a *default VPC* in each Region. A default VPC is a VPC that is already configured and ready for you to use. For example, there is a default subnet for each Availability Zone in each default VPC, an internet gateway attached to the VPC, and there's a route in the main route table that sends all traffic (0.0.0.0/0) to the internet gateway. You can modify the configuration of your default VPCs as needed. For example, you can add subnets and route tables. ![\[We create a default VPC in each Region, with a default subnet in each Availability Zone.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/default-vpc.png) ## Nondefault VPCs Instead of using a default VPC for your resources, you can create your own VPC, as described in [Create a VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html) in the *Amazon VPC User Guide*. Here are some things to consider when creating a VPC for your EC2 instances. + You can use the default suggestion for the IPv4 CIDR block or enter the CIDR block required by your application or network. + To ensure high availability, create subnets in multiple Availability Zones. + If your instances must be accessible from the internet, do one of the following: + If your instances can be in a public subnet, add public subnets. Keep both DNS options enabled. You can optionally add private subnets now or later on. + If your instances must be in a private subnet, add only private subnets. You can add a NAT gateway to provide internet access to instances in the private subnets. If your instances send or receive a significant volume of traffic across Availability Zones, create a NAT gateway in each Availability Zone. Otherwise, you can create a NAT gateway in just one of the Availability Zones and launch instances that send or receive cross-zone traffic in the same Availability Zone as the NAT gateway. ## Internet access Instances launched into a default subnet in a default VPC have access to the internet, as default VPCs are configured to assign public IP addresses and DNS hostnames, and the main route table is configured with a route to an internet gateway attached to the VPC. For instances that you launch in nondefault subnets and VPCs, you can use one of the following options to ensure that the instances that you launch in these subnets have access to the internet: + Configure an internet gateway. For more information, see [Connect to the internet using an internet gateway](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html) in the *Amazon VPC User Guide*. + Configure a public NAT gateway. For more information, see [Access the internet from a private subnet](https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-scenarios.html#public-nat-internet-access) in the *Amazon VPC User Guide*. ## Shared subnets When launching EC2 instances into shared VPC subnets, note the following: + Participants can run instances in a shared subnet by specifying the ID of the shared subnet. Participants must own any network interfaces that they specify. + Participants can start, stop, terminate, and describe instances that they've created in a shared subnet. Participants can't start, stop, terminate, or describe instances that the VPC owner created in the shared subnet. + VPC owners can't start, stop, terminate, or describe instances created by participants in a shared subnet. + Participants can connect to an instance in a shared subnet using EC2 Instance Connect Endpoint. The participant must create the EC2 Instance Connect Endpoint in the shared subnet. Participants can't use an EC2 Instance Connect Endpoint that the VPC owner created in the shared subnet. For information about shared Amazon EC2 resources, see the following: + [Manage AMI sharing with an organization or OU](share-amis-org-ou-manage.md) + [Shared Capacity Reservations](capacity-reservation-sharing.md) + [Shared placement groups](share-placement-group.md) + [Cross-account Amazon EC2 Dedicated Host sharing](dh-sharing.md) For more information about shared subnets, see [Share your VPC with other accounts](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-sharing.html) in the *Amazon VPC User Guide*. ## IPv6-only subnets An EC2 instance launched in an IPv6-only subnet receives an IPv6 address but not an IPv4 address. Any instances that you launch into an IPv6-only subnet must be [Nitro-based instances](instance-types.md#instance-hypervisor-type). # Secondary Networks Secondary Networks Secondary Networks are virtual networks purpose-built for specialized networking use cases. These networks are logically isolated within partitions of the AWS cloud. You can create resources such as secondary subnets within your secondary network. Secondary networks are tightly coupled with Amazon VPCs, such that select instances are multi-homed and launched into both a VPC and a Secondary Network. Secondary Networks are currently available for select instance types and through capacity reservations with long term commitments. Please reach out to your account team for more information if you think secondary networks may be beneficial for your workload. **Topics** + [ ## What are Secondary Networks? ](#secondary-networks-overview) + [ ## Key concepts ](#secondary-networks-concepts) + [ ## Architecture ](#secondary-networks-architecture) + [ ## Additional considerations ](#secondary-networks-considerations) + [ ## Getting started ](#secondary-networks-getting-started) + [ ## Managing Secondary Network resources ](#secondary-networks-managing-resources) + [ ## Network design best practices ](#secondary-networks-best-practices) + [ ## Troubleshooting ](#secondary-networks-troubleshooting) + [ ## Quotas and limits ](#secondary-networks-quotas-limits) ## What are Secondary Networks? Secondary Networks provide a logical, isolated network, used in combination with a VPC network, such that instances are multi-homed into two independent networks. The benefits of secondary networks include: + High-performance networking for specialized use cases and protocols such as east-west connectivity for ML workloads + Multi-tenant support with logical isolation + Instances seamlessly integrate with VPCs and AWS services ## Key concepts Secondary Network A regional networking construct that provides a logical Layer 3 network with an IPv4 CIDR block (ranging from /28 to /12). Secondary Networks operate independently from VPCs on physically partitioned network infrastructure. Secondary Subnet An Availability Zone-specific construct within a Secondary Network, similar to VPC subnets. Secondary Subnets support CIDR blocks ranging from /28 to /12. Secondary Interface Network interfaces attached to secondary network cards, providing east-west connectivity within secondary subnets. These interfaces are physically and logically separate from Elastic Network Interfaces (ENIs). ## Architecture EC2 instances that support Secondary Networks are multi-homed, meaning they can communicate within both a VPC and a Secondary Network simultaneously: + **VPC**: Provides north-south TCP/IP connectivity to AWS services, storage, databases, networking services, and the internet + **Secondary Network**: Provides east-west connectivity between supported specialized instances ## Additional considerations + Secondary interfaces are managed through RunInstances and they cannot be independently created or deleted. + Secondary interfaces cannot be attached/detached once the instance is launched. + Secondary interfaces IP addresses cannot be changed once launched. + VPC features such as Security Groups, NACLs, Flow Logs are not supported in Secondary Networks. ## Getting started ### Prerequisites Before launching instances with Secondary Networks, ensure you have also configured your VPC in the targeted region and a Subnet in the targeted availability zone of your EC2 capacity. ### Step 1: Create a Secondary Network Create a Secondary Network in the same region as your VPC. This is a regional resource that provides logical isolation for your RDMA traffic. ``` aws ec2 create-secondary-network \ --network-type rdma \ --ipv4-cidr-block 172.31.0.0/16 \ --region us-east-2 ``` **Parameters:** + `--network-type`: Network type (currently only rdma is supported) + `--ipv4-cidr-block`: IPv4 CIDR block between /28 and /12 + `--region`: AWS Region (US-East-2) **Note** **Best Practice**: Choose a CIDR block that does not overlap with your VPC CIDR to simplify routing at the instance level. ### Step 2: Create a Secondary Subnet Create a Secondary Subnet in the same availability zone as your VPC subnet. This is an AZ-specific resource. ``` aws ec2 create-secondary-subnet \ --secondary-network-id sn-1234567890abcdef0 \ --ipv4-cidr-block 172.31.24.0/24 \ --availability-zone us-east-2a ``` **Note** **IP Address Reservation**: Like VPC subnets, Amazon reserves the first 4 IP addresses and the last IP address in each Secondary Subnet for internal use. ### Step 3: Launch an instance Launch an instance into both your VPC subnet and Secondary Subnet. The instance will be multi-homed with connectivity to both networks. ``` aws ec2 run-instances \ --image-id ami-12345678 \ --count 1 \ --instance-type \ --key-name MyKeyPair \ --instance-market-options '{"MarketType": "capacity-block"}' \ --capacity-reservation-specification '{"CapacityReservationTarget": \ {"CapacityReservationId": "cr-1234567890abcdef0"}}' \ --network-interfaces \ "NetworkCardIndex=0,DeviceIndex=0,Groups=sg-12345678,\ SubnetId=subnet-0987654321fedcba0,InterfaceType=interface" \ --secondary-interfaces \ "NetworkCardIndex=1,DeviceIndex=0,SecondarySubnetId=ss-98765421yxz,\ InterfaceType=secondary,PrivateIpAddressCount=1,DeleteOnTermination=true", \ "NetworkCardIndex=2,DeviceIndex=0,SecondarySubnetId=ss-98765421yxz,\ InterfaceType=secondary,PrivateIpAddressCount=1,DeleteOnTermination=true", \ "NetworkCardIndex=3,DeviceIndex=0,SecondarySubnetId=ss-98765421yxz,\ InterfaceType=secondary,PrivateIpAddressCount=1,DeleteOnTermination=true", \ "NetworkCardIndex=4,DeviceIndex=0,SecondarySubnetId=ss-98765421yxz,\ InterfaceType=secondary,PrivateIpAddressCount=1,DeleteOnTermination=true", \ "NetworkCardIndex=5,DeviceIndex=0,SecondarySubnetId=ss-98765421yxz,\ InterfaceType=secondary,PrivateIpAddressCount=1,DeleteOnTermination=true", \ "NetworkCardIndex=6,DeviceIndex=0,SecondarySubnetId=ss-98765421yxz,\ InterfaceType=secondary,PrivateIpAddressCount=1,DeleteOnTermination=true", \ "NetworkCardIndex=7,DeviceIndex=0,SecondarySubnetId=ss-98765421yxz,\ InterfaceType=secondary,PrivateIpAddressCount=1,DeleteOnTermination=true", \ "NetworkCardIndex=8,DeviceIndex=0,SecondarySubnetId=ss-98765421yxz,\ InterfaceType=secondary,PrivateIpAddressCount=1,DeleteOnTermination=true" ``` **Key Parameters:** + `--network-interfaces`: Specifies the primary Nitro ENI for VPC connectivity (Network Card Index 0) + `--secondary-interfaces`: Specifies 8 secondary interfaces for east-west connectivity within Secondary Subnets (Network Card Indices 1-8) + `InterfaceType=secondary`: Indicates a Secondary Interface ## Managing Secondary Network resources ### Describing Secondary Networks View details about your Secondary Networks: ``` aws ec2 describe-secondary-networks \ --secondary-network-id sn-1234567890abcdef0 ``` ### Describing Secondary Subnets View details about your Secondary Subnets: ``` aws ec2 describe-secondary-subnets \ --secondary-subnet-id ss-98765421yxz ``` ### Describing Secondary Interfaces View details about Secondary Network Interfaces attached to your instances: ``` aws ec2 describe-secondary-interfaces \ --filters "Name=attachment.instance-id,Values=i-1234567890abcdef0" ``` ### Deleting resources Delete a Secondary Subnet: ``` aws ec2 delete-secondary-subnet \ --secondary-subnet-id ss-98765421yxz ``` Delete a Secondary Network: ``` aws ec2 delete-secondary-network \ --secondary-network-id sn-1234567890abcdef0 ``` **Important** You must terminate all instances and delete all Secondary Subnets before deleting a Secondary Network. ## Network design best practices ### CIDR planning **Avoid Overlapping CIDRs**: While Secondary Networks are physically isolated from VPCs, using non-overlapping CIDR blocks simplifies routing configuration at the instance operating system level. **Note** Amazon reserves 5 IP addresses per subnet. ### Traffic segregation **Segregate by Secondary Network**: Create separate Secondary Networks for different projects, teams, or security boundaries. Secondary Networks provide logical isolation between instances. Instances cannot communicate across different Secondary Networks. **Use Multiple Subnets**: Within a Secondary Network, use multiple Secondary Subnets to segment traffic by GPU index, availability zone, or workload type. As an example, a common architecture pattern is to deploy a single secondary network with 4 or 8 secondary subnets, where each secondary subnet is aligned to a group of GPUs of common indices. ## Troubleshooting ### Instance launch failures **Problem**: Instance fails to launch with Secondary Network interfaces. **Solutions**: + Verify that your AMI includes proper driver support + Ensure your Secondary Subnet has sufficient available IP addresses + Confirm that your capacity reservation is in "active" state + Check that your Secondary Subnet is in the same availability zone as your VPC subnet ### Connectivity issues **Problem**: Unable to establish RDMA connectivity between instances. **Solutions**: + Verify that all instances are in the same Secondary Network and Secondary Subnet + Check that secondary interface drivers are properly loaded on the instance + Ensure that your application is binding to the correct network interfaces + Instances within the same secondary subnet are reachable via direct routes. Cross subnet communication is available via a static route vended via DHCP. ### API errors **Problem**: API calls for Secondary Network and Secondary Subnet operations fail. **Solutions**: + Verify IAM permissions for ec2:CreateSecondaryNetwork, ec2:CreateSecondarySubnet, etc. + Check that CIDR blocks are within the supported range (/28 to /12) + Verify that you're using the correct region and availability zone ## Quotas and limits To request quota increases, use the AWS Service Quotas or contact AWS Support. **Secondary Networks quotas and limits** | Resource | Limit | Adjustable | | --- | --- | --- | | Secondary Networks per region | 5 | Yes | | Secondary Subnets per Secondary Network | 200 | Yes | | CIDR block size | /28 to /12 | No |