# View web interfaces hosted on Amazon EMR clusters **Important** It is possible to configure a custom security group to allow inbound access to these web interfaces. Keep in mind that any port on which you allow inbound traffic represents a potential security vulnerability. Carefully review custom security groups to ensure that you minimize vulnerabilities. For more information, see [Control network traffic with security groups for your Amazon EMR cluster](emr-security-groups.md). Hadoop and other applications that you install on your EMR cluster publish user interfaces as web sites that are hosted on the primary node. For security reasons, when using Amazon EMR Managed Security Groups, these web sites are only available on the primary node's local web server. For that reason, you need to connect to the primary node to view the web interfaces. For more information, see [Connect to the Amazon EMR cluster primary node using SSH](emr-connect-master-node-ssh.md). Hadoop also publishes user interfaces as web sites hosted on the core and task nodes. These web sites are also only available on local web servers on the nodes. The following table lists web interfaces that you can view on cluster instances. These Hadoop interfaces are available on all clusters. For the master instance interfaces, replace *master-public-dns-name* with the **Master public DNS** listed on the cluster **Summary** tab in the Amazon EMR console. For core and task instance interfaces, replace *coretask-public-dns-name* with the **Public DNS name** listed for the instance. To find an instance's **Public DNS name**, in the Amazon EMR console, choose your cluster from the list, choose the **Hardware** tab, choose the **ID** of the instance group that contains the instance you want to connect to, and then note the **Public DNS name** listed for the instance. | Name of interface | URI | | --- | --- | | Flink history server (EMR version 5.33 and later) | http://master-public-dns-name:8082/ | | Ganglia | http://master-public-dns-name/ganglia/ | | Hadoop HDFS NameNode (EMR version pre-6.x) | http://master-public-dns-name:50070/ | | Hadoop HDFS NameNode (EMR version pre-6.x) | https://master-public-dns-name:50470/ | | Hadoop HDFS NameNode (EMR version 6.x) | http://master-public-dns-name:9870/ | | Hadoop HDFS NameNode (EMR version 6.x) | https://master-public-dns-name:9871/ | | Hadoop HDFS DataNode(EMR version pre-6.x) | http://coretask-public-dns-name:50075/ | | Hadoop HDFS DataNode (EMR version pre-6.x) | https://coretask-public-dns-name:50475/ | | Hadoop HDFS DataNode(EMR version 6.x) | http://coretask-public-dns-name:9864/ | | Hadoop HDFS DataNode (EMR version 6.x) | https://coretask-public-dns-name:9865/ | | HBase | http://master-public-dns-name:16010/ | | Hue | http://master-public-dns-name:8888/ | | JupyterHub | https://master-public-dns-name:9443/ | | Livy | http://master-public-dns-name:8998/ | | Spark HistoryServer | http://master-public-dns-name:18080/ | | Tez | http://master-public-dns-name:8080/tez-ui | | YARN NodeManager | http://coretask-public-dns-name:8042/ | | YARN ResourceManager | http://master-public-dns-name:8088/ | | Zeppelin | http://master-public-dns-name:8890/ | Because there are several application-specific interfaces available on the primary node that are not available on the core and task nodes, the instructions in this document are specific to the Amazon EMR primary node. Accessing the web interfaces on the core and task nodes can be done in the same manner as you would access the web interfaces on the primary node. There are several ways you can access the web interfaces on the primary node. The easiest and quickest method is to use SSH to connect to the primary node and use the text-based browser, Lynx, to view the web sites in your SSH client. However, Lynx is a text-based browser with a limited user interface that cannot display graphics. The following example shows how to open the Hadoop ResourceManager interface using Lynx (Lynx URLs are also provided when you log into the primary node using SSH). ``` lynx http://ip-###-##-##-###.us-west-2.compute.internal:8088/ ``` There are two remaining options for accessing web interfaces on the primary node that provide full browser functionality. Choose one of the following: + Option 1 (recommended for more technical users): Use an SSH client to connect to the primary node, configure SSH tunneling with local port forwarding, and use an Internet browser to open web interfaces hosted on the primary node. This method allows you to configure web interface access without using a SOCKS proxy. + Option 2 (recommended for new users): Use an SSH client to connect to the primary node, configure SSH tunneling with dynamic port forwarding, and configure your Internet browser to use an add-on such as FoxyProxy for Firefox or SwitchyOmega for Chrome to manage your SOCKS proxy settings. This method lets you automatically filter URLs based on text patterns and limit the proxy settings to domains that match the form of the primary node's DNS name. For more information about how to configure FoxyProxy for Firefox and Google Chrome, see [Option 2, part 2: Configure proxy settings to view websites hosted on the Amazon EMR cluster primary node](emr-connect-master-node-proxy.md). **Note** If you modify the port where an application runs via cluster configuration, the hyperlink to the port will not update in the Amazon EMR console. This is because the console doesn't have the functionality to read `server.port` configuration. With Amazon EMR version 5.25.0 or later, you can access Spark history server UI from the console without setting up a web proxy through an SSH connection. For more information, see [One-click access to persistent Spark history server](https://docs.aws.amazon.com/emr/latest/ManagementGuide/app-history-spark-UI.html). **Topics** + [Option 1: Set up an SSH tunnel to the Amazon EMR primary node using local port forwarding](emr-ssh-tunnel-local.md) + [Option 2, part 1: Set up an SSH tunnel to the primary node using dynamic port forwarding](emr-ssh-tunnel.md) + [Option 2, part 2: Configure proxy settings to view websites hosted on the Amazon EMR cluster primary node](emr-connect-master-node-proxy.md) # Option 1: Set up an SSH tunnel to the Amazon EMR primary node using local port forwarding To connect to the local web server on the primary node, you create an SSH tunnel between your computer and the primary node. This is also known as *port forwarding*. If you do not wish to use a SOCKS proxy, you can set up an SSH tunnel to the primary node using local port forwarding. With local port forwarding, you specify unused local ports that are used to forward traffic to specific remote ports on the primary node's local web server. Setting up an SSH tunnel using local port forwarding requires the public DNS name of the primary node and your key pair private key file. For information about how to locate the master public DNS name, see [Retrieve the public DNS name of the primary node](emr-connect-master-node-ssh.md#emr-connect-master-dns). For more information about accessing your key pair, see [Amazon EC2 key pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*. For more information about the sites you might want to view on the primary node, see [View web interfaces hosted on Amazon EMR clusters](emr-web-interfaces.md). ## Set up an SSH tunnel to the primary node using local port forwarding with OpenSSH **To set up an SSH tunnel using local port forwarding in terminal** 1. Ensure you've allowed inbound SSH traffic. For instructions, see [Before you connect to Amazon EMR: Authorize inbound traffic](emr-connect-ssh-prereqs.md). 1. Open a terminal window. On Mac OS X, choose **Applications > Utilities > Terminal**. On other Linux distributions, terminal is typically found at **Applications > Accessories > Terminal**. 1. Type the following command to open an SSH tunnel on your local machine. This example command accesses the ResourceManager web interface by forwarding traffic on local port 8157 (a randomly chosen unused local port) to port 8088 on the master node's local web server.  In the command, replace *\$1/mykeypair.pem* with the location and file name of your `.pem` file and replace *ec2-\$1\$1\$1-\$1\$1-\$1\$1-\$1\$1\$1.compute-1.amazonaws.com* with the master public DNS name of your cluster. To access a different web interface, replace `8088` with the appropriate port number. For example, replace `8088` with `8890` for the Zeppelin interface. ``` ssh -i ~/mykeypair.pem -N -L 8157:ec2-###-##-##-###.compute-1.amazonaws.com:8088 hadoop@ec2-###-##-##-###.compute-1.amazonaws.com ``` `-L` signifies the use of local port forwarding which allows you to specify a local port used to forward data to the identified remote port on the master node's local web server. After you issue this command, the terminal remains open and does not return a response.  1. To open the ResourceManager web interface in your browser, type `http://localhost:8157/` in the address bar. 1. When you are done working with the web interfaces on the primary node, close the terminal windows. # Option 2, part 1: Set up an SSH tunnel to the primary node using dynamic port forwarding To connect to the local web server on the primary node, you create an SSH tunnel between your computer and the primary node. This is also known as *port forwarding*. If you create your SSH tunnel using dynamic port forwarding, all traffic routed to a specified unused local port is forwarded to the local web server on the primary node. This creates a SOCKS proxy. You can then configure your Internet browser to use an add-on such as FoxyProxy or SwitchyOmega to manage your SOCKS proxy settings. Using a proxy management add-on allows you to automatically filter URLs based on text patterns and to limit the proxy settings to domains that match the form of the primary node's public DNS name. The browser add-on automatically handles turning the proxy on and off when you switch between viewing websites hosted on the primary node, and those on the Internet. Before you begin, you need the public DNS name of the primary node and your key pair private key file. For information about how to locate the primary public DNS name, see [Retrieve the public DNS name of the primary node](emr-connect-master-node-ssh.md#emr-connect-master-dns). For more information about accessing your key pair, see [Amazon EC2 key pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*. For more information about the sites you might want to view on the primary node, see [View web interfaces hosted on Amazon EMR clusters](emr-web-interfaces.md). ## Set up an SSH tunnel to the primary node using dynamic port forwarding with OpenSSH **To set up an SSH tunnel using dynamic port forwarding with OpenSSH** 1. Ensure you've allowed inbound SSH traffic. For instructions, see [Before you connect to Amazon EMR: Authorize inbound traffic](emr-connect-ssh-prereqs.md). 1. Open a terminal window. On Mac OS X, choose **Applications > Utilities > Terminal**. On other Linux distributions, terminal is typically found at **Applications > Accessories > Terminal**. 1. Type the following command to open an SSH tunnel on your local machine. Replace *\$1/mykeypair.pem* with the location and file name of your `.pem` file, replace *8157* with an unused, local port number, and replace *ec2-\$1\$1\$1-\$1\$1-\$1\$1-\$1\$1\$1.compute-1.amazonaws.com* with the primary public DNS name of your cluster. ``` 1. ssh -i ~/mykeypair.pem -N -D 8157 hadoop@ec2-###-##-##-###.compute-1.amazonaws.com ``` After you issue this command, the terminal remains open and does not return a response. **Note** `-D` signifies the use of dynamic port forwarding which allows you to specify a local port used to forward data to all remote ports on the primary node's local web server. Dynamic port forwarding creates a local SOCKS proxy listening on the port specified in the command. 1. After the tunnel is active, configure a SOCKS proxy for your browser. For more information, see [Option 2, part 2: Configure proxy settings to view websites hosted on the Amazon EMR cluster primary node](emr-connect-master-node-proxy.md). 1. When you are done working with the web interfaces on the primary node, close the terminal window. ## Set up an SSH tunnel using dynamic port forwarding with the AWS CLI You can create an SSH connection with the primary node using the AWS CLI on Windows and on Linux, Unix, and Mac OS X. If you are using the AWS CLI on Linux, Unix, or Mac OS X, you must set permissions on the `.pem` file as shown in [To configure the key pair private key file permissions](emr-connect-master-node-ssh.md#emr-keypair-file-permission-config). If you are using the AWS CLI on Windows, PuTTY must appear in the path environment variable or you may receive an error such as OpenSSH or PuTTY not available. **To set up an SSH tunnel using dynamic port forwarding with the AWS CLI** 1. Ensure you've allowed inbound SSH traffic. For instructions, see [Before you connect to Amazon EMR: Authorize inbound traffic](emr-connect-ssh-prereqs.md). 1. Create an SSH connection with the primary node as shown in [Connect to the primary node using the AWS CLI](emr-connect-master-node-ssh.md#emr-connect-cli). 1. To retrieve the cluster identifier, type: ``` 1. aws emr list-clusters ``` The output lists your clusters including the cluster IDs. Note the cluster ID for the cluster to which you are connecting. ``` "Status": { "Timeline": { "ReadyDateTime": 1408040782.374, "CreationDateTime": 1408040501.213 }, "State": "WAITING", "StateChangeReason": { "Message": "Waiting after step completed" } }, "NormalizedInstanceHours": 4, "Id": "j-2AL4XXXXXX5T9", "Name": "AWS CLI cluster" ``` 1. Type the following command to open an SSH tunnel to the primary node using dynamic port forwarding. In the following example, replace *j-2AL4XXXXXX5T9* with the cluster ID and replace *\$1/mykeypair.key* with the location and file name of your `.pem` file (for Linux, Unix, and Mac OS X) or `.ppk` file (for Windows). ``` aws emr socks --cluster-id j-2AL4XXXXXX5T9 --key-pair-file ~/mykeypair.key ``` **Note** The socks command automatically configures dynamic port forwarding on local port 8157. Currently, this setting cannot be modified. 1. After the tunnel is active, configure a SOCKS proxy for your browser. For more information, see [Option 2, part 2: Configure proxy settings to view websites hosted on the Amazon EMR cluster primary node](emr-connect-master-node-proxy.md). 1. When you are done working with the web interfaces on the primary node, close the AWS CLI window. For more information on using Amazon EMR commands in the AWS CLI, see [https://docs.aws.amazon.com/cli/latest/reference/emr](https://docs.aws.amazon.com/cli/latest/reference/emr). ## Set up an SSH tunnel to the primary node using PuTTY Windows users can use an SSH client such as PuTTY to create an SSH tunnel to the primary node. Before connecting to the Amazon EMR primary node, you should download and install PuTTY and PuTTYgen. You can download these tools from the [PuTTY download page](http://www.chiark.greenend.org.uk/~sgtatham/putty/). PuTTY does not natively support the key pair private key file format (`.pem`) generated by Amazon EC2. You use PuTTYgen to convert your key file to the required PuTTY format (`.ppk`). You must convert your key into this format (`.ppk`) before attempting to connect to the primary node using PuTTY. For more information about converting your key, see [Converting your private key using PuTTYgen](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html) in the *Amazon EC2 User Guide*. **To set up an SSH tunnel using dynamic port forwarding using PuTTY** 1. Ensure you've allowed inbound SSH traffic. For instructions, see [Before you connect to Amazon EMR: Authorize inbound traffic](emr-connect-ssh-prereqs.md). 1. Double-click `putty.exe` to start PuTTY. You can also launch PuTTY from the Windows programs list. **Note** If you already have an active SSH session with the primary node, you can add a tunnel by right-clicking the PuTTY title bar and choosing **Change Settings**. 1. If necessary, in the **Category** list, choose **Session**. 1. In the **Host Name** field, type **hadoop@***MasterPublicDNS*. For example: **hadoop@***ec2-\$1\$1\$1-\$1\$1-\$1\$1-\$1\$1\$1.compute-1.amazonaws.com*. 1. In the **Category** list, expand **Connection > SSH**, and then choose **Auth**. 1. For **Private key file for authentication**, choose **Browse** and select the `.ppk` file that you generated. **Note** PuTTY does not natively support the key pair private key file format (`.pem`) generated by Amazon EC2. You use PuTTYgen to convert your key file to the required PuTTY format (`.ppk`). You must convert your key into this format (`.ppk`) before attempting to connect to the primary node using PuTTY. 1. In the **Category** list, expand **Connection > SSH**, and then choose **Tunnels**. 1. In the **Source port** field, type `8157` (an unused local port), and then choose **Add**. 1. Leave the **Destination** field blank. 1. Select the **Dynamic** and **Auto** options. 1. Choose **Open**. 1. Choose **Yes** to dismiss the PuTTY security alert. **Important** When you log in to the primary node, type `hadoop` if you are prompted for a user name. 1. After the tunnel is active, configure a SOCKS proxy for your browser. For more information, see [Option 2, part 2: Configure proxy settings to view websites hosted on the Amazon EMR cluster primary node](emr-connect-master-node-proxy.md). 1. When you are done working with the web interfaces on the primary node, close the PuTTY window. # Option 2, part 2: Configure proxy settings to view websites hosted on the Amazon EMR cluster primary node If you use an SSH tunnel with dynamic port forwarding, you must use a SOCKS proxy management add-on to control the proxy settings in your browser. Using a SOCKS proxy management tool allows you to automatically filter URLs based on text patterns and to limit the proxy settings to domains that match the form of the primary node's public DNS name. The browser add-on automatically handles turning the proxy on and off when you switch between viewing websites hosted on the primary node and those on the Internet. To manage your proxy settings, configure your browser to use an add-on such as FoxyProxy or SwitchyOmega. For more information about creating an SSH tunnel, see [Option 2, part 1: Set up an SSH tunnel to the primary node using dynamic port forwarding](emr-ssh-tunnel.md). For more information about the available web interfaces, see [View web interfaces hosted on Amazon EMR clusters](emr-web-interfaces.md). Include the following settings when you set up your proxy add-on: + Use **localhost** as the host address. + Use the same local port number that you selected to establish the SSH tunnel with the primary node in [Option 2, part 1: Set up an SSH tunnel to the primary node using dynamic port forwarding](emr-ssh-tunnel.md). For example, port *8157*. This port must also match the port number you use in PuTTY or any other terminal emulator you use to connect. + Specify the **SOCKS v5** protocol. SOCKS v5 lets you optionally set up user authorization. + **URL Patterns** The following URL patterns should be allow-listed and specified with a wildcard pattern type: + The **\$1ec2\$1.\$1compute\$1.amazonaws.com\$1** and **\$110\$1.amazonaws.com\$1** patterns to match the public DNS name of clusters in US regions. + The **\$1ec2\$1.compute\$1** and **\$110\$1.compute\$1** patterns to match the public DNS name of clusters in all other regions. + A **10.\$1** pattern to provide access to the JobTracker log files in Hadoop. Alter this filter if it conflicts with your network access plan. + The **\$1.ec2.internal\$1** and **\$1.compute.internal\$1** patterns to match the private (internal) DNS names of clusters in the `us-east-1` region and all other regions, respectively. ## Example: Configure FoxyProxy for Firefox The following example demonstrates a FoxyProxy Standard (version 7.5.1) configuration for Mozilla Firefox. FoxyProxy provides a set of proxy management tools. It lets you use a proxy server for URLs that match patterns corresponding to domains used by the Amazon EC2 instances in your Amazon EMR cluster. **To install and configure FoxyProxy using Mozilla Firefox** 1. In Firefox, go to [https://addons.mozilla.org/](https://addons.mozilla.org/), search for FoxyProxy Standard, and follow the instructions to add FoxyProxy to Firefox. 1. Using a text editor, create a JSON file named `foxyproxy-settings.json` from the following example configuration. ``` { "k20d21508277536715": { "active": true, "address": "localhost", "port": 8157, "username": "", "password": "", "type": 3, "proxyDNS": true, "title": "emr-socks-proxy", "color": "#0055E5", "index": 9007199254740991, "whitePatterns": [ { "title": "*ec2*.*compute*.amazonaws.com*", "active": true, "pattern": "*ec2*.*compute*.amazonaws.com*", "importedPattern": "*ec2*.*compute*.amazonaws.com*", "type": 1, "protocols": 1 }, { "title": "*ec2*.compute*", "active": true, "pattern": "*ec2*.compute*", "importedPattern": "*ec2*.compute*", "type": 1, "protocols": 1 }, { "title": "10.*", "active": true, "pattern": "10.*", "importedPattern": "http://10.*", "type": 1, "protocols": 2 }, { "title": "*10*.amazonaws.com*", "active": true, "pattern": "*10*.amazonaws.com*", "importedPattern": "*10*.amazonaws.com*", "type": 1, "protocols": 1 }, { "title": "*10*.compute*", "active": true, "pattern": "*10*.compute*", "importedPattern": "*10*.compute*", "type": 1, "protocols": 1 }, { "title": "*.compute.internal*", "active": true, "pattern": "*.compute.internal*", "importedPattern": "*.compute.internal*", "type": 1, "protocols": 1 }, { "title": "*.ec2.internal* ", "active": true, "pattern": "*.ec2.internal*", "importedPattern": "*.ec2.internal*", "type": 1, "protocols": 1 } ], "blackPatterns": [] }, "logging": { "size": 100, "active": false }, "mode": "patterns", "browserVersion": "68.12.0", "foxyProxyVersion": "7.5.1", "foxyProxyEdition": "standard" } ``` 1. Open the Firefox **Manage Your Extensions** page (go to **about:addons**, then choose **Extensions**). 1. Choose **FoxyProxy Standard**, then choose the more options button (the button that looks like an ellipsis). 1. Select **Options** from the dropdown. 1. Choose **Import Settings** from the left menu. 1. On the **Import Settings** page, choose **Import Settings** under **Import Settings from FoxyProxy 6.0\$1**, browse to the location of the `foxyproxy-settings.json` file you created, select the file, and choose **Open**. 1. Choose **OK** when prompted to overwrite the existing settings and save your new configuration. ## Example: Configure SwitchyOmega for chrome The following example demonstrates how to set up the SwitchyOmega extension for Google Chrome. SwitchyOmega lets you configure, manage, and switch between multiple proxies. **To install and configure SwitchyOmega using Google Chrome** 1. Go to [https://chrome.google.com/webstore/category/extensions](https://chrome.google.com/webstore/category/extensions), search for **Proxy SwitchyOmega**, and add it to Chrome. 1. Choose **New profile** and enter `emr-socks-proxy` as the profile name. 1. Choose **PAC profile** and then **Create**. [Proxy Auto-Configuration (PAC)](https://developer.mozilla.org/en-US/docs/Web/HTTP/Proxy_servers_and_tunneling/Proxy_Auto-Configuration_(PAC)_file) files help you define an allow list for browser requests that should be forwarded to a web proxy server. 1. In the **PAC Script** field, replace the contents with the following script that defines which URLs should be forwarded through your web proxy server. If you specified a different port number when you set up your SSH tunnel, replace *8157* with your port number. ``` function FindProxyForURL(url, host) { if (shExpMatch(url, "*ec2*.*compute*.amazonaws.com*")) return 'SOCKS5 localhost:8157'; if (shExpMatch(url, "*ec2*.compute*")) return 'SOCKS5 localhost:8157'; if (shExpMatch(url, "http://10.*")) return 'SOCKS5 localhost:8157'; if (shExpMatch(url, "*10*.compute*")) return 'SOCKS5 localhost:8157'; if (shExpMatch(url, "*10*.amazonaws.com*")) return 'SOCKS5 localhost:8157'; if (shExpMatch(url, "*.compute.internal*")) return 'SOCKS5 localhost:8157'; if (shExpMatch(url, "*ec2.internal*")) return 'SOCKS5 localhost:8157'; return 'DIRECT'; } ``` 1. Under **Actions**, choose **Apply changes** to save your proxy settings. 1. On the Chrome toolbar, choose SwitchyOmega and select the `emr-socks-proxy` profile. ## Access a web interface in the browser To open a web interface, enter the public DNS name of your primary or core node followed by the port number for your chosen interface into your browser address bar. The following example shows the URL you would enter to connect to the Spark HistoryServer. ``` http://master-public-dns-name:18080/ ``` For instructions on retrieving the public DNS name of a node, see [Retrieve the public DNS name of the primary node](emr-connect-master-node-ssh.md#emr-connect-master-dns). For a complete list of web interface URLs, see [View web interfaces hosted on Amazon EMR clusters](emr-web-interfaces.md).