# Managing job runs with the AWS CLI
This topic covers how to manage job runs with the AWS Command Line Interface (AWS CLI). It goes into detail regarding properties, like security parameters, the driver, and various override settings. It also includes subtopics that cover various ways to configure logging.
**Topics**
+ [Options for configuring a job run](#emr-eks-jobs-parameters)
+ [Configure a job run to use Amazon S3 logs](emr-eks-jobs-s3.md)
+ [Configure a job run to use Amazon CloudWatch Logs](emr-eks-jobs-cloudwatch.md)
+ [List job runs](#emr-eks-jobs-list)
+ [Describe a job run](#emr-eks-jobs-describe)
+ [Cancel a job run](#emr-eks-jobs-cancel)
## Options for configuring a job run
Use the following options to configure job run parameters:
+ `--execution-role-arn`: You must provide an IAM role that is used for running jobs. For more information, see [Using job execution roles with Amazon EMR on EKS](iam-execution-role.md).
+ `--release-label`: You can deploy Amazon EMR on EKS with Amazon EMR versions 5.32.0 and 6.2.0 and later. Amazon EMR on EKS is not supported in previous Amazon EMR release versions. For more information, see [Amazon EMR on EKS releases](emr-eks-releases.md).
+ `--job-driver`: Job driver is used to provide input on the main job. This is a union type field where you can only pass one of the values for the job type that you want to run. Supported job types include:
+ Spark submit jobs - Used to run a command through Spark submit. You can use this job type to run Scala, PySpark, SparkR, SparkSQL and any other supported jobs through Spark Submit. This job type has the following parameters:
+ Entrypoint - This is the HCFS (Hadoop compatible file system) reference to the main jar/py file you want to run.
+ EntryPointArguments - This is an array of arguments you want to pass to your main jar/py file. You should handle reading these parameters using your entrypoint code. Each argument in the array should be separated by a comma. EntryPointArguments cannot contain brackets or parentheses, such as (), \$1\$1, or [].
+ SparkSubmitParameters - These are the additional spark parameters you want to send to the job. Use this parameter to override default Spark properties such as driver memory or number of executors like —conf or —class. For additional information, see [Launching Applications with spark-submit](https://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit).
+ Spark SQL jobs - Used to run a SQL query file through Spark SQL. You can use this job type to run SparkSQL jobs. This job type has the following parameters:
+ Entrypoint - This is the HCFS (Hadoop compatible file system) reference to the SQL query file you want to run.
For a list of additional Spark parameters you can use for a Spark SQL job, see [Running Spark SQL scripts through the StartJobRun API](emr-eks-jobs-spark-sql-parameters.md).
+ `--configuration-overrides`: You can override the default configurations for applications by supplying a configuration object. You can use a shorthand syntax to provide the configuration or you can reference the configuration object in a JSON file. Configuration objects consist of a classification, properties, and optional nested configurations. Properties consist of the settings you want to override in that file. You can specify multiple classifications for multiple applications in a single JSON object. The configuration classifications that are available vary by Amazon EMR release version. For a list of configuration classifications that are available for each release version of Amazon EMR, see [Amazon EMR on EKS releases](emr-eks-releases.md).
If you pass the same configuration in an application override and in Spark submit parameters, the Spark submit parameters take precedence. The complete configuration priority list follows, in order of highest priority to lowest priority.
+ Configuration supplied when creating `SparkSession`.
+ Configuration supplied as part of `sparkSubmitParameters` using `—conf`.
+ Configuration provided as part of application overrides.
+ Optimized configurations chosen by Amazon EMR for the release.
+ Default open source configurations for the application.
To monitor job runs using Amazon CloudWatch or Amazon S3, you must provide the configuration details for CloudWatch. For more information, see [Configure a job run to use Amazon S3 logs](emr-eks-jobs-s3.md) and [Configure a job run to use Amazon CloudWatch Logs](emr-eks-jobs-cloudwatch.md). If the S3 bucket or CloudWatch log group does not exist, then Amazon EMR creates it before uploading logs to the bucket.
+ For an additional list of Kubernetes configuration options, see [Spark Properties on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html#configuration).
The following Spark configurations are not supported.
+ `spark.kubernetes.authenticate.driver.serviceAccountName`
+ `spark.kubernetes.authenticate.executor.serviceAccountName`
+ `spark.kubernetes.namespace`
+ `spark.kubernetes.driver.pod.name`
+ `spark.kubernetes.container.image.pullPolicy`
+ `spark.kubernetes.container.image`
**Note**
You can use `spark.kubernetes.container.image` for customized Docker images. For more information, see [Customizing Docker images for Amazon EMR on EKS](docker-custom-images.md).
# Configure a job run to use Amazon S3 logs
To be able to monitor the job progress and to troubleshoot failures, you must configure your jobs to send log information to Amazon S3, Amazon CloudWatch Logs, or both. This topic helps you get started publishing application logs to Amazon S3 on your jobs that are launched with Amazon EMR on EKS.
**S3 logs IAM policy**
Before your jobs can send log data to Amazon S3, the following permissions must be included in the permissions policy for the job execution role. Replace *amzn-s3-demo-logging-bucket* with the name of your logging bucket.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::amzn-s3-demo-bucket",
"arn:aws:s3:::amzn-s3-demo-bucket/*"
],
"Sid": "AllowS3Putobject"
}
]
}
```
------
**Note**
Amazon EMR on EKS can also create an Amazon S3 bucket. If an Amazon S3 bucket is not available, include the `“s3:CreateBucket”` permission in the IAM policy.
After you've given your execution role the proper permissions to send logs to Amazon S3, your log data are sent to the following Amazon S3 locations when `s3MonitoringConfiguration` is passed in the `monitoringConfiguration` section of a `start-job-run` request, as shown in [Managing job runs with the AWS CLI](emr-eks-jobs-CLI.md).
+ Submitter Logs - /*logUri*/*virtual-cluster-id*/jobs/*job-id*/containers/*pod-name*/(stderr.gz/stdout.gz)
+ Driver Logs - /*logUri*/*virtual-cluster-id*/jobs/*job-id*/containers/*spark-application-id*/spark-*job-id*-driver/(stderr.gz/stdout.gz)
+ Executor Logs - /*logUri*/*virtual-cluster-id*/jobs/*job-id*/containers/*spark-application-id*/*executor-pod-name*/(stderr.gz/stdout.gz)
# Configure a job run to use Amazon CloudWatch Logs
To monitor job progress and to troubleshoot failures, you must configure your jobs to send log information to Amazon S3, Amazon CloudWatch Logs, or both. This topic helps you get started using CloudWatch Logs on your jobs that are launched with Amazon EMR on EKS. For more information about CloudWatch Logs, see [Monitoring Log Files](https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchLogs.html) in the Amazon CloudWatch User Guide.
**CloudWatch Logs IAM policy**
For your jobs to send log data to CloudWatch Logs, the following permissions must be included in the permissions policy for the job execution role. Replace *my\$1log\$1group\$1name* and *my\$1log\$1stream\$1prefix* with names of your CloudWatch log group and log stream names, respectively. Amazon EMR on EKS creates the log group and log stream if they do not exist as long as the execution role ARN has appropriate permissions.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
],
"Resource": [
"arn:aws:logs:*:*:*"
],
"Sid": "AllowLOGSCreatelogstream"
},
{
"Effect": "Allow",
"Action": [
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:*:*:log-group:my_log_group_name:log-stream:my_log_stream_prefix/*"
],
"Sid": "AllowLOGSPutlogevents"
}
]
}
```
------
**Note**
Amazon EMR on EKS can also create a log stream. If a log stream does not exist, the IAM policy should include the`"logs:CreateLogGroup"` permission.
After you've given your execution role the proper permissions, your application sends its log data to CloudWatch Logs when `cloudWatchMonitoringConfiguration` is passed in the `monitoringConfiguration` section of a `start-job-run` request, as shown in [Managing job runs with the AWS CLI](emr-eks-jobs-CLI.md).
In the `StartJobRun` API, *log\$1group\$1name *is the log group name for CloudWatch, and *log\$1stream\$1prefix* is the log stream name prefix for CloudWatch. You can view and search these logs in the AWS Management Console.
+ Submitter logs - *logGroup*/*logStreamPrefix*/*virtual-cluster-id*/jobs/*job-id*/containers/*pod-name*/(stderr/stdout)
+ Driver logs - *logGroup*/*logStreamPrefix*/*virtual-cluster-id*/jobs/*job-id*/containers/*spark-application-id*/spark-*job-id*-driver/(stderrstdout)
+ Executor logs - *logGroup*/*logStreamPrefix*/*virtual-cluster-id*/jobs/*job-id*/containers/*spark-application-id*/*executor-pod-name*/(stderr/stdout)
## List job runs
You can run `list-job-run` to show the states of job runs, as the following example demonstrates.
```
aws emr-containers list-job-runs --virtual-cluster-id
```
## Describe a job run
You can run `describe-job-run` to get more details about the job, such as job state, state details, and job name, as the following example demonstrates.
```
aws emr-containers describe-job-run --virtual-cluster-id cluster-id --id job-run-id
```
## Cancel a job run
You can run `cancel-job-run` to cancel running jobs, as the following example demonstrates.
```
aws emr-containers cancel-job-run --virtual-cluster-id cluster-id --id job-run-id
```