# Connecting to Adobe Marketo Engage Adobe Marketo Engage is a marketing automation platform that enables marketers to manage personalized multi-channel programs and campaigns to prospects and customers. **Topics** + [AWS Glue support for Adobe Marketo Engage](adobe-marketo-engage-support.md) + [Policies containing the API operations for creating and using connections](adobe-marketo-engage-configuring-iam-permissions.md) + [Configuring Adobe Marketo Engage](adobe-marketo-engage-configuring.md) + [Configuring Adobe Marketo Engage connections](adobe-marketo-engage-configuring-connections.md) + [Reading from Adobe Marketo Engage entities](adobe-marketo-engage-reading-from-entities.md) + [Writing to Adobe Marketo Engage entities](adobe-marketo-engage-writing-to-entities.md) + [Adobe Marketo Engage connection options](adobe-marketo-engage-connection-options.md) + [Limitations and notes for Adobe Marketo Engage connector](adobe-marketo-engage-connector-limitations.md) # AWS Glue support for Adobe Marketo Engage AWS Glue supports Adobe Marketo Engage as follows: **Supported as a source?** Yes. You can use AWS Glue ETL jobs to query data from Adobe Marketo Engage. **Supported as a target?** Yes. You can use AWS Glue ETL jobs to write data to Adobe Marketo. **Supported Adobe Marketo Engage API versions** The following Adobe Marketo Engage API versions are supported: + v1 For entity support per version specific, see Supported entities for Source. # Policies containing the API operations for creating and using connections The following sample policy describes the required AWS IAM permissions for creating and using connections. If you are creating a new role, create a policy that contains the following: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:ListConnectionTypes", "glue:DescribeConnectionType", "glue:RefreshOAuth2Tokens", "glue:ListEntities", "glue:DescribeEntity" ], "Resource": "*" } ] } ``` ------ If you don't want to use the above method, alternatively use the following managed IAM policies: + [AWSGlueServiceRole](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole) – Grants access to resources that various AWS Glue processes require to run on your behalf. These resources include AWS Glue, Amazon S3, IAM, CloudWatch Logs, and Amazon EC2. If you follow the naming convention for resources specified in this policy, AWS Glue processes have the required permissions. This policy is typically attached to roles specified when defining crawlers, jobs, and development endpoints. + [AWSGlueConsoleFullAccess](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSGlueConsoleFullAccess) – Grants full access to AWS Glue resources when an identity that the policy is attached to uses the AWS Management Console. If you follow the naming convention for resources specified in this policy, users have full console capabilities. This policy is typically attached to users of the AWS Glue console. # Configuring Adobe Marketo Engage Before you can use AWS Glue to transfer data from Adobe Marketo Engage, you must meet these requirements: ## Minimum requirements The following are minimum requirements: + You have a Adobe Marketo Engage account with client credentials. + Your Adobe Marketo Engage account has API access with a valid license. If you meet these requirements, you’re ready to connect AWS Glue to your Adobe Marketo Engage account. For typical connections, you don't need do anything else in Adobe Marketo Engage. ## Getting OAuth 2.0 credentials To obtain API credentials to make authenticated calls to your instance, see [Adobe Marketo Rest API](https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/rest-api) # Configuring Adobe Marketo Engage connections Adobe Marketo Engage supports the CLIENT CREDENTIALS grant type for OAuth2. + This grant type is considered 2-legged OAuth 2.0 as it is used by clients to obtain an access token outside of the context of a user. AWS Glue is able to use the client ID and client secret to authenticate the Adobe Marketo Engage APIs which are provided by custom services that you define. + Each custom service is owned by an API-only user which has a set of roles and permissions which authorize the service to perform specific actions. An access token is associated with a single custom service. + This grant type results in an access token which is short lived, and may be renewed by calling an identity endpoint. + For public Adobe Marketo Engage documentation for OAuth 2.0 with client credentials, see [Authentication](https://developers.adobe-marketo-engage.com/rest-api/authentication/) in the Adobe Marketo Engage Developer Guide. To configure a Adobe Marketo Engage connection: 1. In AWS Secrets Manager, create a secret with the following details: 1. For the customer managed connected app, the Secret should contain the connected app Consumer Secret with `USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET` as key. 1. Note: You must create a secret per connection in AWS Glue. 1. In AWS Glue Glue Studio, create a connection under **Data Connections** by following the steps below: 1. When selecting a **Connection type**, select Adobe Marketo Engage. 1. Provide the `INSTANCE_URL` of the Adobe Marketo Engage instance you want to connect to. 1. Select the AWS IAM role which AWS Glue can assume and has permissions for the following actions: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:GetSecretValue", "secretsmanager:PutSecretValue", "ec2:CreateNetworkInterface", "ec2:DescribeNetworkInterfaces", "ec2:DeleteNetworkInterface" ], "Resource": "*" } ] } ``` ------ 1. Select the `secretName` which you want to use for this connection in AWS Glue to put the tokens. 1. Select the network options if you want to use your network. 1. Grant the IAM role associated with your AWS Glue job permission to read `secretName`. # Reading from Adobe Marketo Engage entities **Prerequisite** An Adobe Marketo Engage object you would like to read from. You will need the object name such as leads or activities or customobjects. The following tables shows the supported entities. **Supported entities for source (synchronous)**: | Entity | Can be filtered | Supports limit | Supports Order by | Supports Select \$1 | Supports partitioning | | --- | --- | --- | --- | --- | --- | | leads | Yes | Yes | No | Yes | No | | activities | Yes | Yes | No | Yes | No | | customobjects | Yes | Yes | No | Yes | No | **Supported entities for source (asynchronous)**: | Entity | Can be filtered | Supports limit | Supports Order by | Supports Select \$1 | Supports partitioning | | --- | --- | --- | --- | --- | --- | | leads | Yes | No | No | Yes | Yes | | activities | Yes | No | No | Yes | No | | customobjects | Yes | No | No | Yes | Yes | **Example**: ``` adobe-marketo-engage_read = glueContext.create_dynamic_frame.from_options( connection_type="adobe-marketo-engage", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "leads", "API_VERSION": "v2", "INSTANCE_URL": "https://539-t**-6**.mktorest.com" } ``` **Adobe Marketo Engage entity and field details**: **Entities with static metadata**: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/adobe-marketo-engage-reading-from-entities.html) **Entities with dynamic metadata**: For the following entities, Adobe Marketo Engage provides endpoints to fetch metadata dynamically, so that operator support is captured at the datatype level for each entity. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/adobe-marketo-engage-reading-from-entities.html) ## Partitioning queries You can provide the additional Spark options `PARTITION_FIELD`, `LOWER_BOUND`, `UPPER_BOUND`, and `NUM_PARTITIONS` if you want to utilize concurrency in Spark. With these parameters, the original query would be split into `NUM_PARTITIONS` number of sub-queries that can be executed by Spark tasks concurrently. + `PARTITION_FIELD`: the name of the field to be used to partition the query. + `LOWER_BOUND`: an **inclusive** lower bound value of the chosen partition field. For the DateTime field, we accept the value in ISO format. Example of valid value: ``` "2024-07-01T00:00:00.000Z" ``` + `UPPER_BOUND`: an **exclusive** upper bound value of the chosen partition field. + `NUM_PARTITIONS`: the number of partitions. The following table describes the entity partitioning field support details: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/adobe-marketo-engage-reading-from-entities.html) Example: ``` adobe-marketo-engage_read = glueContext.create_dynamic_frame.from_options( connection_type="adobe-marketo-engage", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "leads", "API_VERSION": "v1", "PARTITION_FIELD": "createdAt" "LOWER_BOUND": "2024-07-01T00:00:00.000Z" "UPPER_BOUND": "2024-07-02T00:00:00.000Z" "NUM_PARTITIONS": "10" } ``` # Writing to Adobe Marketo Engage entities **Prerequisites** + An Adobe Marketo object you would like to write to. You will need the object name such as `leads` or `customobjects`. + The Adobe Marketo connector supports three write operations: + INSERT + UPSERT + UPDATE + For `UPSERT` and `UPDATE` write operations, you must provide the `ID_FIELD_NAMES` option to specify the ID field for the records. When working with the `leads` entity, use `email` as `ID_FIELD_NAMES` for `UPSERT` operations and `id` for `UPDATE` operations. For the `customobjects` entity, use `marketoGUID` as `ID_FIELD_NAMES` for both `UPDATE` and `UPSERT` operations. **Supported entities for Destination (Synchronous)** | Entity name | Will be supported as Destination Connector | Can be Inserted | Can be Updated | Can be Upserted | | --- | --- | --- | --- | --- | | leads | Yes | Yes(Bulk) | Yes(Bulk) | Yes(Bulk) | | customobjects | Yes | Yes(Bulk) | Yes(Bulk) | Yes(Bulk) | **Example**: **INSERT Operation:** ``` marketo_write = glueContext.write_dynamic_frame.from_options( frame=frameToWrite, connection_type="marketo", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "leads", "API_VERSION": "v1", "WRITE_OPERATION": "INSERT" } ``` **UPDATE Operation:** ``` marketo_write = glueContext.write_dynamic_frame.from_options( frame=frameToWrite, connection_type="marketo", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "leads", "API_VERSION": "v1", "WRITE_OPERATION": "UPDATE", "ID_FIELD_NAMES": "id" } ``` **Note** For the `leads` and `customobjects` entities, Adobe Marketo provides endpoints to fetch metadata dynamically so the writable fields are identified from the Marketo API response. # Adobe Marketo Engage connection options The following are connection options for Adobe Marketo Engage: + `ENTITY_NAME`(String) - (Required) Used for Read. The name of your object in Adobe Marketo Engage. + `API_VERSION`(String) - (Required) Used for Read. Adobe Marketo Engage Rest API version you want to use. For example: v1. + `SELECTED_FIELDS`(List) - Default: empty(SELECT \$1). Used for Read. Columns you want to select for the object. + `FILTER_PREDICATE`(String) - Default: empty. Used for Read. It should be in the Spark SQL format. + `QUERY`(String) - Default: empty. Used for Read. Full Spark SQL query. + `PARTITION_FIELD`(String) - Used for Read. Field to be used to partition query. + `LOWER_BOUND`(String)- Used for Read. An inclusive lower bound value of the chosen partition field. + `UPPER_BOUND`(String) - Used for Read. An exclusive upper bound value of the chosen partition field. + `NUM_PARTITIONS`(Integer) - Default: 1. Used for Read. Number of partitions for read. + `TRANSFER_MODE`(String) - Default: SYNC. Used for asynchronous read. + `WRITE_OPERATION`(String) - Default: INSERT. Used for write. Value should be INSERT, UPDATE, UPSERT. + `ID_FIELD_NAMES`(String) - Default : null. Required for UPDATE and UPSERT. # Limitations and notes for Adobe Marketo Engage connector The following are limitations or notes for the Adobe Marketo Engage connector: + 'sinceDatetime' and 'activityTypeId' are mandatory filter parameters for the Sync Activities entity. + Subscriptions are allocated 50,000 API calls per day (which resets daily at 12:00AM CST). Additional daily capacity may be purchased as part of a Adobe Marketo Engage subscription. For reference, see [Adobe Marketo Rest API](https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/rest-api). + The maximum time span for the date range filter (`createdAt` or `updatedAt`) is 31 days. For reference, see [Bulk Extract - Marketo Developers](https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/bulk-extract/bulk-extract). + Subscriptions are allocated a maximum of 10 bulk extract jobs in the queue at any given time. For reference, see [Bulk Extract - Marketo Developers](https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/bulk-extract/bulk-extract). + By default, extract jobs are limited to 500 MB per day (which resets daily at 12:00AM CST). Additional daily capacity may be purchased as part of a Adobe Marketo Engage subscription. For reference, see [Bulk Extract - Marketo Developers](https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/bulk-extract/bulk-extract). + The maximum number of concurrent export jobs is 2. For reference, see [Bulk Extract - Marketo Developers](https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/bulk-extract/bulk-extract). + The maximum number of queued export jobs (inclusive of currently exporting jobs) is 10. For reference, see [Bulk Extract - Marketo Developers](https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/bulk-extract/bulk-extract). + The maximum allowed file size is 1 GB to extract from a Bulk job. + Once an asynchronous job is created, the file retention period will be 7 days before expiring. For reference, see [Bulk Extract - Marketo Developers](https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/bulk-extract/bulk-extract). + `createdAt` or `updatedAt` are mandatory filter parameters for the Async Leads entity. + `createdAt` is a mandatory filter parameter for the Async Activities entity. + `updatedAt` is a mandatory filter parameter for the Async Custom Object entity. + When using AWS Glue SaaS connectors, users cannot identify which specific records failed during a write operation to destination SaaS platforms in cases of partial failures. + When using Sync write operations, any fields with null values in the input file will be automatically dropped and not sent to the SaaS platform. + You can create or update up to 300 records in a batch for Sync write. For more information, see [Adobe Marketo Engage Integration Best Practices](https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/marketo-integration-best-practices) and [Bulk Extract](https://https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/bulk-extract/bulk-activity-extract) .