# Connecting to HubSpot Connecting to HubSpot HubSpot's CRM platform has all the tools and integrations you need for marketing, sales, content management, and customer service. + Marketing Hub - Marketing software to help you grow traffic, convert more visitors, and run complete inbound marketing campaigns at scale. + Sales Hub - Sales CRM software to help you get deeper insights into prospects, automate the tasks you have, and close more deals faster. + Service Hub - Customer service software to help you connect with customers, exceed expectations, and turn them into promoters who grow your business. + Operations Hub - Operations software that syncs your apps, cleans and creates customer data, and automates processes — so all your systems and teams work better together. **Topics** + [ # AWS Glue support for HubSpot ](hubspot-support.md) + [ # Policies containing the API operations for creating and using connections ](hubspot-configuring-iam-permissions.md) + [ # Configuring HubSpot ](hubspot-configuring.md) + [ # Configuring HubSpot connections ](hubspot-configuring-connections.md) + [ # Reading from HubSpot entities ](hubspot-reading-from-entities.md) + [ # Writing to HubSpot Entities ](hubspot-writing-to-entities.md) + [ # HubSpot connection options ](hubspot-connection-options.md) + [ # Limitations and notes for HubSpot connector ](hubspot-connector-limitations.md) # AWS Glue support for HubSpot AWS Glue supports HubSpot as follows: **Supported as a source?** Yes – Sync and Async. You can use AWS Glue ETL jobs to query data from HubSpot. **Supported as a target?** Yes. You can use AWS Glue ETL jobs to write data to HubSpot. **Supported HubSpot API versions** The following HubSpot API versions are supported: + v1 + v2 + v3 + v4 For entity support per version specific, see [Supported entities for Sync source](hubspot-reading-from-entities.md#sync-table) and [Supported entities for Async source](hubspot-reading-from-entities.md#async-table). # Policies containing the API operations for creating and using connections IAM policies The following sample policy describes the required AWS IAM permissions for creating and using connections. If you are creating a new role, create a policy that contains the following: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:ListConnectionTypes", "glue:DescribeConnectionType", "glue:RefreshOAuth2Tokens", "glue:ListEntities", "glue:DescribeEntity" ], "Resource": "*" } ] } ``` ------ If you don't want to use the above method, alternatively use the following managed IAM policies: + [AWSGlueServiceRole](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole) – Grants access to resources that various AWS Glue processes require to run on your behalf. These resources include AWS Glue, Amazon S3, IAM, CloudWatch Logs, and Amazon EC2. If you follow the naming convention for resources specified in this policy, AWS Glue processes have the required permissions. This policy is typically attached to roles specified when defining crawlers, jobs, and development endpoints. + [AWSGlueConsoleFullAccess](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSGlueConsoleFullAccess) – Grants full access to AWS Glue resources when an identity that the policy is attached to uses the AWS Management Console. If you follow the naming convention for resources specified in this policy, users have full console capabilities. This policy is typically attached to users of the AWS Glue console. # Configuring HubSpot Before you can use AWS Glue to transfer data from HubSpot, you must meet these requirements: ## Minimum requirements The following are minimum requirements: + You have a HubSpot account. For more information, see [Creating a HubSpot account](#hubspot-configuring-creating-hubspot-account). + Your HubSpot account is enabled for API access. + You should have an app under your HubSpot developer account that provides the client credentials that AWS Glue uses to access your data securely when it makes authenticated calls to your account. For more information, see [Creating a HubSpot developer app](#hubspot-configuring-creating-hubspot-developer-app). If you meet these requirements, you’re ready to connect AWS Glue to your HubSpot account. For typical connections, you don't have do anything else in HubSpot. ## Creating a HubSpot account To create a HubSpot account: 1. Go to the [HubSpot CRM SignUp URL](https://app.hubspot.com/login). 1. Enter your email address and choose **Verify email** (alternatively, you can choose to sign up with a Google, Microsoft or Apple account). 1. Check your inbox for the verification code from HubSpot. 1. Enter the 6-digit verification code and click **Next**. 1. Enter a password and click **Next**. 1. Enter your first name and last name and click **Next**, or sign up using the **Sign up with Google** link. 1. Enter your industry and click **Next**. 1. Enter your job role and click **Next**. 1. Enter your company name and click **Next**. 1. Select the size of your company (number of employees working in your company) and click **Next**. 1. Enter your company website and click **Next**. 1. Select where your data should be hosted (United States or Europe) and click **Create Account**. 1. Select the purpose of your account creation and click **Next**. 1. Choose **Connect Google Account** or choose to add contacts yourself to link your contacts with your HubSpot account. 1. Log in to your Google account if you chose the **Connect Google Account** option to link your contacts and start using your HubSpot account. ## Creating a HubSpot developer app App developer accounts are intended for creating and managing apps, integrations, and developer test accounts. They're also where you can create and manage App Marketplace listings. However, app developer accounts and their associated test accounts aren’t connected to a standard HubSpot account. They can't sync data or assets to or from another HubSpot account. To get the Client ID and Client Secret you create a developers account. 1. Go to https://developers.hubspot.com/ 1. Choose **Create developer account** and scroll down. 1. You will be asked whether you want to create an App developers account, Private App account, or CMS Developer Sandbox account. Choose **Create App developers account**. 1. Since you already created an Account with HubSpot, you can choose **Continue with this user**. 1. Click **Start signup**. 1. Enter your Job Role and click **Next**. 1. Name your developer account and click **Next**, then **Skip**. 1. Choose **Create App**. 1. Once your App is created, choose **Auth**. 1. Under Auth, note the Client ID and Client Secret. 1. Add your region specific **Redirect URL** as https:*//*.console.aws.amazon.com/gluestudio/oauth. For example, add https://us-east-1.console.aws.amazon.com/gluestudio/oauth for the us-east-1 region. 1. Scroll down and find scopes. There are two kinds of scopes you must select under headings "CRM" and "Standard". 1. Add the following scopes: ``` content automation oauth crm.objects.owners.read forms tickets crm.objects.contacts.write e-commerce crm.schemas.custom.read crm.objects.custom.read sales-email-read crm.objects.custom.write crm.objects.companies.write crm.lists.write crm.objects.companies.read crm.lists.read crm.objects.deals.read crm.objects.deals.write crm.objects.contacts.read ``` 1. Click **Save** and your dev account is now ready to use. 1. Scroll above to find the **Client ID**. 1. On the same page, click **Show** to get the **Client secret**. ## Creating a HubSpot developer test account Within app developer accounts, you can create developer test accounts to test apps and integrations without affecting any real HubSpot data. Developer test accounts do not mirror production accounts, but rather have access to a 90-day trial of the Enterprise versions of Marketing, Sales, Service, CMS, and Operations Hub, providing the ability to test most HubSpot tools and APIs. 1. Click **Home**. 1. Click **Create test account**. 1. Click **Create App Test Account**. 1. A new window appears. Enter the app test account name and click **Create**. Your app test account is now created. **Note** The developer account is related to development activities such as API integration, and the app test account is used to see the data which is created or pulled by the developer account. # Configuring HubSpot connections HubSpot supports the AUTHORIZATION\$1CODE grant type for OAuth2. + This grant type is considered "three-legged" OAuth as it relies on redirecting users to a third-party authorization server to authenticate the user. It is used when creating connections via the AWS Glue console. The user creating a connection needs to provide the OAuth related information like Client ID and Client Secret for their HubSpot client application. The AWS Glue console will redirect the user to HubSpot where the user must login and allow AWS Glue the requested permissions to access their HubSpot instance. + Users may still opt to create their own connected app in HubSpot and provide their own client id and client secret when creating connections through the AWS Glue console. In this scenario, they will still be redirected to HubSpot to login and authorize AWS Glue to access their resources. + This grant type results in a refresh token and access token. The access token is short lived, and may be refreshed automatically without user interaction using the refresh token. + For public HubSpot documentation on creating a connected app for Authorization Code OAuth flow, see [Public apps](https://developers.hubspot.com/docs/api/creating-an-app). To configure a HubSpot connection: 1. In AWS Secrets Manager, create a secret with the following details: 1. For the customer managed connected app, the Secret should contain the connected app Consumer Secret with `USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET` as key. 1. Note: You must create a secret for the connection in AWS Glue. 1. In AWS Glue Glue Studio, create a connection under **Data Connections** by following the steps below: 1. When selecting a **Connection type**, select HubSpot. 1. Provide the HubSpot environment. 1. Select the AWS IAM role which AWS Glue can assume and has permissions for following actions: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:GetSecretValue", "secretsmanager:PutSecretValue", "ec2:CreateNetworkInterface", "ec2:DescribeNetworkInterfaces", "ec2:DeleteNetworkInterface" ], "Resource": "*" } ] } ``` ------ 1. Select the `secretName` which you want to use for this connection in AWS Glue to put the tokens. 1. Select the network options if you want to use your network. 1. Grant the IAM role associated with your AWS Glue job permission to read `secretName`. 1. In your AWS Glue job configuration, provide `connectionName` as an **Additional network connection**. # Reading from HubSpot entities **Prerequisite** A HubSpot object you would like to read from. You will need the object name such as contact or task. The following table shows the supported entities for Sync source. ## Supported entities for Sync source | Entity | API version | Can be filtered | Supports limit | Supports Order by | Supports Select \$1 | Supports partioning | | --- | --- | --- | --- | --- | --- | --- | | Campaigns | v1 | No | Yes | No | Yes | No | | Companies | v3 | Yes | Yes | Yes | Yes | Yes | | Contacts | v3 | Yes | Yes | Yes | Yes | Yes | | Contact Lists | v1 | No | Yes | No | Yes | No | | Deals | v3 | Yes | Yes | Yes | Yes | Yes | | CRM Pipeline (Deal Pipelines) | v1 | No | No | No | Yes | No | | Email Events | v1 | No | Yes | No | Yes | No | | Calls | v3 | Yes | Yes | Yes | Yes | Yes | | Notes | v3 | Yes | Yes | Yes | Yes | Yes | | Emails | v3 | Yes | Yes | Yes | Yes | Yes | | Meetings | v3 | Yes | Yes | Yes | Yes | Yes | | Tasks | v3 | Yes | Yes | Yes | Yes | Yes | | Postal Mails | v3 | Yes | Yes | Yes | Yes | Yes | | Custom Objects | v3 | Yes | Yes | Yes | Yes | Yes | | Forms | v2 | No | No | No | Yes | No | | Owners | v3 | No | Yes | No | Yes | No | | Products | v3 | Yes | Yes | Yes | Yes | Yes | | Tickets | v3 | Yes | Yes | Yes | Yes | Yes | | Workflows | v3 | No | No | No | Yes | No | | Associations | v4 | Yes | No | No | Yes | No | | Associations Labels | v4 | No | No | No | Yes | No | **Example**: ``` hubspot_read = glueContext.create_dynamic_frame.from_options( connection_type="hubspot", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "contact", "API_VERSION": "v3" } ``` ## Supported entities for Async source | Entity | API version | Can be filtered | Supports limit | Supports Order by | Supports Select \$1 | Supports partioning | | --- | --- | --- | --- | --- | --- | --- | | Companies | v3 | Yes | No | Yes | Yes | No | | Contacts | v3 | Yes | No | Yes | Yes | No | | Deals | v3 | Yes | No | Yes | Yes | No | | Calls | v3 | Yes | No | Yes | Yes | No | | Notes | v3 | Yes | No | Yes | Yes | No | | Emails | v3 | Yes | No | Yes | Yes | No | | Meetings | v3 | Yes | No | Yes | Yes | No | | Tasks | v3 | Yes | No | Yes | Yes | No | | Postal Mails | v3 | Yes | No | Yes | Yes | No | | Custom Objects | v3 | Yes | No | Yes | Yes | No | | Products | v3 | Yes | No | Yes | Yes | No | | Tickets | v3 | Yes | No | Yes | Yes | No | **Example**: ``` hubspot_read = glueContext.create_dynamic_frame.from_options( connection_type="hubspot", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "contact", "API_VERSION": "v3", "TRANSFER_MODE": "ASYNC" } ``` **HubSpot entity and field details**: **HubSpot API v4**: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/hubspot-reading-from-entities.html) **Note** For the `Associations` object, to fetch associations between two objects, you need to provide the 'from Id' (the ID of the first object) via a mandatory filter while creating an AWS Glue job. If you want to fetch associations for multiple from IDs in that case, you have to provide multiple IDs in the `where` clause. For example: for fetching `Associations` for contact IDs '1' and '151', you need to provide a filter as `where id=1 AND id=151`. **HubSpot API v3**: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/hubspot-reading-from-entities.html) For the following entities, HubSpot provides endpoints to fetch metadata dynamically, so that operator support is captured at the datatype level for each entity. **Note** `DML_STATUS` is a virtual field added on every record at runtime to determine its status (CREATED/UPDATED) in the Sync mode. The `CONTAINS/LIKE` operator is not supported in the Async mode. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/hubspot-reading-from-entities.html) **HubSpot API v2**: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/hubspot-reading-from-entities.html) **HubSpot API v1**: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/hubspot-reading-from-entities.html) ## Partitioning queries You can provide the additional Spark options `PARTITION_FIELD`, `LOWER_BOUND`, `UPPER_BOUND`, and `NUM_PARTITIONS` if you want to utilize concurrency in Spark. With these parameters, the original query would be split into `NUM_PARTITIONS` number of sub-queries that can be executed by Spark tasks concurrently. + `PARTITION_FIELD`: the name of the field to be used to partition the query. + `LOWER_BOUND`: an **inclusive** lower bound value of the chosen partition field. For the DateTime field, we accept the value in ISO format. Examples of valid value: ``` “2024-01-01T10:00:00.115Z" ``` + `UPPER_BOUND`: an **exclusive** upper bound value of the chosen partition field. + `NUM_PARTITIONS`: the number of partitions. The following table describes the entity partitioning field support details: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/hubspot-reading-from-entities.html) Example: ``` hubspot_read = glueContext.create_dynamic_frame.from_options( connection_type="hubspot", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "company", "API_VERSION": "v3", "PARTITION_FIELD": "hs_object_id" "LOWER_BOUND": "50" "UPPER_BOUND": "16726619290" "NUM_PARTITIONS": "10" } ``` # Writing to HubSpot Entities ## Prerequisites + A HubSpot object you would like to write to. You will need the object name such as contact or ticket. + The HubSpot connector supports following write operations: + INSERT + UPDATE + When using the `UPDATE` write operation, the `ID_FIELD_NAMES` option must be provided to specify the ID field for the records. ## Supported entities for Sync Destination | Entity | API Version | Will be supported as Destination Connector | Can be Inserted | Can be Updated | | --- | --- | --- | --- | --- | | Companies | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Contacts | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Deals | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Products | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Calls | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Meetings | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Notes | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Emails | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Tasks | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Postal Mails | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Custom Objects | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Tickets | v3 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | | Associations | v4 | Yes | Yes (Single, Bulk) | No | | Associations Labels | v4 | Yes | Yes (Single, Bulk) | Yes (Single, Bulk) | **Examples**: **INSERT Operation** ``` hubspot_write = glueContext.write_dynamic_frame.from_options( frame=frameToWrite, connection_type="hubspot", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "contact", "API_VERSION": "v3", "WRITE_OPERATION": "INSERT" } ) ``` **UPDATE Operation** ``` hubspot_write = glueContext.write_dynamic_frame.from_options( frame=frameToWrite, connection_type="hubspot", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "deal", "API_VERSION": "v3", "WRITE_OPERATION": "UPDATE", "ID_FIELD_NAMES": "hs_object_id" } ) ``` # HubSpot connection options The following are connection options for HubSpot: + `ENTITY_NAME`(String) - (Required) Used for Read. The name of your object in HubSpot. + `API_VERSION`(String) - (Required) Used for Read. HubSpot Rest API version you want to use. For example: v1,v2,v3,v4. + `SELECTED_FIELDS`(List) - Default: empty(SELECT \$1). Used for Read. Columns you want to select for the object. + `FILTER_PREDICATE`(String) - Default: empty. Used for Read. It should be in the Spark SQL format. + `QUERY`(String) - Default: empty. Used for Read. Full Spark SQL query. + `PARTITION_FIELD`(String) - Used for Read. Field to be used to partition query. + `LOWER_BOUND`(String)- Used for Read. An inclusive lower bound value of the chosen partition field. + `UPPER_BOUND`(String) - Used for Read. An exclusive upper bound value of the chosen partition field. + `NUM_PARTITIONS`(Integer) - Default: 1. Used for Read. Number of partitions for read. + `TRANSFER_MODE`(String) - Used to indicate whether the query should be run on Async mode. + `WRITE_OPERATION`(String) - Default: INSERT. Used for write. Value should be INSERT or UPDATE. + `ID_FIELD_NAMES`(String) - Default : null. Required for UPDATE. # Limitations and notes for HubSpot connector The following are limitations or notes for the HubSpot connector: + The search endpoints are limited to 10,000 total results for any given query. Any partition having more than 10,000 records will result in a 400 error. + Other important limitations for the connector are described in [Limitations](https://developers.hubspot.com/docs/api/crm/search#limitations). + A maximum of three filtering statements are accepted by HubSpot. + Currently, HubSpot supports Associations between standard HubSpot objects (e.g. contact, company, deal, or ticket) and custom objects. + For a Free account: you can create only up to 10 association types between each object pairing (e.g. contacts and companies). + For a Super Admin account: you can create only up to 50 association types between each object pairing. + For more information, see [Associations v4](https://developers.hubspot.com/docs/api/crm/) and [Create and use association labels](https://knowledge.hubspot.com/object-settings/create-and-use-association-labels). + The 'Quote' and 'Communications' objects are not present for Associations as they are currently not supported in the connector. + For Async, SaaS sorts the values in ascending order only. + For the `Ticket` entity, SaaS doesn't return the `hs_object_id` field in the Async mode.