# Connecting to Datadog
Datadog is a monitoring and analytics platform for cloud-scale applications, including infrastructure, applications, services, and tools.
**Topics**
+ [AWS Glue support for Datadog](datadog-support.md)
+ [Policies containing the API operations for creating and using connections](datadog-configuring-iam-permissions.md)
+ [Configuring Datadog](datadog-configuring.md)
+ [Configuring Datadog connections](datadog-configuring-connections.md)
+ [Reading from Datadog entities](datadog-reading-from-entities.md)
+ [Datadog connection options](datadog-connection-options.md)
+ [Creating a Datadog account](datadog-create-account.md)
+ [Limitations](datadog-connector-limitations.md)
# AWS Glue support for Datadog
AWS Glue supports Datadog as follows:
**Supported as a source?**
Yes. You can use AWS Glue ETL jobs to query data from Datadog.
**Supported as a target?**
No.
**Supported Datadog API versions**
+ v1
+ v2
# Policies containing the API operations for creating and using connections
The following sample policy describes the required AWS permissions for creating and using connections. If you are creating a new role, create a policy that contains the following:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"glue:ListConnectionTypes",
"glue:DescribeConnectionType",
"glue:RefreshOAuth2Tokens",
"glue:ListEntities",
"glue:DescribeEntity"
],
"Resource": "*"
}
]
}
```
------
If you don't want to use the preceding method, alternatively, use the following managed IAM policies:
+ [ AWSGlueServiceRole ](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole) – Grants access to resources that various AWS Glue processes require to run on your behalf. These resources include AWS Glue, Amazon S3, IAM, CloudWatch Logs, and Amazon EC2. If you follow the naming convention for resources specified in this policy, AWS Glue processes have the required permissions. This policy is typically attached to roles specified when defining crawlers, jobs, and development endpoints.
+ [ AWSGlueConsoleFullAccess ](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSGlueConsoleFullAccess) – Grants full access to AWS Glue resources when an identity that the policy is attached to uses the AWS Management console. If you follow the naming convention for resources specified in this policy, users have full console capabilities. This policy is typically attached to users of the AWS Glue console.
# Configuring Datadog
Before you can use AWS Glue to transfer from Datadog, you must meet the following requirements:
## Minimum requirements
+ You have a Datadog account with DD-API-KEY and DD-APPLICATION-KEY. For more information about creating an account, see [Creating a Datadog account](datadog-create-account.md).
+ Your Datadog account has API access with valid license.
Datadog supports the following six URLs. All Datadog API clients are configured by default to consume Datadog US1 site APIs. If you are on the Datadog EU site, you must select https://api.datadoghq.eu URL with the `DD-API-KEY` and `DD-APPLICATION-KEY` of the Datadog EU site to access the APIs. Similarly, for other sites, you should select the respective URLs with the `DD-API-KEY and DD-APPLICATION-KEY` of the respective site.
+ US1 API URL — [https://api.datadoghq.com](https://api.datadoghq.com)https://api.datadoghq.com
+ EU API URL — [https://api.datadoghq.eu ](https://api.datadoghq.eu)
+ US3 API URL — [https://api.us3.datadoghq.com](https://api.us3.datadoghq.com)
+ US5 API URL — [https://api.us5.datadoghq.com](https://api.us5.datadoghq.com)
+ S1-FED API URL — [https://api.ddog-gov.com](https://api.ddog-gov.com)
+ Japan API URL — [https://api.ap1.datadoghq.com](https://api.ap1.datadoghq.com)
If you meet these requirements, you’re ready to connect AWS Glue to your Datadog account.
# Configuring Datadog connections
Datadog supports custom authentication. Following are the steps to configure Datadog connection:
To configure a Datadog connection:
1. In AWS Secrets Manager, create a secret with the following details:
For customer managed connected app – Secret should contain the connected app Consumer Secret with `API_KEY` and `APPLICATION_KEY` as keys.
**Note**
It is a must to create a secret per connection in AWS Glue.
1. In AWS Glue Studio, create a connection under **Data Connections** by following the steps below:
1. When selecting a **Connection type**, select Datadog.
1. Provide the `Instance_Url` of the Datadog you want to connect to.
1. Select the IAM role for which AWS Glue can assume and has permissions for following actions:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"ec2:CreateNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface"
],
"Resource": "*"
}
]
}
```
------
1. Select the `secretName` which you want to use for this connection in AWS Glue to put the tokens.
1. Select the network options if you want to use your network.
1. Grant the IAM role associated with your AWS Glue job permission to read `secretName`.
# Reading from Datadog entities
**Prerequisites**
A Datadog Object you would like to read from. Refer the supported entities table below to check the available entities.
**Supported entities**
| Entity | Can be Filtered | Supports Limit | Supports Order By | Supports Select \$1 | Supports Partitioning |
| --- | --- | --- | --- | --- | --- |
| Metrics Timeseries | Yes | No | No | Yes | No |
| Log Queries | Yes | Yes | Yes | Yes | No |
**Example**
```
Datadog_read = glueContext.create_dynamic_frame.from_options(
connection_type="datadog",
connection_options={
"connectionName": "connectionName",
"ENTITY_NAME": "log-queries",
"API_VERSION": "v2",
"INSTANCE_URL": "https://api.datadoghq.com",
"FILTER_PREDICATE": "from = `2023-10-03T09:00:26Z`"
}
```
**Datadog entity and field details**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/datadog-reading-from-entities.html)
# Datadog connection options
The following are connection options for Datadog:
+ `ENTITY_NAME`(String) – (Required) Used for Read/Write. The name of your Object in Datadog.
+ `API_VERSION`(String) – (Required) Used for Read/Write. Datadog Rest API version you want to use. `v1` version supports `metrics-timeseries` entity whereas, `v2` version supports `log-queries` entity.
+ `INSTANCE_URL`(String) – (Required) Used for Read. Datadog instance URL. Datadog instance URL varies per region.
+ `SELECTED_FIELDS`(List) – Default: empty(SELECT \$1). Used for Read. Columns you want to select for the object.
+ `FILTER_PREDICATE`(String) – Default: empty. Used for Read. It should be in the Spark SQL format.
+ `QUERY`(String) – Default: empty. Used for Read. Full Spark SQL query.
# Creating a Datadog account
1. Go to [https://www.datadoghq.com/](https://api.datadoghq.com).
1. Choose **GET STARTED FREE**.
1. Enter the required information and sign up.
1. Install the **Datadog Agent Installer** as suggested.
1. Ensure that the account gets registered with a valid organization (from the list available) that has an active Datadog subscription.
1. After logging in to your Datadog account, hover over your username in the top-right corner to view the **Keys** details:
1. To get your API key, choose **API Keys**.
1. To get your application key, choose **Application Keys**.
# Limitations
The following are limitations for the Datadog connector:
+ Datadog doesn’t support either field based or record based partitioning.
+ `from` is mandatory filter parameter for `Log Queries` entity.
+ `from_to_date` and `query` are mandatory filter parameters for `Metrics Timeseries` entity.