# Connecting to Twilio Connecting to Twilio Twilio provides programmable communication tools for making and receiving phone calls, sending, and receiving text messages, and performing other communication functions using its web service APIs. Twilio’s APIs power its platform for communications. Behind these APIs is a software layer connecting and optimizing communications networks around the world to allow your users to call and message anyone, globally. As a Twilio user, you can connect AWS Glue to your Twilio account. Then, you can use Twilio as a data source in your ETL jobs. Run these jobs to transfer data between Twilio and AWS services or other supported applications. **Topics** + [ # AWS Glue support for Twilio ](twilio-support.md) + [ # Policies containing the API operations for creating and using connections ](twilio-configuring-iam-permissions.md) + [ # Configuring Twilio ](twilio-configuring.md) + [ # Configuring Twilio connections ](twilio-configuring-connections.md) + [ # Reading from Twilio entities ](twilio-reading-from-entities.md) + [ # Twilio connection options ](twilio-connection-options.md) + [ # Limitations and notes for Twilio connector ](twilio-connector-limitations.md) # AWS Glue support for Twilio AWS Glue supports Twilio as follows: **Supported as a source?** Yes. You can use AWS Glue ETL jobs to query data from Twilio. **Supported as a target?** No. **Supported Twilio API versions** The following Twilio API versions are supported: + v1 + 2010-04-01 # Policies containing the API operations for creating and using connections IAM policies The following sample policy describes the required AWS IAM permissions for creating and using connections. If you are creating a new role, create a policy that contains the following: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:ListConnectionTypes", "glue:DescribeConnectionType", "glue:RefreshOAuth2Tokens", "glue:ListEntities", "glue:DescribeEntity" ], "Resource": "*" } ] } ``` ------ If you don't want to use the above method, alternatively use the following managed IAM policies: + [AWSGlueServiceRole](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole) – Grants access to resources that various AWS Glue processes require to run on your behalf. These resources include AWS Glue, Amazon S3, IAM, CloudWatch Logs, and Amazon EC2. If you follow the naming convention for resources specified in this policy, AWS Glue processes have the required permissions. This policy is typically attached to roles specified when defining crawlers, jobs, and development endpoints. + [AWSGlueConsoleFullAccess](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSGlueConsoleFullAccess) – Grants full access to AWS Glue resources when an identity that the policy is attached to uses the AWS Management Console. If you follow the naming convention for resources specified in this policy, users have full console capabilities. This policy is typically attached to users of the AWS Glue console. # Configuring Twilio Before you can use AWS Glue to transfer data from Twilio, you must meet these requirements: ## Minimum requirements The following are minimum requirements: + You have a Twilio account with username and password. + Your Twilio account is enabled for API access. If you meet these requirements, you’re ready to connect AWS Glue to your Twilio account. For typical connections, you don't need do anything else in Twilio. # Configuring Twilio connections Twilio supports username and password for Basic Authentication. Basic Authentication is a simple authentication method where clients provide credentials directly to access protected resources. AWS Glue is able to use the username (Account SID) and password (Auth Token) to authenticate Twilio APIs. For public Twilio documentation for Basic Authentication flow, see [Basic Authentication \$1 Twilio](https://www.twilio.com/docs/glossary/what-is-basic-authentication). To configure a Twilio connection: 1. In AWS Secrets Manager, create a secret with the following details: + For Basic Authentication: the Secret should contain the connected app Consumer Secret with the **Account SID** (Username) and **Auth Token** (Password). **Note** You must create a secret for your connections in AWS Glue. 1. In AWS Glue Glue Studio, create a connection under **Data Connections** by following the steps below: 1. When selecting a **Connection type**, select Twilio. 1. Provide the `[Edge\$1Location](https://www.twilio.com/docs/global-infrastructure/edge-locations)` of the Twilio instance you want to connect to. 1. Select the AWS IAM role which AWS Glue can assume and has permissions for following actions: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:GetSecretValue", "secretsmanager:PutSecretValue", "ec2:CreateNetworkInterface", "ec2:DescribeNetworkInterfaces", "ec2:DeleteNetworkInterface" ], "Resource": "*" } ] } ``` ------ 1. Select the `secretName` which you want to use for this connection in AWS Glue to put the tokens. 1. Select the network options if you want to use your network. 1. Grant the IAM role associated with your AWS Glue job permission to read `secretName`. # Reading from Twilio entities **Prerequisite** A Twilio object you would like to read from. You will need the object name such as `SMS-Message` or `SMS-CountryPricing`. **Supported entities for source**: | Entity | Interface | Can be filtered | Supports limit | Supports Order by | Supports Select \$1 | Supports partitioning | | --- | --- | --- | --- | --- | --- | --- | | SMS-Message | REST | Yes | Yes | No | Yes | Yes | | SMS-CountryPricing | REST | No | No | No | Yes | No | | Voice-Call | REST | Yes | Yes | No | Yes | No | | Voice-Application | REST | Yes | Yes | No | Yes | No | | Voice-OutgoingCallerID | REST | Yes | Yes | No | Yes | No | | Voice-Queue | REST | Yes | Yes | No | Yes | No | | Conversations-Conversation | REST | Yes | Yes | No | Yes | No | | Conversations-User | REST | No | Yes | No | Yes | No | | Conversations-Role | REST | No | Yes | No | Yes | No | | Conversations-Configuration | REST | No | No | No | Yes | No | | Conversations-AddressConfiguration | REST | Yes | Yes | No | Yes | No | | Conversations-WebhookConfiguration | REST | No | No | No | Yes | No | | Conversations-ParticipantConversation | REST | No | No | No | Yes | No | | Conversations-Credential | REST | No | Yes | No | Yes | No | | Conversations-ConversationService | REST | No | Yes | No | Yes | No | **Example**: ``` twilio_read = glueContext.create_dynamic_frame.from_options( connection_type="twilio", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "sms-message", "API_VERSION": "2010-04-01", "Edge_Location": "sydney.us1" } ``` **Twilio entity and field details**: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/glue/latest/dg/twilio-reading-from-entities.html) ## Partitioning queries **Fields supporting partitioning**: In Twilio, the DateTime datatype fields support field-based partitioning. You can provide the additional Spark options `PARTITION_FIELD`, `LOWER_BOUND`, `UPPER_BOUND`, and `NUM_PARTITIONS` if you want to utilize concurrency in Spark. With these parameters, the original query would be split into `NUM_PARTITIONS` number of sub-queries that can be executed by Spark tasks concurrently. + `PARTITION_FIELD`: the name of the field to be used to partition the query. + `LOWER_BOUND`: an **inclusive** lower bound value of the chosen partition field. For the Datetime field, we accept the Spark timestamp format used in Spark SQL queries. Examples of valid value: ``` "2024-05-01T20:55:02.000Z" ``` + `UPPER_BOUND`: an **exclusive** upper bound value of the chosen partition field. + `NUM_PARTITIONS`: the number of partitions. Example: ``` twilio_read = glueContext.create_dynamic_frame.from_options( connection_type="twilio", connection_options={ "connectionName": "connectionName", "ENTITY_NAME": "sms-message", "API_VERSION": "2010-04-01", "PARTITION_FIELD": "date_sent" "LOWER_BOUND": "2024-05-01T20:55:02.000Z" "UPPER_BOUND": "2024-06-01T20:55:02.000Z" "NUM_PARTITIONS": "10" } ``` # Twilio connection options The following are connection options for Twilio: + `ENTITY_NAME`(String) - (Required) Used for Read. The name of your object in Twilio. + `EDGE_LOCATION`(String) - (Required) A valid Twilio edge location. + `API_VERSION`(String) - (Required) Used for Read. Twilio Rest API version you want to use. Twilio supports two API versions: ‘v1’ and ‘2010-04-01’. + `SELECTED_FIELDS`(List) - Default: empty(SELECT \$1). Used for Read. Columns you want to select for the object. + `FILTER_PREDICATE`(String) - Default: empty. Used for Read. It should be in the Spark SQL format. + `QUERY`(String) - Default: empty. Used for Read. Full Spark SQL query. + `PARTITION_FIELD`(String) - Used for Read. Field to be used to partition query. + `LOWER_BOUND`(String)- Used for Read. An inclusive lower bound value of the chosen partition field. + `UPPER_BOUND`(String) - Used for Read. An exclusive upper bound value of the chosen partition field. + `NUM_PARTITIONS`(Integer) - Default: 1. Used for Read. Number of partitions for Read. + `INSTANCE_URL`(String) - (Required) Used for Read. A valid Twilio instance URL. # Limitations and notes for Twilio connector The following are limitations or notes for the Twilio connector: + Record-based partitioning is not supported, as there is no provision to retrieve the total count of records from Twilio. + The fields `date_sent`, `start_time`, and `end_time` are of the Datetime datatype, but when filtering, they only support date values (time components are not considered). + Filtering the "from" or "to" fields works only if the values do not include any prefix (for example, a protocol or label). If a prefix is present, filtering for the respective field does not work. For example if you pass "to": "whatsapp:\$114xxxxxxxxxx" as a filter then Twilio won't return a response. You need to pass it as "to": "\$114xxxxxxxx", then it will return records if they exist. + The "identity" field filter is mandatory when querying the `conversation-participant-conversation` entity.