# Bringing Amazon Redshift data into the AWS Glue Data Catalog
You can manage analytic data in Amazon Redshift data warehouses in the AWS Glue Data Catalog (Data Catalog), and unify Amazon S3 data lakes and Amazon Redshift data warehouses. Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the AWS Cloud. An Amazon Redshift data warehouse is a collection of computing resources called *nodes*, which are organized into a group called a *cluster*. Each cluster runs an Amazon Redshift engine and contains one or more databases.
In Amazon Redshift, you can create Amazon Redshift provisioned clusters and serverless namespaces, and register them with the Data Catalog. By doing this, you can unify data in Amazon Redshift managed storage (RMS) and Amazon S3 buckets, and access data from Apache Iceberg compatible analytical engines.
By registering namespaces and clusters, you can provide access to data without the need to copy it or move it. For more information about registering clusters and namespaces in Amazon Redshift, see [Registering Amazon Redshift clusters and namespaces to the AWS Glue Data Catalog](https://docs.aws.amazon.com/redshift/latest/dg/iceberg-integration-register.html).
In Amazon Redshift, you can perform data sharing through datashares or by registering namespaces and clusters with Data Catalog. With datashares, which operate at the individual database object level, you have to enable sharing for each table or view. In contrast, namespace publishing functions at the cluster or namespace level. When you register a cluster or namespace with the Data Catalog, all databases and tables within it are automatically shared, without you having to configure sharing for individual objects.
In the Data Catalog, you can create a federated catalog for each namespace or cluster. A catalog is referred to as a *federated catalog* when it points to an entity outside of the Data Catalog. Tables and views in the Amazon Redshift namespace are listed as individual tables in the Data Catalog. You can share databases and tables in the federated catalog with selected IAM principals and SAML users within the same account, or in another account with Lake Formation. You can also include row and column filter expressions to restrict access to certain data. For more information, see [Data filtering and cell-level security in Lake Formation](data-filtering.md).
The Data Catalog supports a three-level metadata hierarchy comprising catalogs, databases, and tables (and views). When you register a namespace with the Data Catalog, the Amazon Redshift data hierarchy is mapped to the Data Catalog's 3-level hierarchy as follows:
+ The Amazon Redshift namespace becomes a multi-level catalog in the Data Catalog.
+ The associated Amazon Redshift database is registered as a catalog in the Data Catalog.
+ The Amazon Redshift schema becomes a database in the Data Catalog.
+ The Amazon Redshift table becomes a table in the Data Catalog.
![\[Shows the catalog-level mapping between the Amazon Redshift namespace and the Data Catalog.\]](http://docs.aws.amazon.com/lake-formation/latest/dg/images/rs-catalog-mapping.png)
With this three-level metadata hierarchy, you can access Amazon Redshift tables by using the 3-part notation - "catalog1/catalog2.database.table" in the Data Catalog. Also, data teams can maintain the same organization that Amazon Redshift uses to organize tables within the Data Catalog account.
In Lake Formation, you can securely manage the data from Amazon Redshift using fine-grained access control for the Data Catalog resources. With this integration, you can manage, secure, and query analytical data from a single catalog with a common access control mechanism.
For limitations, see [Limitations for bringing Amazon Redshift data warehouse data into the AWS Glue Data Catalog](notes-ns-catalog.md).
**Topics**
+ [Key benefits](#namespace-publish-benefits)
+ [Roles and responsibilities](#namespace-roles-responsibilities)
+ [Prerequisites for managing Amazon Redshift namespaces in the AWS Glue Data Catalog](redshift-ns-prereqs.md)
+ [Creating Amazon Redshift federated catalogs](create-ns-catalog.md)
+ [Viewing catalog objects](view-ns-catalog-resources.md)
+ [Updating a federated catalog](update-fed-catalog-steps.md)
+ [Accessing a shared federated catalog](catalog-resource-link.md)
+ [Deleting a federated catalog](delete-rs-fed-catalog.md)
+ [Querying federated catalogs](query-redshift-fed-catalog.md)
+ [Additional resources](additional-resources-byod.md)
## Key benefits
Registering Amazon Redshift clusters and namespaces with the AWS Glue Data Catalog and unifying data across Amazon S3 data lakes and Amazon Redshift data warehouses, offers the following benefits:
+ **Uniform querying experience** – Query your Amazon Redshift managed data and data in the Amazon S3 buckets using any query engine compatible with Apache Iceberg, such as Amazon EMR Serverless and Amazon Athena without having to move or copy data.
+ **Consistent data access across services** – You don't need to update database and table names in your data pipelines when accessing the same federated data sources from different AWS analytics services, as the data sources are registered in the Data Catalog.
+ **Fine-grained access control **– You can apply Lake Formation permissions to manage access to the federated data sources using fine-grained access control permissions.
## Roles and responsibilities
| | |
| --- |--- |
| Role | Responsibility |
| Amazon Redshift producer cluster administrator | Registers the cluster or namespace with the Data Catalog. |
| Lake Formation data lake administrator | Accepts the cluster or namespace invitation, creates federated catalogs, and grants access on the federated catalogs to other principals. |
| Lake Formation read only administrator | Discovers the federated catalog, queries Amazon Redshift tables in the federated catalog. |
| Data transfer role | Amazon Redshift assumes on your behalf to transfer data to and from the Amazon S3 bucket. |
The following are the high-level steps to provide users access to an Amazon Redshift namespace:
1. In Amazon Redshift, the producer cluster administrator registers a cluster or namespace with the Data Catalog.
1. The data lake administrator accepts the namespace invitation from the Amazon Redshift producer cluster administrator, and creates a federated catalog in the Data Catalog.
After completing this step, you can manage the Amazon Redshift namespace catalog within the Data Catalog.
1. Grant permissions to users on catalogs, databases and tables. You can share the entire namespace catalog or a subset of tables with users in the same account or another account.
# Prerequisites for managing Amazon Redshift namespaces in the AWS Glue Data Catalog
1. Create a data lake administrator - Create an IAM role that is authorized to accept the namespace invitation, and creates the AWS Glue Data Catalog objects (catalogs, databases, tables/views), and grant Lake Formation permissions to other users.
For step-by-step instructions on creating a data lake administrator, see [Create a data lake administrator](initial-lf-config.md#create-data-lake-admin).
1. Update data lake administrator permissions.
In addition to data lake administrator permissions, the data lake administrator requires the following permissions to accept an Amazon Redshift namespace invitation in Lake Formation, create or update the Data Catalog resources, and enable data lake access:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Id": "glue-enable-datalake-access",
"Statement": [{
"Effect": "Allow",
"Action": [
"redshift:AssociateDataShareConsumer",
"redshift:DescribeDataSharesForConsumer",
"redshift:DescribeDataShares",
"redshift-serverless:CreateNamespace",
"redshift-serverless:CreateWorkgroup",
"redshift-serverless:DeleteNamespace",
"redshift-serverless:DeleteWorkgroup",
"ec2:DescribeAccountAttributes",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeAvailabilityZones",
"s3:createBucket",
"s3:deleteBucket",
"s3:putBucketPolicy",
"s3:putEncryptionConfiguration",
"s3:putLifecycleConfiguration",
"s3:putBucketVersioning",
"iam:CreateRole"
],
"Resource": "*"
},
{
"Action": [
"iam:PassRole"
],
"Effect": "Allow",
"Resource": "arn:aws:iam::*:role/data transfer role name",
"Condition": {
"StringLike": {
"iam:PassedToService": [
"glue.amazonaws.com"
]
}
}
}
]
}
```
------
1. If the IAM role used for creating federated catalogs is not a data lake administrator, you need to grant the role the `Create catalog` permission.
**To create catalog creators**
1. Open the Lake Formation console at [https://console.aws.amazon.com/lakeformation/](https://console.aws.amazon.com/lakeformation/).
1. Choose **Administrative roles and tasks** under **Administration**.
1. Choose **Grant.**
1. On the **Grant permissions** screen, choose an IAM user or role.
1. Select **Create catalog** permission.
1. Optionally, you can also grant grantable **Create catalog** permission. Grantable permission allows the catalog creator to grant the `Create catalog` permission to other principals.
1. Choose **Grant**.
AWS CLI example for granting permissions to create a federated catalog.
```
aws lakeformation grant-permissions \
--cli-input-json \
'{
"Principal": {
"DataLakePrincipalIdentifier":"arn:aws:iam::123456789012:role/Admin"
},
"Resource": {
"Catalog": {
}
},
"Permissions": [
"CREATE_CATALOG",
"DESCRIBE"
]
}'
```
1. Create a read only administrator role to discover the Amazon Redshift federated catalogs in the Data Catalog from Amazon Redshift Query Editor v2.
To query the Amazon Redshift tables in the federated catalog from Amazon Redshift Query Editor v2, ensure that the Read only administrator role policy contains the ARN for the Amazon Redshift service-linked role-`AWSServiceRoleForRedshift`.
```
aws lakeformation put-data-lake-settings
--region us-east-1 \
--data-lake-settings \
'{
"DataLakeAdmins": [{"DataLakePrincipalIdentifier":"arn:aws:iam::123456789012:role/Admin"}],
"ReadOnlyAdmins":[{"DataLakePrincipalIdentifier":"arn:aws:iam::123456789012:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift"}],
"CreateDatabaseDefaultPermissions":[],
"CreateTableDefaultPermissions":[],
"Parameters":{"CROSS_ACCOUNT_VERSION":"4","SET_CONTEXT":"TRUE"}
}'
```
1. Create a data transfer role that Amazon Redshift can assume on your behalf to transfer data to and from the Amazon S3 bucket.
When you enable data lake access for Apache Iceberg compatible query engines such as Athena, Amazon EMR on Amazon EC2 to access the Amazon Redshift resources in the Data Catalog, you need to create an IAM role with the required permissions to perform data transfer to and from the Amazon S3 bucket.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Id": "glue-enable-datalake-access",
"Statement": [{
"Sid": "DataTransferRolePolicy",
"Effect": "Allow",
"Action": [ "glue:GetCatalog",
"glue:GetDatabase",
"kms:GenerateDataKey",
"kms:Decrypt"],
"Resource": "*"
}
]
}
```
------
1. Add the following trust policy to the data transfer role for AWS Glue and Amazon Redshift services to assume the role to transfer data to and from the Amazon S3 bucket.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": [
"redshift.amazonaws.com",
"glue.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}]
}
```
------
1. Add the following key policy to the AWS KMS key if you're using a customer managed key to encrypt the data in the Amazon Redshift cluster/namespace. Replace the account number with a valid AWS account number, and specify data transfer role name. By default, the data in the Amazon Redshift cluster is encrypted using an KMS key. Lake Formation provides an option to create your custom KMS key for encryption. If you're using a customer managed key, you must add specific key policies to the key.
For more information about managing the permissions of a customer managed key, see [Customer managed keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk).
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Id": "auto-redshift-3",
"Statement": [{
"Sid": "RedshiftAllowAccessPolicy",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:CreateGrant",
"kms:DescribeKey"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:CallerAccount": "111122223333",
"kms:ViaService": "redshift.us-east-1.amazonaws.com"
}
}
},
{
"Sid": "RedshiftServerlessAllowAccessPolicy",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:CreateGrant",
"kms:DescribeKey"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:CallerAccount": "111122223333",
"kms:ViaService": "redshift-serverless.us-east-1.amazonaws.com"
}
}
},
{
"Sid": "DirectMetadataAccess",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:root"
},
"Action": [
"kms:Describe*",
"kms:Get*",
"kms:List*",
"kms:RevokeGrant"
],
"Resource": "*"
},
{
"Sid": "GenerateDataKeyDecryptDataTransferRole",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/data-transfer-role-name"
},
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:ViaService": "s3.us-east-1.amazonaws.com"
}
}
}
]
}
```
------
# Creating Amazon Redshift federated catalogs
This topic describes the steps you need to follow to accept a cluster or namespace invitation, create a federated multi-level catalog, and grant permissions to other principals. You can complete these tasks using the Lake Formation console, the AWS Command Line Interface (AWS CLI), or APIs/SDKs. The examples in this topic show the producer cluster/namespace, the Data Catalog, and the data consumer in the same account.
To learn more about Lake Formation cross-account capabilities, see [Cross-account data sharing in Lake Formation](cross-account-permissions.md).
**To manage a Amazon Redshift namespace in the Data Catalog**
1. Review a namespace invitation and accept it.
------
#### [ Console ]
1. Sign in to the Lake Formation console as a data lake administrator at [https://console.aws.amazon.com/lakeformation/](https://console.aws.amazon.com/lakeformation/). Navigate to the **Catalogs** page under **Data Catalog**.
1. Review the namespace invitation that you're authorized to access. The **Status** column indicates your current participation status for the namespace. The **Not accepted** status indicates that you have been added to the namespace, but you have not yet accepted it or have rejected the invitation.
![\[The catalogs page with pending invitations.\]](http://docs.aws.amazon.com/lake-formation/latest/dg/images/catalog-details.png)
1. To respond to a namespace or cluster invitation, select the invitation name and choose **Review invitation**. In **Accept or reject invitation**, review the invitation details. Choose **Accept** to accept the invitation or **Reject** to decline the invitation. You don't get access to the namespace if you reject the invitation.
------
#### [ AWS CLI ]
The following examples show how to view, accept, and register the invitation. Replace the AWS account ID with a valid AWS account ID. Replace the `data-share-arn` with the actual Amazon Resource Name (ARN) that references the namespace.
1. View a pending invitation.
```
aws redshift describe-data-shares \
--data-share-arn 'arn:aws:redshift:us-east-1:123456789012:datashare:abcd1234-1234-ab12-cd34-1a2b3c4d5e6f/ds_internal_namespace' \
```
1. Accept an invitation.
```
aws redshift associate-data-share-consumer \
--data-share-arn 'arn:aws:redshift:us-east-1:123456789012:datashare:abcd1234-1234-ab12-cd34-1a2b3c4d5e6f/ds_internal_namespace' \
--consumer-arn 'arn:aws:glue:us-east-1:123456789012:catalog'
```
1. Register the cluster or namespace in the Lake Formation account. Use the [RegisterResource](https://docs.aws.amazon.com/lake-formation/latest/APIReference/API_RegisterResource.html) API operation to register the datashare in Lake Formation. `DataShareArn` is the input parameter for `ResourceArn`.
**Note**
This is a mandatory step.
```
aws lakeformation register-resource \
--resource-arn 'arn:aws:redshift:us-east-1:123456789012:datashare:abcd1234-1234-ab12-cd34-1a2b3c4d5e6f/ds_internal_namespace'
```
------
1. Create a federated catalog.
After you’ve accepted an invitation, you need to create a federated catalog in the Data Catalog that maps the objects in the Amazon Redshift namespace to the Data Catalog. You must be a data lake administrator or a user or role that has required permissions to create a catalog.
------
#### [ Console ]
1. After accepting the namespace **Invitation**, the **Set catalog details** page is displayed.
1. On the **Set catalog details** page, enter a unique name for the catalog. Use lower case for catalog names. Catalog names must be less than or equal to 255 characters long. You use this identifier for mapping the namespace internally in the metadata hierarchy (catalogid.dbName.schema.table).
1. Enter a description for the catalog. Description must be less than or equal to 2048 characters long.
1. Next, choose the **Access this catalog from Iceberg compatible engines** check box to enable accessing the Amazon Redshift resources using Apache Iceberg compatible analytical engines such as Athena and Apache Spark on Amazon EMR.
You don't need to enable data lake access to access the federated catalogs using Amazon Redshift.
![\[The create catalog page with access enablement options.\]](http://docs.aws.amazon.com/lake-formation/latest/dg/images/catalog-access.png)
1. To enable these query engines to read and write to Amazon Redshift namespaces, AWS Glue creates a managed Amazon Redshift cluster with the compute and storage resources required to perform read and write operations without impacting Amazon Redshift data warehouse workloads.
You also need to provide an IAM role with the permissions required to transfer data to and from the Amazon S3 bucket.
1. By default, the data in the Amazon Redshift cluster is encrypted using an AWS managed key. Lake Formation provides an option to create your custom KMS key for encryption. If you're using a customer managed key, you must add specific key policies to the key.
Choose the **Customize encryption settings** if you're using a customer managed key to encrypt the data in the Amazon Redshift cluster/namespace. To use a custom key, you must add additional custom managed key policy to your KMS key. For more information, see [Prerequisites for managing Amazon Redshift namespaces in the AWS Glue Data Catalog](redshift-ns-prereqs.md).
------
#### [ AWS CLI ]
Use the following example code to create a catalog with the Amazon Redshift data published to the Data Catalog using the AWS CLI.
```
aws glue create-catalog
--cli-input-json \
'{
"Name": "nscatalog",
"CatalogInput": {
"Description": "Redshift federated catalog",
"CreateDatabaseDefaultPermissions" : [],
"CreateTableDefaultPermissions": [],
"FederatedCatalog": {
"Identifier": "arn:aws:redshift:us-east-1:123456789012:datashare:11524d7f-f56d-45fe-83f7-d7bb0a4d6d71/ds_internal_namespace",
"ConnectionName": "aws:redshift"
},
"CatalogProperties": {
"DataLakeAccessProperties" : {
"DataLakeAccess" : true,
"DataTransferRole" : "arn:aws:iam::123456789012:role/DataTransferRole"
}
}
}
}'
```
------
1. Grant permissions to users in your account or in external accounts.
------
#### [ AWS Management Console ]
1. Choose **Next** to grant permissions to other users on the shared catalogs, databases, and tables.
1. On the **Add permissions** screen, choose the principals and the types of permissions to grant.
![\[The catalog permissions page with principal type and grant options.\]](http://docs.aws.amazon.com/lake-formation/latest/dg/images/catalog-permissions.png)
1. In the **Principals** section, choose a principal type and then specify principals to grant permissions.
+ **IAM users and roles** – Choose one or more users or roles from the IAM users and roles list.
+ ** SAML users and groups** – For SAML and Amazon Quick users and groups, enter one or more Amazon Resource Names (ARNs) for users or groups federated through SAML, or ARNs for Amazon Quick users or groups. Press **Enter** after each ARN.
For information about how to construct the ARNs, see AWS CLI grant and revoke AWS CLI commands.
+ **External accounts** – For AWS, AWS organization, or IAM Principal enter one or more valid AWS account IDs, organization IDs, organizational unit IDs, or ARN for the IAM user or role. Press Enter after each ID. An organization ID consists of "o-" followed by 10–32 lower-case letters or digits. An organizational unit ID starts with "ou-" followed by 4–32 lowercase letters or digits (the ID of the root that contains the OU). This string is followed by a second "-" dash and 8 to 32 additional lowercase letters or digits.
1. In the **Permissions** section, select permissions and grantable permissions.
Under **Catalog permissions**, select one or more permissions to grant. Under **Grantable permissions**, select the permissions that the grant recipient can grant to other principals in their AWS account. This option is not supported when you are granting permissions to an IAM principal from an external account.
Choose **Super user** to grant the user unrestricted permissions to the resources (databases, tables, views) within the catalog.
1. Choose **Add**.
------
#### [ AWS CLI ]
Use the following examples to grant catalog, database, and table permissions using the AWS CLI:
+ The following example shows how to grant permissions on the federated catalog.
```
aws lakeformation grant-permissions
--cli-input-cli-json \
'{
"Principal": {
"DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/non-admin"
},
"Resource": {
"Catalog": {
"Id": "123456789012:nscatalog"
}
},
"Permissions": [
"DESCRIBE","CREATE_CATALOG"
],
"PermissionsWithGrantOption": [
]
}'
```
+ Use the following example to grant permissions on a database.
```
aws lakeformation grant-permissions \
--cli-input-json \
'{
"Principal": {
"DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/non-admin"
},
"Resource": {
"Database": {
"CatalogId": "123456789012:nscatalog/dev",
"Name": "public"
}
},
"Permissions": [
"ALL"
]
}'
```
+ The following example shows how to grant permissions on a table in the Amazon Redshift database.
```
aws lakeformation grant-permissions \
--cli-input-json \
'{
"Principal": {
"DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/non-admin"
},
"Resource": {
"Table": {
"CatalogId": "123456789012:nscatalog2/dev",
"DatabaseName": "public",
"TableWildcard" : {}
}
},
"Permissions": [
"ALL"
]
}'
```
------
1. Choose **Next** to review the catalog details and create a federated catalog. The newly created federated catalog and the catalog objects appear in the **Catalogs** page.
An Amazon Redshift federated catalog is referenced with `catalogID = 123456789012:Redshift-federated catalog id`.
# Viewing catalog objects
After you create the federated catalog, you can view the objects in the catalog using the Lake Formation console or AWS CLI.
------
#### [ AWS Management Console ]
1. Open the Lake Formation console at [https://console.aws.amazon.com/lakeformation/](https://console.aws.amazon.com/lakeformation/).
1. Choose **Catalogs** under Data Catalog.
1. Choose a federated catalog from the list on the **Catalogs** page.
1. The catalog summary page shows the catalog objects (databases and tables) that you have permissions on. The **Permissions** tab shows the IAM principals who has been granted permissions on these objects.
------
#### [ AWS CLI ]
+ The following AWS CLI example shows how to request the top-level catalog.
```
aws glue get-catalog \
--catalog-id 123456789012:nscatalog
```
*Response*
```
{
"Catalog": {
"CatalogId": "123456789012:nscatalog",
"Name": "nscatalog",
"ResourceArn": "arn:aws:glue:us-east-1:123456789012:catalog/nscatalog",
"Description": "Redshift published Catalog",
"CreateTime": "2024-09-05T14:49:16-07:00",
"FederatedCatalog": {
"Identifier": "arn:aws:redshift:us-east-1:123456789012:datashare:b1234589-e823-4a14-ad8e-077085540a50/ds_internal_namespace",
"ConnectionName": "aws:redshift"
},
"CatalogProperties": {
"DataLakeAccessProperties": {
"DataLakeAccess": true,
"DataTransferRole": "arn:aws:iam::123456789012:role/DataTransferRole",
"KmsKey": "AWS_OWNED_KMS_KEY",
"ManagedWorkgroupName": "123456789012:nscatalog",
"ManagedWorkgroupStatus": "AVAILABLE",
"RedshiftDatabaseName": "dev"
}
},
"CatalogIdentifier": "e2309c2c2fb048f1a3069dfdc1c7883e",
"CreateTableDefaultPermissions": [],
"CreateDatabaseDefaultPermissions": []
}
}
```
+ The following example shows how to request all catalogs in the account.
```
aws glue get-catalogs \
--recursive
```
+ The following example request shows how to get a Amazon Redshift database-level catalog.
```
aws glue get-catlog \
--catalog-id 123456789012:namespace catalog name/redshift database name
```
+ The following example request shows how to get the databases in the Amazon Redshift database-level catalog.
```
aws glue get-databases \
--catalog-id 123456789012:namespace catalog name/redshift database name
```
+ The following example request shows how to get a Amazon Redshift table in the catalog.
```
aws glue get-table \
--catalog-id 123456789012:parent catalog name/redshift database \
--database-name redshift schema name \
--name table name
```
+ Following example shows how to get all tables from the Amazon Redshift database.
```
aws glue get-tables \
--catalog-id 123456789012:namespace catalog name/redshift database name \
--database-name RS schema name
```
------
# Updating a federated catalog
You can update a Amazon Redshift federated catalog in the Data Catalog using the Lake Formation console, the AWS CLI or the [UpdateCatalog](https://docs.aws.amazon.com/glue/latest/webapi/API_UpdateCatalog.html) API operation.
------
#### [ AWS Management Console ]
Follow these steps to update your federated catalog using Lake Formation console.
1. Sign in to the AWS Management Console, and open the Lake Formation console at [https://console.aws.amazon.com/lakeformation/](https://console.aws.amazon.com/lakeformation/).
1. In the left navigation pane, choose **Catalogs** under **Data Catalog**.
1. On the **Catalogs** page, choose the Amazon Redshift federated catalog you want to update.
1. Under **Actions**, choose **Edit**.
1. On the **Set Catalog details** screen, under the **Access from engines** section, choose **Access this catalog from Iceberg compatible engines**. Checking this option will enable data lake access for Apache Iceberg compatible query engines.
1. Next, create a new IAM role or choose an existing IAM role that has the policy that grants permissions to perform data transfer to and from the Amazon S3 bucket.
For more information on permissions, see [Prerequisites for managing Amazon Redshift namespaces in the AWS Glue Data Catalog](redshift-ns-prereqs.md).
1. By default, your data in the Amazon Redshift cluster is encrypted using an AWS managed key. If you choose to encrypt data using a customer managed key, either create a KMS key or choose an existing key that has the permissions defined in the [Prerequisites for managing Amazon Redshift namespaces in the AWS Glue Data Catalog](redshift-ns-prereqs.md) section.
1. Choose **Save**.
Upon successful completion, the **Catalog details** page shows the managed workgroup name with the status as "Success".
------
#### [ AWS CLI ]
The following is an example of the `update-catalog` CLI input with the data lake access disabled by setting the `DataLakeAacess` parameter value as `false`.
```
aws glue update-catalog --cli-input-json \
'{
"Name": "nscatalog",
"CatalogInput": {
"Description": "Redshift published catalog",
"CreateDatabaseDefaultPermissions" : [],
"CreateTableDefaultPermissions": [],
"FederatedCatalog": {
"Identifier": "arn:aws:redshift:us-east-1:123456789012:datashare:11524d7f-f56d-45fe-83f7-d7bb0a4d6d71/ds_internal_namespace",
"ConnectionName": "aws:redshift"
},
"CatalogProperties": {
"DataLakeAccessProperties" : {
"DataLakeAccess" : false
}
}
}
}'
```
------
# Accessing a shared federated catalog
AWS Lake Formation cross-account capabilities allow users to securely share distributed data lakes across multiple AWS accounts, AWS organizations, or directly with IAM principals in another account providing fine-grained access to the metadata and underlying data.
Lake Formation uses the AWS Resource Access Manager (AWS RAM) service to facilitate resource sharing. When you share a catalog resource with another account, AWS RAM sends an invitation to the grantee account to accept or reject the resource grant.
Integrated analytical services such as Amazon Athena and Redshift Spectrum require resource links to be able to include shared resources in queries. Principals need to create a resource link in their AWS Glue Data Catalog to a shared resource from another AWS account. For more information about resource links, see [How resource links work in Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/resource-links-about.html).
A *Catalog link container* is a Data Catalog object, which references a local or cross-account federated database-level catalog from other AWS accounts. You can also create database links and table links within a catalog link container. When you create a database link or a table link, you must specify a target resource that resides under the same target Amazon Redshift database-level catalog (Amazon Redshift database).
To create a catalog link container, you need the Lake Formation `CREATE_CATALOG` or the `glue:CreateCatalog` permission.
You must have **Cross account version settings** version 4 or higher for sharing databases or tables in the federated catalog across AWS accounts.
## Creating a catalog link container to a cross-account federated catalog
You can create a catalog link container that points to a Redshift database-level federated catalog in any AWS Region by using the AWS Lake Formation console, AWS Glue `CreateCatalog` API, or AWS Command Line Interface (AWS CLI).
**To create a catalog link container to a shared catalog (console)**
1. Open the AWS Lake Formation console at [https://console.aws.amazon.com/lakeformation/](https://console.aws.amazon.com/lakeformation/). Sign in as a principal who has the Lake Formation `CREATE_CATALOG` permission.
1. In the navigation pane, choose **Catalogs**, and then choose **Create catalog**.
1. On the **Set catalog details** page, provide the following information:
**Name**
Enter a name that adheres to the same rules as a catalog name. The name can be the same as the target shared catalog.
**Type**
Choose **Catalog link container** as the type of catalog.
**Source**
Choose `Redshift`.
**Target Redshift catalog**
Select a Redshift database-level federated catalog or choose a local (owned) catalog from the list.
The list contains all the catalogs shared with your account. Note the catalog owner account ID is listed with each catalog. If you don't see a catalog that you know was shared with your account, check the following:
+ If you aren't a data lake administrator, check that the data lake administrator granted you Lake Formation permissions on the catalog.
+ If you are a data lake administrator, and your account is not in the same AWS organization as the granting account, ensure that you have accepted the AWS Resource Access Manager (AWS RAM) resource share invitation for the catalog. For more information, see [Accepting a resource share invitation from AWS RAM](accepting-ram-invite.md).
When creating a catalog link container through the console, the **Target Redshift Catalog** dropdown might display `No matches` when attempting to select a cross-account Redshift catalog. Despite this display, you can manually enter the target ARN of the Amazon Redshift federated database-level catalog (Amazon Redshift database) in the input field, and the form will still work correctly. For example: `arn:aws:glue:us-east-1:123456789012:catalog/federated-catalog-redshift/dev`.
This behavior occurs because the console can only search for potential candidates within the currently signed-in account. The dropdown is intended as an auto-complete feature, but you can still manually input ARNs for cross-account access.
1. To enable Apache Iceberg query engines to read and write to Amazon Redshift namespaces, AWS Glue creates a managed Amazon Redshift cluster with the compute and storage resources required to perform read and write operations without impacting Amazon Redshift data warehouse workloads. You need to provide an IAM role with the permissions required to transfer data to and from the Amazon S3 bucket.
1. Choose **Next**.
1. (Optional) Choose **Add permissions** to grant permissions to other principals.
However, granting permissions on a catalog link container doesn't grant permissions on the target (linked) catalog. You must grant permissions on the target catalog separately for the catalog link to be visible in Athena.
1. Next, review the catalog link container details and choose **Create catalog**.
You can then view the link container name under the **Catalogs** page.
Now, you can create database links and table links in the catalog link container to enable access from query engines.
**Create a catalog link container CLI example**
+ In the following example, the `TargetRedshiftCatalog` object specifies the arn of the Amazon Redshift federated database-level catalog (Amazon Redshift database). The `DataLakeAccess` must be enabled when you create the catalog link container.
```
aws glue create-catalog \
--cli-input-json
'{
"Name": "linkcontainer",
"CatalogInput": {
"TargetRedshiftCatalog": {
"CatalogArn": "arn:aws:us-east-1:123456789012:catalog/nscatalog/dev"
},
"CatalogProperties": {
"DataLakeAccessProperties" : {
"DataLakeAccess" : true,
"DataTransferRole" : "arn:aws:iam::111122223333:role/DataTransferRole"
}
}
}
}'
```
## Creating resource links under the catalog link container
You can create resource links to databases and tables links under a catalog link container. When you create database resource links or table resource links, you must specify a target resource that resides under the same target Amazon Redshift database-level catalog (Amazon Redshift database) that the link container points to.
You can create a resource link to a shared Amazon Redshift database or a table by using the AWS Lake Formation console, API, or AWS Command Line Interface (AWS CLI).
+ For detailed instructions, see [Creating a resource link to a shared Data Catalog database](create-resource-link-database.md).
Following is a AWS CLI example to create a database resource link under a catalog link container.
```
aws glue create-database \
--cli-input-json \
'{
"CatalogId": "111122223333:linkcontainer",
"DatabaseInput": {
"Name": "dblink",
"TargetDatabase": {
"CatalogId": "123456789012:nscatalog/dev",
"DatabaseName": "schema1"
}
}
}'
```
+ To create a table resource link under a catalog link container, you need to first create a AWS Glue database in the local AWS Glue Data Catalog to contain the table resource link.
For more information on creating resource links to shared tables, see [Creating a resource link to a shared Data Catalog table](create-resource-link-table.md).
+ Create a database to contain the table resource link example
```
aws glue create-database \
--cli-input-json \
'{
"CatalogId": "111122223333:linkcontainer",
"DatabaseInput": {
"Name": "db1",
"Description": "creating parent database for table link"
}
}'
```
+ Create table resource link example
```
aws glue create-table \
--cli-input-json \
'{
"CatalogId": "111122223333:linkcontainer",
"DatabaseName": "db1",
"TableInput": {
"Name": "tablelink",
"TargetTable": {
"CatalogId": "123456789012:nscatalog/dev",
"DatabaseName": "schema1",
"Name": "table1"
}
}
}'
```
# Deleting a federated catalog
You can delete the federated catalogs that you created in the AWS Glue Data Catalog using the `glue:DeleteCatalog` operation or the AWS Lake Formation console.
**To delete a federated catalog (console)**
1. Open the Lake Formation console at [https://console.aws.amazon.com/lakeformation/](https://console.aws.amazon.com/lakeformation/).
1. In the navigation pane, choose **Catalogs** under **Data Catalog**.
1. Choose the catalog that you want to delete from the catalogs list.
1. Choose **Delete** from **Actions**.
1. Choose **Drop** to confirm and the federated catalog will be deleted from the Data Catalog.
![\[The delete catalog confirmation.\]](http://docs.aws.amazon.com/lake-formation/latest/dg/images/delete-fed-catalog.png)
**To delete a federated catalog (CLI)**
+
```
aws glue delete-catalog
--catalog-id 123456789012:catalog name
```
# Querying federated catalogs
After you grant permissions to other principals, they can sign in and start querying the tables in the federated catalogs by logging into the SQL tools using Amazon Redshift, Amazon EMR, Amazon Athena, and AWS Glue ETL.
For more information on connecting to the AWS Glue Data Catalog using Apache Iceberg Rest extension endpoint or standalone Spark application, see [Accessing the AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/access_catalog.html) section in the AWS Glue Developer Guide.
You can use the data definition language (DDL) queries to create and manage tables in the database using Apache Spark on Amazon EMR. To create and delete tables in the Amazon Redshift database, the principal must have Lake Formation `Create table`, `Drop` permissions.
For more information on granting Data Catalog permissions, see [Granting permissions on Data Catalog resources](granting-catalog-permissions.md).
For more information on querying the catalog resources from Amazon Athena, see [Querying AWS Glue Data Catalog from Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/gdc-register.html) in Amazon Athena User Guide.
# Additional resources
You can use [Amazon SageMaker Lakehouse](https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/lakehouse.html) to achieve unified access to data in both data warehouses and data lakes. Through SageMaker Lakehouse, you can use preferred analytics, machine learning, and business intelligence engines through an open, Apache Iceberg REST API to help ensure secure access to data with consistent, fine-grained access controls.
+ [Amazon SageMaker workshop ](https://catalog.us-east-1.prod.workshops.aws/workshops/107188af-3663-4bbf-bb35-93d514d406da/en-US/03lakehouse)
+ [ Simplify data access for your enterprise using Amazon SageMaker Lakehouse ](https://aws.amazon.com/blogs/big-data/simplify-data-access-for-your-enterprise-using-amazon-sagemaker-lakehouse/)