# Deliver data to Apache Iceberg Tables with Amazon Data Firehose Apache Iceberg is a high-performance open-source table format for performing big data analytics. Apache Iceberg brings the reliability and simplicity of SQL tables to Amazon S3 data lakes, and makes it possible for open-source analytics engines like Spark, Flink, Trino, Hive, and Impala to work with the same data concurrently. For more information, see [Apache Iceberg](https://iceberg.apache.org) and [Considerations and limitations](apache-iceberg-considerations.md). You can use Firehose to deliver streaming data to Apache Iceberg Tables in Amazon S3. Your Apache Iceberg Tables can be in self-managed in Amazon S3 or hosted in Amazon S3 Tables. In self-managed Iceberg tables, you manage all the table optimizations such as compaction, and snapshot expiration. Amazon S3 Tables provide storage that is optimized for large-scale analytics workloads, with features that continuously improve query performance and reduce storage costs for tabular data. For more information on Amazon S3 Tables, see [Amazon S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html). This feature allows you to route records from a single stream into different Apache Iceberg Tables. You can automatically apply insert, update, and delete operations to records in those tables. It also supports fine-grained data access control on Apache Iceberg tables in Amazon S3 with AWS Lake Formation. You can specify access controls centrally in AWS Lake Formation and provide more granular table-level and column-level permissions for Firehose. # Considerations and limitations **Note** Firehose supports Apache Iceberg Tables as a destination in all [AWS Regions](https://docs.aws.amazon.com/glossary/latest/reference/glos-chap.html?icmpid=docs_homepage_addtlrcs#region) except China Regions, AWS GovCloud (US) Regions, Asia Pacific (Taipei), Asia Pacific (Malaysia), Asia Pacific (New Zealand), and Mexico (Central). Firehose support for Apache Iceberg tables has the following considerations and limitations. + **Throughput **– If you use **Direct PUT** as the source to deliver data to Apache Iceberg tables, then the maximum throughput per stream is 5 MiB/second in US East (N. Virginia), US West (Oregon), and Europe (Ireland) Regions and 1 MiB/second in all other AWS Regions. If you want to insert data to Iceberg tables with no updates and deletes and you want higher throughput for your stream, then you can use the [Firehose Limits form](https://support.console.aws.amazon.com/support/home#/case/create%3FissueType=service-limit-increase%26limitType=kinesis-firehose-limits) to request a throughput limit increase. You can also set the `AppendOnly` flag to `True` if you want to only insert data and not perform updates and deletes. By setting the `AppendOnly` flag to `True`, Firehose automatically scales to match your throughput. Currently, you can set this flag only with the [CreateDeliveryStream](https://docs.aws.amazon.com/firehose/latest/APIReference/API_CreateDeliveryStream.html) API operation. If a **Direct PUT** stream experiences throttling due to higher data ingest volumes that exceed the throughput capacity of a Firehose stream, then Firehose automatically increases the throughput limit of the stream until the throttling is contained. Depending on increased throughput and throttling, it might take longer for Firehose to increase the throughput of a stream to the desired levels. Because of this, continue to retry the failed data ingest records. If you expect the data volume to increase in sudden large bursts, or if your new stream needs a higher throughput than the default throughput limit, request to increase the throughput limit. + **S3 Transaction Per Second (TPS) **– To optimize S3 performance, if you are using Kinesis Data Streams or Amazon MSK as a source, we recommend that you partition the source record using a proper partition key. In that way, data records that are routed to the same Iceberg table are mapped to one or a few source partitions know as shards. If possible, spread data records belonging to different target Iceberg tables into different partitions/shards, so that you can use all the aggregate throughput available across all the partitions/shards of the source topic/stream. + **Columns** – For column names and values, Firehose takes only the first level of nodes in a multi-level nested JSON. For example, Firehose selects the nodes that are available in the first level including the position field. The column names and the data types of the source data must exactly match those of the target tables for Firehose to deliver successfully. In this case, Firehose expects that you have either struct or map data type column in your Iceberg tables to match the position field. Firehose supports 16 levels of nesting. Following is an example of a nested JSON. ``` { "version":"2016-04-01", "deviceId":"", "sensorId":"", "timestamp":"2024-01-11T20:42:45.000Z", "value":"", "position":{ "x":143.595901, "y":476.399628, "z":0.24234876 } } ``` If the column names or data types do not match, then Firehose throws an error and delivers data to the S3 error bucket. If all the column names and data types match in the Apache Iceberg tables, but you have an additional field present in the source record, Firehose skips the new field. + **One JSON object per record** – You can send only one JSON object in one Firehose record. If you aggregate and send multiple JSON objects inside a record, Firehose throws an error and delivers data to the S3 error bucket. If you aggregate records with [KPL](https://docs.aws.amazon.com/streams/latest/dev/kpl-with-firehose.html) and ingest data into Firehose with Amazon Kinesis Data Streams as source, then Firehose automatically de-aggregates and uses one JSON object per record. + **Compaction and storage optimization** – Every time you write to Iceberg Tables using Firehose, it commits and generates snapshots, data files and delete files. Having many data files increases metadata overhead and affects read performance. To get efficient query performance, you might want to consider a solution that periodically takes small data files and rewrites them into fewer larger data files. This process is called compaction. AWS Glue Data Catalog supports automatic compaction of your Apache Iceberg Tables. For more information, see [Compaction management](https://docs.aws.amazon.com/glue/latest/dg/compaction-management.html) in the *AWS Glue User Guide*. For additional information, see [Automatic compaction of Apache Iceberg Tables](https://aws.amazon.com/blogs/aws/aws-glue-data-catalog-now-supports-automatic-compaction-of-apache-iceberg-tables/). Alternatively, you can run the Athena Optimize command to perform compaction manually. For more information about the Optimize command, see [Athena Optimize](https://docs.aws.amazon.com/athena/latest/ug/optimize-statement.html). Besides compaction of data files, you can also optimize storage consumption with the [VACUUM](https://docs.aws.amazon.com/athena/latest/ug/vacuum-statement.html) statement that performs table maintenance on Apache Iceberg tables, such as snapshot expiration and orphan file removal. Alternatively, you can use AWS Glue Data Catalog that also supports managed table optimization of Apache Iceberg tables by automatically removing the data files, orphaned files, and expire snapshots that are no longer needed. For more information, see this blog post on [Storage optimization of Apache Iceberg Tables](https://aws.amazon.com/blogs/big-data/the-aws-glue-data-catalog-now-supports-storage-optimization-of-apache-iceberg-tables/). + We do not support Amazon MSK Serverless source for Apache Iceberg Tables as a destination. + For an update operation, Firehose puts a delete file followed by an insert operation. Putting delete files incurs Amazon S3 put charges. + Firehose does not recommend using multiple Firehose streams to write data to the same Apache Iceberg table. This is because Apache Iceberg relies on [Optimistic Concurrency Control (OCC)](https://iceberg.apache.org/docs/1.6.0/reliability/#concurrent-write-operations). If multiple Firehose streams attempt to write to a single Iceberg table concurrently, then only one stream is succeed in committing the data at a given time. The other streams that fail to commit back-off, and retry the commit operation until the configured retry duration expires. Once the retry duration is exhausted, the data and delete file keys (Amazon S3 paths) are sent to the configured Amazon S3 error prefix. + The current Iceberg Library version that Firehose supports is version 1.5.2. + For delivering encrypted data to Amazon S3 Tables, you should configure AWS Key Management Service parameters in Amazon S3 Tables, and not in the Firehose configuration. If you configure AWS Key Management Service parameters in Firehose for delivering encrypted data to Amazon S3 Tables, then Firehose cannot use those parameters to encrypt. For more information, see [Using server-side encryption with AWS KMS keys](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-kms-encryption.html). + Firehose streams only support delivery to databases and tables that are created through Iceberg’s GlueCatalog API. Delivery to databases and tables that are created through the Glue SDK are not supported. Note that a hyphen (`-`) is not a supported character for the database and the table name in the Iceberg library. For more details, see the [Glue Database Regex](https://github.com/apache/iceberg/blob/main/aws/src/main/java/org/apache/iceberg/aws/glue/IcebergToGlueConverter.java#L62) and the [Glue Table Regex](https://github.com/apache/iceberg/blob/main/aws/src/main/java/org/apache/iceberg/aws/glue/IcebergToGlueConverter.java#L63]) that are supported by the Iceberg library. + All files written by Firehose are computed using the partition that is present in the record. This also applies to deleted files. Global deletes, such as writing unpartitioned delete files for a partitioned table, is not supported. + Firehose does not currently support bloom filter properties when delivering data to Apache Iceberg tables. When bloom filter properties are configured on Iceberg tables, Firehose will ignore these properties during data delivery operations. # Prerequisites to use Apache Iceberg Tables as a destination Choose from the following options to complete the required prerequisites. **Topics** + [Prerequisites to deliver to Iceberg Tables in Amazon S3](#iceberg-tables-prerequisites) + [Prerequisites to deliver to Amazon S3 Tables](#s3-tables-prerequisites) ## Prerequisites to deliver to Iceberg Tables in Amazon S3 Before you begin, complete the following prerequisites. + **Create an Amazon S3 bucket** – You must create an Amazon S3 bucket to add metadata file path during tables creation. For more information, see [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). + **Create an IAM role with required permissions** – Firehose needs an IAM role with specific permissions to access AWS Glue tables and write data to Amazon S3. The same role is used to grant AWS Glue access to Amazon S3 buckets. You need this IAM role when you create an Iceberg Table and a Firehose stream. For more information, see [Grant Firehose access to Amazon S3 Tables](controlling-access.md#using-s3-tables). + **Create Apache Iceberg Tables** – If you are configuring unique keys in the Firehose stream for updates and deletes, Firehose validates if the table and unique keys exist as a part of stream creation. For this scenario, you must create tables before creating the Firehose stream. You can use AWS Glue to create Apache Iceberg Tables. For more information, see [Creating Apache Iceberg tables](https://docs.aws.amazon.com/glue/latest/dg/populate-otf.html#creating-iceberg-tables). If you are not configuring unique keys in the Firehose stream, then you don't require to create Iceberg tables before creating a Firehose stream. **Note** Firehose supports the following table version and format for Apache Iceberg tables. **Table format version** – Firehose only supports [V2 table format](https://iceberg.apache.org/spec/#version-2). Do not create tables in V1 format, else you get an error and data is delivered to the S3 error bucket instead. **Data storage format** – Firehose writes data to Apache Iceberg Tables in Parquet format. **Row level operation** – Firehose supports the Merge-on-Read (MOR) mode of writing data to Apache Iceberg Tables. ## Prerequisites to deliver to Amazon S3 Tables To deliver data to Amazon S3 table buckets, complete the following prerequisites. + Create an S3 Table bucket, namespace, tables in the table bucket, and other integration steps outlined in [Getting started with Amazon S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-getting-started.html). Column names must be lowercase because of the limitations imposed by the S3 Tables catalog integration, as specified in [S3 tables catalog integration limitations](https://docs.aws.amazon.com/lake-formation/latest/dg/notes-s3-catalog.html). + **Create an IAM role with required permissions** – Firehose needs an IAM role with specific permissions to access AWS AWS Glue tables and write data to tables in an Amazon S3 table bucket. To write to tables in an Amazon S3 table bucket, you must also provide the IAM role with the required permissions. The permissions required for Amazon S3 Tables catalog depend on the access control mode you use: + **IAM access control** – The Firehose delivery role needs IAM permissions directly on Amazon S3 Tables resources. + **Lake Formation access control** – The Firehose delivery role needs AWS AWS Lake Formation permissions for managing access to your table resources. AWS Lake Formation uses its own permissions model that enables fine-grained access control for Data Catalog resources. You configure this IAM role when you create a Firehose stream. For more information, see [Grant Firehose access to Amazon S3 Tables](https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html#using-s3-tables). For step-by-step integration, refer to the blog [Build a data lake for streaming data with Amazon S3 Tables and Amazon Data Firehose](https://aws.amazon.com/blogs/storage/build-a-data-lake-for-streaming-data-with-amazon-s3-tables-and-amazon-data-firehose/). For additional information, also refer to [Using Amazon S3 Tables with AWS analytics services](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html). # Set up the Firehose stream To create a Firehose stream with Apache Iceberg Tables as your destination you must configure the following. **Note** The setup of a Firehose stream for delivering to tables in S3 table buckets is the same as Apache Iceberg Tables in Amazon S3. ## Configure source and destination To deliver data to Apache Iceberg Tables, choose the source for your stream. To configure your source for your stream, see [Configure source settings](configure-source.md). Next, choose **Apache Iceberg Tables** as the destination and provide a Firehose stream name. ## Configure data transformation To perform custom transformations on your data, such as adding or modifying records in your incoming stream, you can add a Lambda function to your Firehose stream. For more information on data transformation using Lambda in a Firehose stream, see [Transform source data in Amazon Data Firehose](data-transformation.md). For Apache Iceberg Tables, you must specify how you want to route incoming records to different destination tables and the operations that you want to perform. One of the ways to provide the required routing information to Firehose is using a Lambda function. For more information, see [Route records to different Iceberg tables](apache-iceberg-format-input-record.md). ## Connect data catalog Apache Iceberg requires a data catalog to write to Apache Iceberg Tables. Firehose integrates with AWS Glue Data Catalog for Apache Iceberg Tables. You can use AWS Glue Data Catalog in the same account as your Firehose stream or in a cross-account and in the same Region as your Firehose stream (default), or in a different Region. If you are delivering to an Amazon S3 Table and you are using the console to set up your Firehose stream, then select the catalog that corresponds to your Amazon S3 Table catalog. If you are using the CLI to set up your Firehose stream, then in the `CatalogConfiguration` input, use `CatalogARN` with the format: `arn:aws:glue:::catalog/s3tablescatalog/`. For more information, see [Setting up a Firehose stream to Amazon S3 tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-firehose.html#firehose-stream-tables). **Note** Firehose supports three operations for Iceberg tables: insert, update, and delete. Without a specified operation, Firehosedefaults to insert, adding each incoming record as a new row, and preserving duplicates. To modify existing records instead, specify the "update" operation, which uses primary keys to locate, and change existing rows. Example: Default (insert): Multiple identical customer records create duplicate rows. Specified update: New customer address updates the existing record. ## Configure JQ expressions For Apache Iceberg Tables, you must specify how you want to route incoming records to different destination tables and the operations such as insert, update, and delete that you want to perform. You can do this by configuring JQ expressions for Firehose to parse and get the required information. For more information, see [Provide routing information to Firehose with JSONQuery expression](apache-iceberg-format-input-record-different.md#apache-iceberg-route-jq). ## Configure unique keys **Updates and Deletes with more than one table** – Unique keys are one or more fields in your source record that uniquely identify a row in Apache Iceberg Tables. If you have insert only scenario with more than one table, then you do not have to configure unique keys. If you want to do updates and deletes on certain tables, then you must configure unique keys for those required tables. Note that update will automatically insert the row if the row in the tables is missing. If you have only a single table, then you can configure unique keys. For an update operation, Firehose puts a delete file followed by an insert. You can either configure unique keys per table as a part of Firehose stream creation or you can set [identifier-field-ids](https://iceberg.apache.org/spec/#identifier-field-ids) natively in Iceberg during [create table](https://iceberg.apache.org/docs/1.5.1/spark-ddl/#create-table) or [alter table](https://iceberg.apache.org/docs/1.5.1/spark-ddl/#alter-table-set-identifier-fields) operation. Configuring unique keys per table during stream creation is optional. If you don’t configure unique keys per table during stream creation, Firehose checks for `identifier-field-ids` for required tables and will use them as unique keys. If both are not configured, then delivery of data with update and delete operations fails. To configure this section, provide the database name, table name, and unique keys for the tables where you want to update or delete data. You can have only entry for each table in the configuration. You don’t need to configure this section for append-only scenarios. Optionally, you can also choose to provide an error bucket prefix if data from the table fails to deliver as shown in the following example. ``` [ { "DestinationDatabaseName": "MySampleDatabase", "DestinationTableName": "MySampleTable", "UniqueKeys": [ "COLUMN_PLACEHOLDER" ], "S3ErrorOutputPrefix": "OPTIONAL_PREFIX_PLACEHOLDER" } ] ``` Firehose supports the configuration of unique keys if the supplied column name is unique across the entire table. However, it does not support fully qualified column names as unique keys. For instance, a key named `top._id` is not considered a unique key if the column name `_id` is also present at the top-level. If `_id` is unique across the entire table, then it is utilized regardless of its location within the table structure. This is whether it's a top-level column or a nested column. In the following example, `_id` is a valid unique key for the schema because the column name is unique across the schema. ``` [ "schema": { "type": "struct", "fields": [ { "name": "top", "type": { "type": "struct", "fields": [ { "name": "_id", "type": "string" }, { "name": "name", "type": "string" } ] } }, { "name": "user", "type": "string" } ] } ] ``` In the following example, `_id` is not a valid unique key for the schema because it is used in both the top-level column, and the nested struct. ``` [ "schema": { "type": "struct", "fields": [ { "name": "top", "type": { "type": "struct", "fields": [ { "name": "_id", "type": "string" }, { "name": "name", "type": "string" } ] } }, { "name": "_id", "type": "string" } ] } ] ``` ## Specify retry duration You can use this configuration to specify the duration in seconds for which Firehose should attempt to retry, if it encounters failures in writing to Apache Iceberg Tables in Amazon S3. You can set any value from 0 to 7200 seconds for performing retries. By default, Firehose retries for 300 seconds. ## Handle failed delivery or processing You must configure Firehose to deliver records to an S3 backup bucket in case it encounters failures in processing or delivering a stream after expiry of retry duration. For this, configure the **S3 backup bucket** and **S3 backup bucket error output prefix** from **Backup settings** in console. ## Handle errors Firehose sends all delivery errors to CloudWatch Logs, and Amazon S3 error buckets. List of errors: | **Error Message** | ****Description**** | | --- | --- | | `Iceberg.NoSuchTable` | Firehose is writing to a table that doesn't exist, or the table is not in V2 format. Firehose doesn't support tables in V1 format. | | `Iceberg.InvalidTableName` | A null or empty table name is passed, or the table is not in V2 format. Firehosedoesn't support tables in V1 format. | | `S3.AccessDenied` | Ensure that the IAM role created in the prerequisites step has the required permissions, and trust policy. | | `Glue.AccessDenied` | Ensure that the IAM role created in the prerequisites step has the required permissions, and trust policy. | ## Configure buffer hints Firehose buffers incoming streaming data in memory to a certain size (**Buffering size**) and for a certain period of time (**Buffering interval**) before delivering it to Apache Iceberg Tables. You can choose a buffer size of 1–128 MiBs and a buffer interval of 0–900 seconds. Higher buffer hints results in a lower number of S3 writes, less cost of compaction due to larger data files, and faster query runtime, but with a higher latency. Lower buffer hint values deliver the data with lower latency. ## Configure advanced settings You can configure server-side encryption, error logging, permissions, and tags for your Apache Iceberg Tables. For more information, see [Configure advanced settings](create-configure-advanced.md). You must add the IAM role that you created as part of the [Prerequisites to use Apache Iceberg Tables as a destination](apache-iceberg-prereq.md). Firehose will assume the role to access AWS Glue tables and write to Amazon S3 buckets. Firehose stream creation can take several minutes to complete. After you successfully create the Firehose stream, you can start ingesting data into it and can view the data in Apache Iceberg tables. # Route incoming records to a single Iceberg table If you want Firehose to insert data to a single Iceberg table, simply configure a single database and table in your stream configuration as shown in the following example JSON. For a single table, you do not require JQ expression and Lambda function for providing the routing information to Firehose. If you provide these fields along with JQ or Lambda, then Firehose will take input from JQ or Lambda. ``` [ { "DestinationDatabaseName": "UserEvents", "DestinationTableName": "customer_id", "UniqueKeys": [ "COLUMN_PLACEHOLDER" ], "S3ErrorOutputPrefix": "OPTIONAL_PREFIX_PLACEHOLDER" } ] ``` In this example, Firehose routes all input records to `customer_id` table in `UserEvents` database. If you want to perform update or delete operations on a single table, then you must provide the operation for each incoming record to Firehose using either the [JSONQuery method](apache-iceberg-format-input-record-different.md#apache-iceberg-route-jq) or [Lambda method](apache-iceberg-format-input-record-different.md#apache-iceberg-route-lambda). # Route incoming records to different Iceberg tables Amazon Data Firehose can route incoming records in a stream to different Iceberg tables based on the content of the record. Records are not kept in order when delivered from Amazon Data Firehose. Consider the following sample input record. ``` { "deviceId": "Device1234", "timestamp": "2024-11-28T11:30:00Z", "data": { "temperature": 21.5, "location": { "latitude": 37.3324, "longitude": -122.0311 } }, "powerlevel": 84, "status": "online" } ``` ``` { "deviceId": "Device4567", "timestamp": "2023-11-28T10:40:00Z", "data": { "pressure": 1012.4, "location": { "zipcode": 24567 } }, "powerlevel": 82, "status": "online" } ``` In this example, the `deviceId` field has two possible values – `Device1234` and `Device4567`. When an incoming record has `deviceId` field as `Device1234`, we want to write the record to an Iceberg table named `Device1234`, and when an incoming record has `deviceId` field as `Device4567`, we want to write the record to a table named `Device4567`. Note that the records with `Device1234` and `Device4567` might have a different set of fields that map to different columns in the corresponding Iceberg table. The incoming records might have a nested JSON structure where the **`deviceId`** can be nested within the JSON record. In the upcoming sections, we discuss how you can route records to different tables by providing the appropriate routing information to Firehose in such scenarios. ## Provide routing information to Firehose with JSONQuery expression The simplest and most cost effective way to provide record routing information to Firehose is by providing a JSONQuery expression. With this approach, you provide JSONQuery expressions for three parameters – `Database Name`, `Table Name`, and (optionally) `Operation`. Firehose uses the expression that you provide to extract information from your incoming stream records to route the records. The `Database Name` parameter specifies the name of the destination database. The `Table Name` parameter specifies the name of the destination table. `Operation` is an optional parameter that indicates whether to insert the incoming stream record as a new record into the destination table, or to modify or delete an existing record in the destination table. The Operation field must have one of the following values – `insert`, `update`, or `delete`. For each of these three parameters, you can either provide a static value or a dynamic expression where the value is retrieved from the incoming record. For example, if you want to deliver all incoming stream records to a single database named `IoTevents`, the Database Name would have a static value of `“IoTevents”`. If the destination table name must be obtained from a field in the incoming record, the Table Name is a dynamic expression that specifies the field in the incoming record from which the destination table name needs to be retrieved. In the following example, we use a static value for Database Name, a dynamic value for Table Name, and a static value for operation. Note that specifying the Operation is optional. If no operation is specified, Firehose inserts the incoming records into the destination table as new records by default. ``` Database Name : "IoTevents" Table Name : .deviceId Operation : "insert" ``` If the `deviceId` field is nested within the JSON record, we specify Table Name with the nested field information as `.event.deviceId`. **Note** When you specify the operation as `update` or `delete`, you must either specify unique keys for the destination table when you set up your Firehose stream, or set [identifier-field-ids](https://iceberg.apache.org/spec/#identifier-field-ids) in Iceberg when you run [create table](https://iceberg.apache.org/docs/1.5.1/spark-ddl/#create-table) or [alter table](https://iceberg.apache.org/docs/1.5.1/spark-ddl/#alter-table-set-identifier-fields) operations in Iceberg. If you fail to specify this, then Firehose throws an error and delivers data to an S3 error bucket. The `Database Name` and `Table Name` values must exactly match with your destination database and table names. If they do not match, then Firehose throws an error and delivers data to an S3 error bucket. ## Provide routing information using an AWS Lambda function There might be scenarios where you have complex rules that determine how to route incoming records to a destination table. For example, you might have a rule that defined if a field contains the value A, B, or F, that should be routed to a destination table named `TableX` or you might want to augment the incoming stream record by adding additional attributes. For example, if a record contains a field `device_id` as 1, you might want to add another field that `device_type` as “modem“, and write the additional field to the destination table column. In such cases, you can transform the source stream by using an AWS Lambda function in Firehose and provide routing information as part of the output of the Lambda transformation function. To understand how you can transform the source stream by using an AWS Lambda function in Firehose, see [Transform source data in Amazon Data Firehose](data-transformation.md). When you use Lambda for transformation of a source stream in Firehose, the output must contain `recordId`, `result`, and `data` or `KafkaRecordValue` parameters. The parameter `recordId` contains the input stream record, `result` indicates whether the transformation was successful, and `data` contains the Base64-encoded transformed output of your Lambda function. For more information, see [Required parameters for data transformation](data-transformation-status-model.md). ``` { "recordId": "49655962066601463032522589543535113056108699331451682818000000", "result": "Ok", "data": "1IiwiI6ICJmYWxsIiwgImdgU21IiwiI6ICJmYWxsIiwg==tcHV0ZXIgU2NpZW5jZSIsICJzZW1" } ``` To specify routing information to Firehose on how to route the stream record to a destination table as part of your Lambda function, the output of your Lambda function must contain an additional section for `metadata`. The following example shows how the metadata section is added to the Lambda output for a Firehose stream that uses Kinesis Data Streams as a data source to instruct Firehose that it must insert the record as a new record into table named `Device1234` of the database `IoTevents`. ``` { "recordId": "49655962066601463032522589543535113056108699331451682818000000", "result": "Ok", "data": "1IiwiI6ICJmYWxsIiwgImdgU21IiwiI6ICJmYWxsIiwg==tcHV0ZXIgU2NpZW5jZSIsICJzZW1", "metadata":{ "otfMetadata":{ "destinationTableName":"Device1234", "destinationDatabaseName":"IoTevents", "operation":"insert" } } } ``` Similarly, the following example shows how you can add the metadata section to the Lambda output for a Firehose that uses Amazon Managed Streaming for Apache Kafka as a data source to instruct Firehose that it must insert the record as a new record into a table named `Device1234` in the database `IoTevents`. ``` { "recordId": "49655962066601463032522589543535113056108699331451682818000000", "result": "Ok", "kafkaRecordValue": "1IiwiI6ICJmYWxsIiwgImdgU21IiwiI6ICJmYWxsIiwg==tcHV0ZXIgU2NpZW5jZSIsICJzZW1", "metadata":{ "otfMetadata":{ "destinationTableName":"Device1234", "destinationDatabaseName":"IoTevents", "operation":"insert" } } } ``` For this example, + `destinationDatabaseName` refers to the name of the target database and is a required field. + `destinationTableName` refers to the name of the target table and is a required field. + `operation` is an optional field with possible values as `insert`, `update`, and `delete`. If you do not specify any values, the default operation is `insert`. **Note** When you specify the operation as `update` or `delete`, you must either specify unique keys for the destination table when you set up your Firehose stream, or set [identifier-field-ids](https://iceberg.apache.org/spec/#identifier-field-ids) in Iceberg when you run [create table](https://iceberg.apache.org/docs/1.5.1/spark-ddl/#create-table) or [alter table](https://iceberg.apache.org/docs/1.5.1/spark-ddl/#alter-table-set-identifier-fields) operations in Iceberg. If you fail to specify this, then Firehose throws an error and delivers data to an S3 error bucket. The `Database Name` and `Table Name` values must exactly match with your destination database and table names. If they do not match, then Firehose throws an error and delivers data to an S3 error bucket. When your Firehose stream has both a Lambda transformation function and a JSONQuery expression, Firehose first checks for the metadata field in Lambda output to determine how to route the record to the appropriate destination table, and then look at the output of your JSONQuery expression for missing fields. If the Lambda or JSONQuery expression don't provide the required routing information, then Firehose assumes this as a single table scenario and looks for single table information in the unique keys configuration. For more information, see the [Route incoming records to a single Iceberg table](apache-iceberg-format-input-record.md). If Firehose fails to determine routing information and match the record to a specified destination table, it delivers the data to your specified S3 error bucket. ### Sample Lambda function This Lambda function is a sample Python code that parses the incoming stream records and adds required fields to specify how the data should be written to specific tables. You can use this sample code to add the metadata section for routing information. ``` import json import base64 def lambda_handler(firehose_records_input, context): print("Received records for processing from DeliveryStream: " + firehose_records_input['deliveryStreamArn']) firehose_records_output = {} firehose_records_output['records'] = [] for firehose_record_input in firehose_records_input['records']: # Get payload from Lambda input, it could be different with different sources if 'kafkaRecordValue' in firehose_record_input: payload_bytes = base64.b64decode(firehose_record_input['kafkaRecordValue']).decode('utf-8') else payload_bytes = base64.b64decode(firehose_record_input['data']).decode('utf-8') # perform data processing on customer payload bytes here # Create output with proper record ID, output data (may be different with different sources), result, and metadata firehose_record_output = {} if 'kafkaRecordValue' in firehose_record_input: firehose_record_output['kafkaRecordValue'] = base64.b64encode(payload_bytes.encode('utf-8')) else firehose_record_output['data'] = base64.b64encode(payload_bytes.encode('utf-8')) firehose_record_output['recordId'] = firehose_record_input['recordId'] firehose_record_output['result'] = 'Ok' firehose_record_output['metadata'] = { 'otfMetadata': { 'destinationDatabaseName': 'your_destination_database', 'destinationTableName': 'your_destination_table', 'operation': 'insert' } } firehose_records_output['records'].append(firehose_record_output) return firehose_records_output ``` # Monitor metrics For data delivery to Apache Iceberg Tables, Firehose emits the following CloudWatch metrics at a stream level. | Metric | Description | | --- | --- | | DeliveryToIceberg.Bytes | The number of bytes delivered to Apache Iceberg Tables over the specified time period. Units: Bytes | | DeliveryToIceberg.IncomingRowCount | Number of records that Firehose attempts to deliver to Apache Iceberg Tables. Units: Count | | DeliveryToIceberg.SuccessfulRowCount | Number of successful rows delivered to Apache Iceberg Tables. Units: Count | | DeliveryToIceberg.FailedRowCount | Number of failed rows delivered to S3 backup bucket. Units: Count | | DeliveryToIceberg.DataFreshness | The age (from getting into Firehose to now) of the earliest record in Firehose. Any record earlier than this age has been delivered to Apache Iceberg Tables.Units: Seconds | | DeliveryToIceberg.Success | Sum of successful commits to Apache Iceberg Tables. | | JQProcessing.Duration | The amount of time it took to run the JQ expression.Units: Milliseconds | # Understand supported data types Firehose supports all the primitive and complex data types that Apache Iceberg supports. For more information, see [Schemas and Data Types](https://iceberg.apache.org/spec/#schemas-and-data-types). When sending binary data as a string, you must use Firehose supported encoding types - Basic Base64, MIME Base64, URL and filename safe Base64, and Hex. For Timestamp data types, you must always send in microseconds. ## Data types examples The following section shows examples of different data types. **MapType** ``` { "destination_column_0": {"WP5o0JOkuIQcDPcsvpJJygF1xzaOSq0wUlgTwuIeCEzgVneGxA":"PO3ReF3auyDqbfonx9Cd8NTmcQnqnw7JuZOCWwI1jqgQKpsdMASWuU9rzb98Nm0WQe6l7i8TTEgH5gpqohIaQ58xwquYYho3ephkXfCgLryWdAhcyNcKf5SR8Xc5iv1SSsucXDqSoEceYiHTN6","eJbYbzmmasbNjun5WQhDgW81WJBQJWY0XwcNmgUn2qj2f7X5VWVAwnH1u3DzuglPicBOKLAWesjLXciJJYMn3E3vE7FgUZCAeo6Haf7":"XLnTauZzUiesOCv1kcI8QgO3eH0zEkiJiaExywuNoVk1feh4FpTL","vgHzvjlfqXG6gVnvKqMu18YiOvEqixeoK3DhcGafkkVfE9MVSDHBhc9L2OsF5dIasTWhKYYQnZrzxro3Fc1bzgKm1UHiYRrtg5eKHTJUW4fC8qR65Y73HynS0hRNfdD0miQi4jaENTrnona348iIjiVL9mvYR":"VsVIcn1RAEWYyIRH2iOPzIBzZBWJYjjcPGOlRt6PfE0Gt97dfkOZ7MpNNXTtJ1g3AUHoAvpo6asfWDKELcHmZEgu5eo8fga","8UjOp19MIgfe1DMzIWbE44":"CtenADor4MMiZfRq6eN0LLKEcsn5qVHLsQrBsHjqocb1dIUZ8fk1Bo2YIzT0rcPxx0LAbzOHx","8FSZWB13VL9csnpX7RAcajIlUcnO7vH8qjkaVMYFD3uS0QlJ5gQL63DLih7Mh6TwVOhWbrTAEK4zffn5iyWKhYqslG79UiRn9A35zdYeLt6rJUS71Yu9HWybxhoYBRwX0wCd9YjJAiTKxNunvKTEyqgEM2BcavcAT5RENSa819Y3nDGhdekmdhkUP1J6tcsEGio3":"liMdGIy12FC5iGlETX1pkfQZr6BrTAI7nvzlQUd7xELs2JVSXUGG9OmY3EFPvQ8IS7h4vyJikvTgczun6JMrq6Cw6Xs083FUh9LQpXfQCBXafhBCPIWwLtRX5ilMf8ZJ6ndJTDiFCrhXuFPC","2puYqZivt4CrkeLsmcuPw38oUkEUnpXrXfK1e9WafRJqN3pR1rakR2RFq5jyQcYkQXnOjdQi2cxthoSnA":"icdnS1DMLt950mwURyUR8tENXk3vJgGmTabJHmIuRSyoWHY7LbyGnRN8U1lQyOTRPCxkE7zKrSFOe13Jo1dlWDClVrxUUrrYb2Hc61zmbCagXcfp5FQhqDL5gMeK4wW32g","ZtTLkLZBcUdem2bBu4F7ngM3JxG8CLyRwdVxXMhPB0zeBEnGVRxUqjmwZIZ4H3e":"rrF4n8bPvzkUhslj2kGDodvpoGGswlOhDaCnLGW0cq3FNbDFj1PAn95VHiUul5d8tmt8kmA0Rgr24dT70vPfERpVHPlXcyXTg8o","0lipyfJwb45pGxet5rggbUoWByZdmRvuSJyPGdhZz2IjPNr5pkGUuc0beVwiwgIu2dNXRJFT25xz8dYP67fasdzkNsxqfWFmOjZvW5b":"V4qDnpR9ZgYflvGnIiScYnzsrDW797UGDMZjEg0nIrkmRrnc6FYXPlm0aiUBXuK6ZGNsfRmEl0kw6GHuUtYMUCcWIfwUo5wK8n0s3KTzMGOPsjl5MTaA8HWqUIMc3q3JlIZOXAFAENj7EtduzPpHBKRHWDbtmjjGtSsx8GNmZ108Gv43J92Q5G","qbgztJdIywmuKauxGnmRBSKMVmtyreab6dITzul47t2QiIQ5fEXQ4J5OonT5":"6JB70Wd1rwxBQR36b5MqMgXHr7TYZSLJ9xTIHFCSkknxcuvmbot6xEkTv9ijtoLEQrxr9fwI8wsMCXsSYV7j3Fs4VPH8loyZMtYYbtUbuljLHjwJye1hlMeiLy","jwy5FY9Mi0vAo5w6l7tRM2Q4sKGaT9s":"5m4glPW6k7fm3zPYT2hVR1MfraLG8TEGPCUemRqC36KHbLoNCxLjGfaYLAI2CeE4IuMa1WIlrsXvPgVIvH7LSQED7uQfedVN4KZavMQycBqJrYkZpHdMO9JaMSo5N4ncNuvs0pgLLSketqdW8qUOjZvdnMTLcZpJFioQGVGL7HCWvWm4qxhKjHOxi0zua2hROLnMkK","HNGNf2jAaR6A07QHxF6KY0Ce4wITpCKQNTlkHWDqAm48ocUjWaTRmc8jBGN0Gs8iZalJiTLzJrquHVThx":"QneTrdTPwtT3sEuRc1KdYZNsfrDTBeKX3tPNovIB7ZTuQGbgtf70mIvGnt8vmdDyDKleDdF9WjESkAsz42PpJHpqYtwqwJbUbB47VfSMF4sNJRPo0eaZceqzDyHHNu6xJs","oKEKH0tyt2o42iXQffNvKrhkZHS6bqcqGIkSCMxrTbgRWl0sTiCpPzfkoY1XwMBRxAYuYRimZ3W5nM58clYiOxL2oVYdA2":"9FsQsrpJKN","bekToN5OdByhB9DRS7wb5daFxDI6KGBHSecKzl6n5UEjxaAp4dO0KTXlzzflR7YRrPDrR3pQAiy7Y1tYry":"dcwHnwApWtoZWtwzI1Syv6ak0XZPMmk7Tyq1pi3qB0axsPyRJztmjYslzW07muyAMiRIzX2SfMulhTAtjoHahwb6xLLdTrL9FfQvR"}, "destination_column_1": "{\"{\\\"destination_nested_column_0\\\": \\\"18:56:14.974\\\", \\\"destination_nested_column_1\\\": 241.86246}\":\"M07kAvYdHvBh61F7RzfxtEd39YQI33LnM2NbGS67DOFFsRUyUUujKT5VnK7Wtfz1mHNeIix6FAY9cYpwTdedgr9XnFwG0BHMO51LZPYXerDqAUmqhldyKUvxy1KMQkU\",\"{\\\"destination_nested_column_0\\\": \\\"18:56:14.974\\\", \\\"destination_nested_column_1\\\": 562.56384}\":\"9GlxhDCt95LxBo51HybBZihqOqf6EU8jrDu7NMpxtGB2dY6q6kXpvxIrFuMdqHCJKIZIcDikwggLniUm8kgE4d\",\"{\\\"destination_nested_column_0\\\": \\\"18:56:14.974\\\", \\\"destination_nested_column_1\\\": 496.03268}\":\"keTJZYLNvLRB50DMKzEI6M0AM4mueyNnA1m2YVnYdDwyxUpPqkb72Q6LiX0B9s8gCjZ6trW6C1PFk9KNBIpxYsj5Tc5Xsl13go3qLqFRwsFiW7peHp6xBJ7NcJm4RxSbIDEnTr1FLpmnKC18VZeY\",\"{\\\"destination_nested_column_0\\\": \\\"18:56:14.974\\\", \\\"destination_nested_column_1\\\": 559.0878}\":\"mG0ZET84BUF28E312UCIWgmypyQFSUODH9NAMAnF3LJEutbooZWcBt97PP5AHaopNvC8pQZ4mGXB9hmVmjUNmuj5QanyxXrKtCU8uivE3R8jPx3jur0iepx9ckgbHF7J13lDCXW92a\",\"{\\\"destination_nested_column_0\\\": \\\"18:56:14.974\\\", \\\"destination_nested_column_1\\\": 106.845245}\":\"aidoVYrzu8gcLRkVVUyTKCN9gqTUFYi8uJQsrXEFEYl1f9ool7JhAtg9QKG5BBu67Ngb95ENsNKQyCHNImSu5x4hMnmHUB6qRkfOsk9BzTqkM\"}" } ``` **DecimalType** ``` { "destination_column_0": 9455262425851.1342772, "destination_column_1": "9455262425851.1342772", "destination_column_2": 9455262425852 } ``` **BinaryType (base64-default, base64-mime, base64-url-safe, hex)** ``` { "destination_column_0": "AsYhnHD\/Ra54hITl1daNV9glOjtWPEfopH+PjgUKHYB6K7UcYi4K19b80wD4J\/93x5tyh+0y+k5cMljVRlmfIkIuLxl9ERBiPPLhf4+yoJ2k70VavPnYWmNLs1hLDHlfeEMIfVhrqOGzJMoA+CBAWXfIuiG420JSQP5iAx5xFG\/mOfkM5zYothje8OGXltdthcCL6WYBiP0SlwXcE0uMeRfwclAc9fTOBz6RzdJlHhUDjoAXg+4cvly27F82XpuGMNwpUj98AOrgbh2MoU9yvsM9ZrjD0eGVgOZP8Ky7Za4oE\/oK8j+qABF6XV712iA6pVtTNJFvX6Ey3ssNYvno+LYF5ZsySs2rB5AbVM73RfOPqdS\/c\/r3MEqoEqt+nPx6eGam4WSA+0swztt7aLdrlX6yK7xJeIJ0rTlIDBo0ZUaw011ykY\/8Bvy+4byoPlmr4Z5yhN1z3ZTOkx7eDR6xMv+vDVsDBtItVazDwHgDy41r\/hQNeNedPKrozc8TY9k7wZre\/6V2lCa3BmT8Uu9b9ydjR9z+fCSdG+VRv35nz5kdqdKy8YIrynYs4eOcjh8jH3UwVYrYQcnWkBAfF7Xk9CoPVnL3ciHZtyiZOaTGIj9rO0xX\/W5dGe9\/4YChs6LbD584kxLTxvHgSl4vadaTGNKci3SvNmZNsz8ducxtNXF\/Tv2DUub465hzgpaLPur3+MB+kfdN2YXUfqb+xJAgxThWfUe151nrH0EPow9lgSlp21rUBGznJAvPRl1ExGIAuc7JYAoUrJUkx5Hfl6PekPDhqt7+yJwCB8qxhTTryxo+bjtai4ndRCGcuCaxT8KkOcXsS37urd3YGSDMinZdMNVc646s25415qK6nBRlqqAY8+EYmcUIVB9XcNdke4zoUfhVQoruwidzDU\/kFafoulo5DEoMOyaH1N2HCSxG5tZXNQocSZPaY8efZYMCpmDXsPAzkmgSkYRDSu\/r3wUqROa2tGK5\/pQY24v+Jq0U\/jQ99GShlU283nZ85ot2ocbtMAgD\/WsrSEh6lNt9RaI3HfA7\/HcH\/fgr9jsTtxDgZhabTBwwDwX0zjWGx1bCuTLKBN7byxg9ZvAVgqwPS4HERLer5T5UkKf74zn9Eq3HYH1Q5JpyDUx+im7mte1sprf1+A24kksVU\/MD9aP9N8\/QDsQ13gkhOn5KwFMz3BC2Vw5gL+gGNHFKDRL6wGIfhuYcx9LucolZ1yNy9Gbb3ioWSSufyFpyXqtndDLPI5QS1SJpJm2KDyqcH1SmRLIhd9MNRUC73EAEm+NO5wxPzBRSjhCHZpf8SrYITWJl7K3XzGOfPFh2NgES3jMP9cvSXO6yyICcep2HBYGbFflni89+Rw==", "destination_column_1": "AsYhnHD\/Ra54hITl1daNV9glOjtWPEfopH+PjgUKHYB6K7UcYi4K19b80wD4J\/93x5tyh+0y+k5c\r\nMljVRlmfIkIuLxl9ERBiPPLhf4+yoJ2k70VavPnYWmNLs1hLDHlfeEMIfVhrqOGzJMoA+CBAWXfI\r\nuiG420JSQP5iAx5xFG\/mOfkM5zYothje8OGXltdthcCL6WYBiP0SlwXcE0uMeRfwclAc9fTOBz6R\r\nzdJlHhUDjoAXg+4cvly27F82XpuGMNwpUj98AOrgbh2MoU9yvsM9ZrjD0eGVgOZP8Ky7Za4oE\/oK\r\n8j+qABF6XV712iA6pVtTNJFvX6Ey3ssNYvno+LYF5ZsySs2rB5AbVM73RfOPqdS\/c\/r3MEqoEqt+\r\nnPx6eGam4WSA+0swztt7aLdrlX6yK7xJeIJ0rTlIDBo0ZUaw011ykY\/8Bvy+4byoPlmr4Z5yhN1z\r\n3ZTOkx7eDR6xMv+vDVsDBtItVazDwHgDy41r\/hQNeNedPKrozc8TY9k7wZre\/6V2lCa3BmT8Uu9b\r\n9ydjR9z+fCSdG+VRv35nz5kdqdKy8YIrynYs4eOcjh8jH3UwVYrYQcnWkBAfF7Xk9CoPVnL3ciHZ\r\ntyiZOaTGIj9rO0xX\/W5dGe9\/4YChs6LbD584kxLTxvHgSl4vadaTGNKci3SvNmZNsz8ducxtNXF\/\r\nTv2DUub465hzgpaLPur3+MB+kfdN2YXUfqb+xJAgxThWfUe151nrH0EPow9lgSlp21rUBGznJAvP\r\nRl1ExGIAuc7JYAoUrJUkx5Hfl6PekPDhqt7+yJwCB8qxhTTryxo+bjtai4ndRCGcuCaxT8KkOcXs\r\nS37urd3YGSDMinZdMNVc646s25415qK6nBRlqqAY8+EYmcUIVB9XcNdke4zoUfhVQoruwidzDU\/k\r\nFafoulo5DEoMOyaH1N2HCSxG5tZXNQocSZPaY8efZYMCpmDXsPAzkmgSkYRDSu\/r3wUqROa2tGK5\r\n\/pQY24v+Jq0U\/jQ99GShlU283nZ85ot2ocbtMAgD\/WsrSEh6lNt9RaI3HfA7\/HcH\/fgr9jsTtxDg\r\nZhabTBwwDwX0zjWGx1bCuTLKBN7byxg9ZvAVgqwPS4HERLer5T5UkKf74zn9Eq3HYH1Q5JpyDUx+\r\nim7mte1sprf1+A24kksVU\/MD9aP9N8\/QDsQ13gkhOn5KwFMz3BC2Vw5gL+gGNHFKDRL6wGIfhuYc\r\nx9LucolZ1yNy9Gbb3ioWSSufyFpyXqtndDLPI5QS1SJpJm2KDyqcH1SmRLIhd9MNRUC73EAEm+NO\r\n5wxPzBRSjhCHZpf8SrYITWJl7K3XzGOfPFh2NgES3jMP9cvSXO6yyICcep2HBYGbFflni89+Rw==", "destination_column_2": "AsYhnHD_Ra54hITl1daNV9glOjtWPEfopH-PjgUKHYB6K7UcYi4K19b80wD4J_93x5tyh-0y-k5cMljVRlmfIkIuLxl9ERBiPPLhf4-yoJ2k70VavPnYWmNLs1hLDHlfeEMIfVhrqOGzJMoA-CBAWXfIuiG420JSQP5iAx5xFG_mOfkM5zYothje8OGXltdthcCL6WYBiP0SlwXcE0uMeRfwclAc9fTOBz6RzdJlHhUDjoAXg-4cvly27F82XpuGMNwpUj98AOrgbh2MoU9yvsM9ZrjD0eGVgOZP8Ky7Za4oE_oK8j-qABF6XV712iA6pVtTNJFvX6Ey3ssNYvno-LYF5ZsySs2rB5AbVM73RfOPqdS_c_r3MEqoEqt-nPx6eGam4WSA-0swztt7aLdrlX6yK7xJeIJ0rTlIDBo0ZUaw011ykY_8Bvy-4byoPlmr4Z5yhN1z3ZTOkx7eDR6xMv-vDVsDBtItVazDwHgDy41r_hQNeNedPKrozc8TY9k7wZre_6V2lCa3BmT8Uu9b9ydjR9z-fCSdG-VRv35nz5kdqdKy8YIrynYs4eOcjh8jH3UwVYrYQcnWkBAfF7Xk9CoPVnL3ciHZtyiZOaTGIj9rO0xX_W5dGe9_4YChs6LbD584kxLTxvHgSl4vadaTGNKci3SvNmZNsz8ducxtNXF_Tv2DUub465hzgpaLPur3-MB-kfdN2YXUfqb-xJAgxThWfUe151nrH0EPow9lgSlp21rUBGznJAvPRl1ExGIAuc7JYAoUrJUkx5Hfl6PekPDhqt7-yJwCB8qxhTTryxo-bjtai4ndRCGcuCaxT8KkOcXsS37urd3YGSDMinZdMNVc646s25415qK6nBRlqqAY8-EYmcUIVB9XcNdke4zoUfhVQoruwidzDU_kFafoulo5DEoMOyaH1N2HCSxG5tZXNQocSZPaY8efZYMCpmDXsPAzkmgSkYRDSu_r3wUqROa2tGK5_pQY24v-Jq0U_jQ99GShlU283nZ85ot2ocbtMAgD_WsrSEh6lNt9RaI3HfA7_HcH_fgr9jsTtxDgZhabTBwwDwX0zjWGx1bCuTLKBN7byxg9ZvAVgqwPS4HERLer5T5UkKf74zn9Eq3HYH1Q5JpyDUx-im7mte1sprf1-A24kksVU_MD9aP9N8_QDsQ13gkhOn5KwFMz3BC2Vw5gL-gGNHFKDRL6wGIfhuYcx9LucolZ1yNy9Gbb3ioWSSufyFpyXqtndDLPI5QS1SJpJm2KDyqcH1SmRLIhd9MNRUC73EAEm-NO5wxPzBRSjhCHZpf8SrYITWJl7K3XzGOfPFh2NgES3jMP9cvSXO6yyICcep2HBYGbFflni89-Rw==", "destination_column_3": "02c6219c70ff45ae788484e5d5d68d57d8253a3b563c47e8a47f8f8e050a1d807a2bb51c622e0ad7d6fcd300f827ff77c79b7287ed32fa4e5c3258d546599f22422e2f197d1110623cf2e17f8fb2a09da4ef455abcf9d85a634bb3584b0c795f7843087d586ba8e1b324ca00f820405977c8ba21b8db425240fe62031e71146fe639f90ce73628b618def0e19796d76d85c08be9660188fd129705dc134b8c7917f072501cf5f4ce073e91cdd2651e15038e801783ee1cbe5cb6ec5f365e9b8630dc29523f7c00eae06e1d8ca14f72bec33d66b8c3d1e19580e64ff0acbb65ae2813fa0af23faa00117a5d5ef5da203aa55b5334916f5fa132decb0d62f9e8f8b605e59b324acdab07901b54cef745f38fa9d4bf73faf7304aa812ab7e9cfc7a7866a6e16480fb4b30cedb7b68b76b957eb22bbc49788274ad39480c1a346546b0d35d72918ffc06fcbee1bca83e59abe19e7284dd73dd94ce931ede0d1eb132ffaf0d5b0306d22d55acc3c07803cb8d6bfe140d78d79d3caae8cdcf1363d93bc19adeffa5769426b70664fc52ef5bf7276347dcfe7c249d1be551bf7e67cf991da9d2b2f1822bca762ce1e39c8e1f231f7530558ad841c9d690101f17b5e4f42a0f5672f77221d9b7289939a4c6223f6b3b4c57fd6e5d19ef7fe180a1b3a2db0f9f389312d3c6f1e04a5e2f69d69318d29c8b74af36664db33f1db9cc6d35717f4efd8352e6f8eb987382968b3eeaf7f8c07e91f74dd985d47ea6fec49020c538567d47b5e759eb1f410fa30f65812969db5ad4046ce7240bcf465d44c46200b9cec9600a14ac9524c791df97a3de90f0e1aadefec89c0207cab18534ebcb1a3e6e3b5a8b89dd44219cb826b14fc2a439c5ec4b7eeeadddd81920cc8a765d30d55ceb8eacdb9e35e6a2ba9c1465aaa018f3e11899c508541f5770d7647b8ce851f855428aeec227730d4fe415a7e8ba5a390c4a0c3b2687d4dd87092c46e6d657350a1c4993da63c79f658302a660d7b0f0339268129184434aefebdf052a44e6b6b462b9fe9418db8bfe26ad14fe343df464a1954dbcde767ce68b76a1c6ed300803fd6b2b48487a94db7d45a2371df03bfc7707fdf82bf63b13b710e066169b4c1c300f05f4ce3586c756c2b932ca04dedbcb183d66f01582ac0f4b81c444b7abe53e5490a7fbe339fd12adc7607d50e49a720d4c7e8a6ee6b5ed6ca6b7f5f80db8924b1553f303f5a3fd37cfd00ec435de09213a7e4ac05333dc10b6570e602fe80634714a0d12fac0621f86e61cc7d2ee728959d72372f466dbde2a16492b9fc85a725eab677432cf239412d52269266d8a0f2a9c1f54a644b22177d30d4540bbdc40049be34ee70c4fcc14528e10876697fc4ab6084d6265ecadd7cc639f3c5876360112de330ff5cbd25ceeb2c8809c7a9d8705819b15f9678bcf7e47" } ``` **TimeType (Epoch in Microseconds, LocalTime Java Object)** ``` { "destination_column_0": 68175096000, "destination_column_1": "18:56:15.096" } ``` **TimestampType.withZone (Epoch in Microseconds, OffsetDateTime Java Object, LocalDateTime Java Object)** ``` { "destination_column_0": 1725476175099000, "destination_column_1": "2024-09-04T18:56:15.099Z", "destination_column_2": "2024-09-04T18:56:15.099" } ``` **DoubleType** ``` { "destination_column_0": 9.18477568715142, "destination_column_1": "9.18477568715142" } ``` **BooleanType** ``` { "destination_column_0": true, "destination_column_1": "false", "destination_column_2": 1, "destination_column_3": 0 } ``` **FloatType** ``` { "destination_column_0": 0.6242226, "destination_column_1": "0.6242226" } ``` **IntegerType** ``` { "destination_column_0": 7, "destination_column_1": "7" } ``` **TimestampType.withoutZone (Epoch in Microseconds, LocalDateTime Java Object, OffsetDateTime Java Object, ZonedDateTime Java Object)** ``` { "destination_column_0": 1725476175114000, "destination_column_1": "2024-09-04T18:56:15.114", "destination_column_2": "2024-09-04T18:56:15.114Z", "destination_column_3": "2024-09-04T18:56:15.114-07:00" } ``` **DateType** ``` { "destination_column_0": 19970, "destination_column_1": "2024-09-04" } ``` **LongType** ``` { "destination_column_0": 8, "destination_column_1": "8" } ``` **UUIDType (UUID Java Object)** ``` { "destination_column_0": "21c5521c-a6d4-48d4-b2c8-7f6d842f72c3" } ``` **ListType** ``` { "destination_column_0": ["s1FSrgb0lGDxfn2iYT0EtlP47aHSjwmLZgrdr1JqRs0dmbeCcQoaLr4Xhi2KIVvmus9ppFdpWIcOHnJ0omhAPhXH0ynsaicCqnpQ8VAYp4bBPDgangwz9KOz60d0qSqCFxwEhoXjV4AouN5mwW","VUYtsT1pcw8q5WBxOiL47IZqea4fFloZwV7iJanDrvqK6AGa1yBuQyqd6R0axrcz8wcGBP","blsmImCYglNKNBcThVuKGeLOmKJdX19pRMR08ayc1EXN2rP7dIRfbhkk76XgAR3IsIwpB0jiQG8PW40PSlzn4sxUMN5wwQgyxmkZbk3e0dW0uqwVIm7KFkOPEHNmqgw01cT9KKMBwXkOGUmtX1","1234567890abcdef0","2234567890abcdef0"], "destination_column_1": "[{\"destination_nested_column_0\":\"bb00f8e6-db82-4241-a5c5-0d9c0d2f71a4\",\"destination_nested_column_1\":907.35345},{\"destination_nested_column_0\":\"2c77b702-d405-4fe1-beee-fb541d7ab833\",\"destination_nested_column_1\":544.0026},{\"destination_nested_column_0\":\"68389200-d6b1-413d-bcd9-fdb931708395\",\"destination_nested_column_1\":153.683},{\"destination_nested_column_0\":\"bc31cbaa-39cd-4e2f-b357-9ea9ce75532b\",\"destination_nested_column_1\":977.5165},{\"destination_nested_column_0\":\"b7d627f9-0d5b-41b7-903a-525488259fba\",\"destination_nested_column_1\":434.17215},{\"destination_nested_column_0\":\"06b6ec1e-1952-4582-b285-46aaf40064b8\",\"destination_nested_column_1\":580.33124},{\"destination_nested_column_0\":\"f04b3bbf-61ad-4c5c-8740-6f666f57c431\",\"destination_nested_column_1\":550.75793}]" } ``` # Resources Use the following resources to learn more: + [Stream real-time data into Apache Iceberg tables in Amazon S3 using Amazon Data Firehose](https://aws.amazon.com/blogs/big-data/stream-real-time-data-into-apache-iceberg-tables-in-amazon-s3-using-amazon-data-firehose/) + [Streamline AWS WAF log analysis with Apache Iceberg and Amazon Data Firehose](https://aws.amazon.com/blogs/big-data/streamline-aws-waf-log-analysis-with-apache-iceberg-and-amazon-data-firehose/) + [Build a data lake for streaming data with Amazon S3 Tables and Amazon Data Firehose](https://aws.amazon.com/blogs/storage/build-a-data-lake-for-streaming-data-with-amazon-s3-tables-and-amazon-data-firehose/)