# Managing Amazon SNS messages across multiple delivery stream destinations Managing messages across multiple delivery stream destinations [ delivery streams](sns-firehose-as-subscriber.md) allow you to manage Amazon SNS messages across multiple destinations, enabling integration with Amazon S3, Amazon OpenSearch Service, Amazon Redshift, and HTTP endpoints for storage, indexing, and analysis. By properly configuring message formatting and delivery, you can store Amazon SNS notifications in Amazon S3 for later processing, analyze structured message data using Amazon Athena, index messages in OpenSearch for real-time search and visualization, and structure archives in Amazon Redshift for advanced querying. # Storing and analyzing Amazon SNS messages in Amazon S3 destinations Storing and analyzing messages in Amazon S3 destinations This topic explains how delivery streams publish data to Amazon Simple Storage Service (Amazon S3). ![\[The integration and workflow of Amazon services for message handling. It shows how a publisher sends messages to an Amazon SNS topic, which then fans out messages to multiple Amazon SQS queues and an Data Firehose delivery stream. From there, messages can be processed by Lambda functions or stored persistently in an Amazon S3 bucket.\]](http://docs.aws.amazon.com/sns/latest/dg/images/firehose-architecture-s3.png) **Topics** + [Formatting notifications for storage in Amazon S3 destinations](firehose-archived-message-format-S3.md) + [Analyzing messages stored in Amazon S3 using Athena](firehose-message-analysis-s3.md) # Formatting Amazon SNS notifications for storage in Amazon S3 destinations Formatting notifications for storage in Amazon S3 destinations The following example shows an Amazon SNS notification sent to an Amazon Simple Storage Service (Amazon S3) bucket, with indentation for readability. **Note** In this example, raw message delivery is disabled for the published message. When raw message delivery is disabled, Amazon SNS adds JSON metadata to the message, including these properties: `Type` `MessageId` `TopicArn` `Subject` `Timestamp` `UnsubscribeURL` `MessageAttributes` For more information about raw delivery, see [Amazon SNS raw message delivery](sns-large-payload-raw-message-delivery.md). ``` { "Type": "Notification", "MessageId": "719a6bbf-f51b-5320-920f-3385b5e9aa56", "TopicArn": "arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic", "Subject": "My 1st subject", "Message": "My 1st body", "Timestamp": "2020-11-26T23:48:02.032Z", "UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic:0b410f3c-ee5e-49d8-b59b-3b4aa6d8fcf5", "MessageAttributes": { "myKey1": { "Type": "String", "Value": "myValue1" }, "myKey2": { "Type": "String", "Value": "myValue2" } } } ``` The following example shows three SNS messages sent through an delivery stream to the same Amazon S3 bucket. Buffering is applied, and line breaks separate each message. ``` {"Type":"Notification","MessageId":"d7d2513e-6126-5d77-bbe2-09042bd0a03a","TopicArn":"arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic","Subject":"My 1st subject","Message":"My 1st body","Timestamp":"2020-11-27T00:30:46.100Z","UnsubscribeURL":"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:313276652360:my-kinesis-test-topic:0b410f3c-ee5e-49d8-b59b-3b4aa6d8fcf5","MessageAttributes":{"myKey1":{"Type":"String","Value":"myValue1"},"myKey2":{"Type":"String","Value":"myValue2"}}} {"Type":"Notification","MessageId":"0c0696ab-7733-5bfb-b6db-ce913c294d56","TopicArn":"arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic","Subject":"My 2nd subject","Message":"My 2nd body","Timestamp":"2020-11-27T00:31:22.151Z","UnsubscribeURL":"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:313276652360:my-kinesis-test-topic:0b410f3c-ee5e-49d8-b59b-3b4aa6d8fcf5","MessageAttributes":{"myKey1":{"Type":"String","Value":"myValue1"}}} {"Type":"Notification","MessageId":"816cd54d-8cfa-58ad-91c9-8d77c7d173aa","TopicArn":"arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic","Subject":"My 3rd subject","Message":"My 3rd body","Timestamp":"2020-11-27T00:31:39.755Z","UnsubscribeURL":"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:313276652360:my-kinesis-test-topic:0b410f3c-ee5e-49d8-b59b-3b4aa6d8fcf5"} ``` # Analyzing Amazon SNS messages stored in Amazon S3 using Athena Analyzing messages stored in Amazon S3 using Athena This page explains how to analyze Amazon SNS messages that are sent through delivery streams to Amazon Simple Storage Service (Amazon S3) destinations. **To analyze SNS messages sent through Firehose delivery streams to Amazon S3 destinations** 1. Configure your Amazon S3 resources. For instructions, see [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/CreatingABucket.html) in the *Amazon Simple Storage Service User Guide* and [Working with Amazon S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) in the *Amazon Simple Storage Service User Guide*. 1. Configure your delivery stream. For instructions, see [Choose Amazon S3 for Your Destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-s3) in the *Amazon Data Firehose Developer Guide*. 1. Use [Amazon Athena](https://console.aws.amazon.com/athena) to query the Amazon S3 objects using standard SQL. For more information, see [Getting Started](https://docs.aws.amazon.com/athena/latest/ug/getting-started.html) in the *Amazon Athena User Guide*. ## Example query For this example query, assume the following: + Messages are stored in the `notifications` table in the `default` schema. + The `notifications` table includes a `timestamp` column with a type of `string`. The following query returns all SNS messages received in the specified date range: ``` SELECT * FROM default.notifications WHERE from_iso8601_timestamp(timestamp) BETWEEN TIMESTAMP '2020-12-01 00:00:00' AND TIMESTAMP '2020-12-02 00:00:00'; ``` # Integrating Amazon SNS messages with Amazon OpenSearch Service destinations Integrating messages with Amazon OpenSearch Service destinations This section explains how delivery streams publish data to Amazon OpenSearch Service (OpenSearch Service). ![\[A publisher sends messages to an Amazon SNS topic, which then distributes these messages to multiple Amazon SQS queues. Messages from these queues can be processed by Lambda functions or sent through an Data Firehose delivery stream to an Amazon OpenSearch Service, creating a searchable message index. This setup demonstrates an advanced message routing and processing scenario using AWS services.\]](http://docs.aws.amazon.com/sns/latest/dg/images/firehose-architecture-es.png) **Topics** + [Archived message format](firehose-archived-message-format-elasticsearch.md) + [Analyzing messages](firehose-message-analysis-elasticsearch.md) # Storing and formatting Amazon SNS Notifications in OpenSearch Service indices Archived message format The following example demonstrates an Amazon SNS notification sent to an Amazon OpenSearch Service (OpenSearch Service) index called `my-index`. This index has a time filter field on the `Timestamp` field. The SNS notification is placed in the `_source` property of the payload. **Note** In this example, raw message delivery is disabled for the published message. When raw message delivery is disabled, Amazon SNS adds JSON metadata to the message, including these properties: `Type` `MessageId` `TopicArn` `Subject` `Timestamp` `UnsubscribeURL` `MessageAttributes` For more information about raw delivery, see [Amazon SNS raw message delivery](sns-large-payload-raw-message-delivery.md). ``` { "_index": "my-index", "_type": "_doc", "_id": "49613100963111323203250405402193283794773886550985932802.0", "_version": 1, "_score": null, "_source": { "Type": "Notification", "MessageId": "bf32e294-46e3-5dd5-a6b3-bad65162e136", "TopicArn": "arn:aws:sns:us-east-1:111111111111:my-topic", "Subject": "Sample subject", "Message": "Sample message", "Timestamp": "2020-12-02T22:29:21.189Z", "UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:111111111111:my-topic:b5aa9bc1-9c3d-452b-b402-aca2cefc63c9", "MessageAttributes": { "my_attribute": { "Type": "String", "Value": "my_value" } } }, "fields": { "Timestamp": [ "2020-12-02T22:29:21.189Z" ] }, "sort": [ 1606948161189 ] } ``` # Analyzing Amazon SNS messages for OpenSearch Service destinations Analyzing messages This topic explains how to analyze Amazon SNS messages sent through delivery streams to Amazon OpenSearch Service (OpenSearch Service) destinations. **To analyze SNS messages sent through Firehose delivery streams to OpenSearch Service destinations** 1. Configure your OpenSearch Service resources. For instructions, see [Getting Started with Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/es-gsg.html) in the *Amazon OpenSearch Service Developer Guide*. 1. Configure your delivery stream. For instructions, see [Choose OpenSearch Service for Your Destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-elasticsearch) in the *Amazon Data Firehose Developer Guide*. 1. Run a query using OpenSearch Service queries and Kibana. For more information, see [Step 3: Search Documents in an OpenSearch Service Domain](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/es-gsg-search.html) and [Kibana](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/es-kibana.html) in the *Amazon OpenSearch Service Developer Guide*. ## Example query The following example queries the `my-index` index for all SNS messages received in the specified date range: ``` POST https://search-my-domain.us-east-1.es.amazonaws.com/my-index/_search { "query": { "bool": { "filter": [ { "range": { "Timestamp": { "gte": "2020-12-08T00:00:00.000Z", "lte": "2020-12-09T00:00:00.000Z", "format": "strict_date_optional_time" } } } ] } } } ``` # Configuring Amazon SNS message delivery and analysis in Amazon Redshift destinations Configuring message delivery and analysis in Amazon Redshift destinations This topic explains how to fan out Amazon SNS notifications to an delivery stream, which then publishes data to Amazon Redshift. With this setup, you can connect to the Amazon Redshift database and use a SQL query tool to retrieve Amazon SNS messages that match specific criteria. ![\[Messages published by a sender to an Amazon SNS topic are distributed to multiple Amazon SQS queues for processing by Lambda functions, and also sent through an Data Firehose delivery stream to an Amazon Redshift cluster for storage and analysis in a message data warehouse. This setup demonstrates a robust message handling and data warehousing architecture using AWS services.\]](http://docs.aws.amazon.com/sns/latest/dg/images/firehose-architecture-rs.png) **Topics** + [Structuring message archives in Amazon Redshift tables](firehose-archive-table-structure-redshift.md) + [Analyzing messages stored in Amazon Redshift destinations](firehose-message-analysis-redshift.md) # Structuring Amazon SNS message archives in Amazon Redshift tables Structuring message archives in Amazon Redshift tables For Amazon Redshift endpoints, Amazon SNS messages are archived as rows in a table. Here's an example of how the data is stored: **Note** In this example, raw message delivery is disabled for the published message. When raw message delivery is disabled, Amazon SNS adds JSON metadata to the message, including these properties: `Type` `MessageId` `TopicArn` `Subject` `Message` `Timestamp` `UnsubscribeURL` `MessageAttributes` For more information about raw delivery, see [Amazon SNS raw message delivery](sns-large-payload-raw-message-delivery.md). Although Amazon SNS adds properties to the message using the capitalization shown in this list, column names in Amazon Redshift tables appear in all lowercase characters. To transform the JSON metadata for the Amazon Redshift endpoint, you can use the SQL `COPY` command. For more information, see [Copy from JSON examples](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#r_COPY_command_examples-copy-from-json) and [Load from JSON data using the 'auto ignorecase' option](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#copy-from-json-examples-using-auto-ignorecase) in the *Amazon Redshift Database Developer Guide*. | type | messageid | topicarn | subject | message | timestamp | unsubscribeurl | messageattributes | | --- | --- | --- | --- | --- | --- | --- | --- | | Notification | ea544832-a0d8-581d-9275-108243c46103 | arn:aws:sns:us-east-1:111111111111:my-topic | Sample subject | Sample message | 2020-12-02T00:33:32.272Z | https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:111111111111:my-topic:326deeeb-cbf4-45da-b92b-ca77a247813b | \$1\$1"my\$1attribute\$1":\$1\$1"Type\$1":\$1"String\$1",\$1"Value\$1":\$1"my\$1value\$1"\$1\$1 | | Notification | ab124832-a0d8-581d-9275-108243c46114 | arn:aws:sns:us-east-1:111111111111:my-topic | Sample subject 2 | Sample message 2 | 2020-12-03T00:18:11.129Z | https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:111111111111:my-topic:326deeeb-cbf4-45da-b92b-ca77a247813b | \$1\$1"my\$1attribute2\$1":\$1\$1"Type\$1":\$1"String\$1",\$1"Value\$1":\$1"my\$1value\$1"\$1\$1 | | Notification | ce644832-a0d8-581d-9275-108243c46125 | arn:aws:sns:us-east-1:111111111111:my-topic | Sample subject 3 | Sample message 3 | 2020-12-09T00:08:44.405Z | https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:111111111111:my-topic:326deeeb-cbf4-45da-b92b-ca77a247813b | \$1\$1"my\$1attribute3\$1":\$1\$1"Type\$1":\$1"String\$1",\$1"Value\$1":\$1"my\$1value\$1"\$1\$1 | For more information about fanning out notifications to Amazon Redshift endpoints, see [Configuring Amazon SNS message delivery and analysis in Amazon Redshift destinations](firehose-redshift-destinations.md). # Analyzing Amazon SNS messages stored in Amazon Redshift destinations Analyzing messages stored in Amazon Redshift destinations This topic describes how to analyze Amazon SNS messages that are sent through delivery streams to Amazon Redshift destinations. **To analyze SNS messages sent through Firehose delivery streams to Amazon Redshift destinations** 1. Configure your Amazon Redshift resources. For instructions, see [Getting started with Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html) in the *Amazon Redshift Getting Started Guide*. 1. Configure your delivery stream. For instructions, see [Choose Amazon Redshift for Your Destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-redshift) in the *Amazon Data Firehose Developer Guide*. 1. Run a query. For more information, see [Querying a database using the query editor](https://docs.aws.amazon.com/redshift/latest/mgmt/query-editor.html) in the *Amazon Redshift Management Guide*. ## Example query For this example query, assume the following: + Messages are stored in the `notifications` table in the default `public` schema. + The `Timestamp` property from the SNS message is stored in the table's `timestamp` column with a column data type of `timestamptz`. **Note** To transform the JSON metadata for the Amazon Redshift endpoint, you can use the SQL `COPY` command. For more information, see [Copy from JSON examples](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#r_COPY_command_examples-copy-from-json) and [Load from JSON data using the 'auto ignorecase' option](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#copy-from-json-examples-using-auto-ignorecase) in the *Amazon Redshift Database Developer Guide*. The following query returns all SNS messages received in the specified date range: ``` SELECT * FROM public.notifications WHERE timestamp > '2020-12-01T09:00:00.000Z' AND timestamp < '2020-12-02T09:00:00.000Z'; ``` # Configuring Amazon SNS message delivery to HTTP destinations using Configuring message delivery to HTTP destinations using This topic explains how delivery streams publish data to HTTP endpoints. ![\[A publisher to an Amazon SNS topic, which then distributes the messages to multiple Amazon SQS queues. These messages are processed by Lambda functions and also sent through an Data Firehose delivery stream to an HTTP endpoint. This setup showcases how AWS services work together to facilitate message handling and integration with external HTTP services.\]](http://docs.aws.amazon.com/sns/latest/dg/images/firehose-architecture-http.png) **Topics** + [Notification format for delivery to HTTP destinations](firehose-delivered-message-format-http.md) # Amazon SNS notification format for delivery to HTTP destinations Notification format for delivery to HTTP destinations Here’s an example of an HTTP POST request body from Amazon SNS, sent through an delivery stream to an HTTP endpoint. The Amazon SNS notification is encoded as a base64 payload within the records property. **Note** In this example, raw message delivery is disabled for the published message. For more information about raw delivery, see [Amazon SNS raw message delivery](sns-large-payload-raw-message-delivery.md). ``` "body": { "requestId": "ebc9e8b2-fce3-4aef-a8f1-71698bf8175f", "timestamp": 1606255960435, "records": [ { "data": "eyJUeXBlIjoiTm90aWZpY2F0aW9uIiwiTWVzc2FnZUlkIjoiMjFkMmUzOGQtMmNhYi01ZjYxLTliYTItYmJiYWFhYzg0MGY2IiwiVG9waWNBcm4iOiJhcm46YXdzOnNuczp1cy1lYXN0LTE6MTExMTExMTExMTExOm15LXRvcGljIiwiTWVzc2FnZSI6IlNhbXBsZSBtZXNzYWdlIGZvciBBbWF6b24gS2luZXNpcyBEYXRhIEZpcmVob3NlIGVuZHBvaW50cyIsIlRpbWVzdGFtcCI6IjIwMjAtMTEtMjRUMjI6MDc6MzEuNjY3WiIsIlVuc3Vic2NyaWJlVVJMIjoiaHR0cHM6Ly9zbnMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1VbnN1YnNjcmliZSZTdWJzY3JpcHRpb25Bcm49YXJuOmF3czpzbnM6MTExMTExMTExMTExOm15LXRvcGljOjAxYjY5MTJjLTAwNzAtNGQ4Yi04YjEzLTU1NWJmYjc2ZTdkNCJ9" } ] } ```