# Using Amazon SNS for application-to-application messaging
Amazon SNS simplifies application-to-application (A2A) messaging by separating publishers from subscribers, which supports microservices, distributed systems, and serverless applications. Messages are sent to Amazon SNS topics, where they can be filtered and delivered to subscribers like Lambda, Amazon SQS, or HTTP endpoints. If delivery fails, the messages are stored in a dead-letter queue for further analysis or reprocessing.
![\[Amazon SNS facilitates application-to-application messaging by decoupling publishers from subscribers using topics. Messages from systems or services are routed through an Amazon SNS topic, where they can be filtered and distributed to subscribers like Lambda, Amazon SQS, or email systems. If delivery fails, messages are stored in a dead-letter queue for later analysis or reprocessing.\]](http://docs.aws.amazon.com/sns/latest/dg/images/sns-a2a-overview.png)
# Fanout to Firehose delivery streams
You can subscribe [ delivery streams](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) to Amazon SNS topics, allowing you to send notifications to additional storage and analytics endpoints. Messages published to an Amazon SNS topic are sent to the subscribed Firehose delivery stream, and delivered to the destination as configured in Firehose. A subscription owner can subscribe up to five Firehose delivery streams to an Amazon SNS topic. Each Firehose delivery stream has a [default quota](https://docs.aws.amazon.com/firehose/latest/dev/limits.html) for requests and throughput per second. This limit could result in more messages published (inbound traffic) than delivered (outbound traffic). When there's more inbound than outbound traffic, your subscription can accumulate a large message backlog, causing high message delivery latency. You can request an [increase in quota](https://support.console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase) based on the publish rate to avoid adverse impact on your workload.
Through Firehose delivery streams, you can fan out Amazon SNS notifications to Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service (OpenSearch Service), and to third-party service providers such as Datadog, New Relic, MongoDB, and Splunk.
For example, you can use this functionality to permanently store messages sent to a topic in an Amazon S3 bucket for compliance, archival, or other purposes. To do this, create a Firehose delivery stream with an Amazon S3 bucket destination, and subscribe that delivery stream to the Amazon SNS topic. As another example, to perform analysis on messages sent to an Amazon SNS topic, create a delivery stream with an OpenSearch Service index destination. You can then subscribe the Firehose delivery stream to the Amazon SNS topic.
Amazon SNS also supports message delivery status logging for notifications sent to Firehose endpoints. For more information, see [Amazon SNS message delivery status](sns-topic-attributes.md).
# Prerequisites for subscribing Firehose delivery streams to Amazon SNS topics
To subscribe an delivery stream to an SNS topic, your AWS account must have:
+ A standard SNS topic. For more information, see [Creating an Amazon SNS topic](sns-create-topic.md).
+ A Firehose delivery stream. For more information, see [Creating an Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html) and [Grant Your Application Access to Your Firehose Resources](https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html#access-to-firehose) in the *Amazon Data Firehose Developer Guide*.
+ An AWS Identity and Access Management (IAM) role that trusts the Amazon SNS service principal and has permission to write to the delivery stream. You'll enter this role's Amazon Resource Name (ARN) as the `SubscriptionRoleARN` when you create the subscription. Amazon SNS assumes this role, which allows Amazon SNS to put records in the Firehose delivery stream.
The following example policy shows the recommended permissions:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Action": [
"firehose:DescribeDeliveryStream",
"firehose:ListDeliveryStreams",
"firehose:ListTagsForDeliveryStream",
"firehose:PutRecord",
"firehose:PutRecordBatch"
],
"Resource": [
"arn:aws:firehose:us-east-1:111111111111:deliverystream/firehose-sns-delivery-stream"
],
"Effect": "Allow"
}
]
}
```
------
To provide full permission for using Firehose, you can also use the AWS managed policy `AmazonKinesisFirehoseFullAccess`. Or, to provide stricter permissions for using Firehose, you can create your own policy. At minimum, the policy must provide permission to run the `PutRecord` operation on a specific delivery stream.
In all cases, you must also edit the trust relationship to include the Amazon SNS service principal. For example:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
------
For more information on creating roles, see [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) in the *IAM User Guide*.
After you've completed these requirements, you can [subscribe the delivery stream to the SNS topic](firehose-endpoints-subscribe.md).
# Subscribing a Firehose delivery stream to an Amazon SNS topic
To deliver Amazon SNS notifications to [ delivery streams](sns-firehose-as-subscriber.md), first make sure that you've addressed all the [prerequisites](prereqs-kinesis-data-firehose.md). For a list of supported endpoints, see [ endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/fh.html) in the *Amazon Web Services General Reference*.
**To subscribe a Firehose delivery stream to a topic**
1. Sign in to the [Amazon SNS console](https://console.aws.amazon.com/sns/home).
1. In the navigation pane, choose **Subscriptions**.
1. On the **Subscriptions** page, choose **Create subscription**.
1. On the **Create subscription** page, in the **Details** section, do the following:
1. For **Topic ARN**, choose the Amazon Resource Name (ARN) of a standard topic.
1. For **Protocol**, choose **Firehose**.
1. For **Endpoint**, choose the ARN of a Firehose delivery stream that can receive notifications from Amazon SNS.
1. For **Subscription role ARN**, specify the ARN of the AWS Identity and Access Management (IAM) role that you created for writing to Firehose delivery streams. For more information, see [Prerequisites for subscribing Firehose delivery streams to Amazon SNS topics](prereqs-kinesis-data-firehose.md).
1. (Optional) To remove any Amazon SNS metadata from published messages, choose **Enable raw message delivery**. For more information, see [Amazon SNS raw message delivery](sns-large-payload-raw-message-delivery.md).
1. (Optional) To configure a filter policy, expand the **Subscription filter policy** section. For more information, see [Amazon SNS subscription filter policies](sns-subscription-filter-policies.md).
1. (Optional) To configure a dead-letter queue for the subscription, expand the **Redrive policy (dead-letter queue)** section. For more information, see [Amazon SNS dead-letter queues](sns-dead-letter-queues.md).
1. Choose **Create subscription**.
The console creates the subscription and opens the subscription's **Details** page.
# Managing Amazon SNS messages across multiple delivery stream destinations
[ delivery streams](sns-firehose-as-subscriber.md) allow you to manage Amazon SNS messages across multiple destinations, enabling integration with Amazon S3, Amazon OpenSearch Service, Amazon Redshift, and HTTP endpoints for storage, indexing, and analysis. By properly configuring message formatting and delivery, you can store Amazon SNS notifications in Amazon S3 for later processing, analyze structured message data using Amazon Athena, index messages in OpenSearch for real-time search and visualization, and structure archives in Amazon Redshift for advanced querying.
# Storing and analyzing Amazon SNS messages in Amazon S3 destinations
This topic explains how delivery streams publish data to Amazon Simple Storage Service (Amazon S3).
![\[The integration and workflow of Amazon services for message handling. It shows how a publisher sends messages to an Amazon SNS topic, which then fans out messages to multiple Amazon SQS queues and an Data Firehose delivery stream. From there, messages can be processed by Lambda functions or stored persistently in an Amazon S3 bucket.\]](http://docs.aws.amazon.com/sns/latest/dg/images/firehose-architecture-s3.png)
**Topics**
+ [Formatting notifications for storage in Amazon S3 destinations](firehose-archived-message-format-S3.md)
+ [Analyzing messages stored in Amazon S3 using Athena](firehose-message-analysis-s3.md)
# Formatting Amazon SNS notifications for storage in Amazon S3 destinations
The following example shows an Amazon SNS notification sent to an Amazon Simple Storage Service (Amazon S3) bucket, with indentation for readability.
**Note**
In this example, raw message delivery is disabled for the published message. When raw message delivery is disabled, Amazon SNS adds JSON metadata to the message, including these properties:
`Type`
`MessageId`
`TopicArn`
`Subject`
`Timestamp`
`UnsubscribeURL`
`MessageAttributes`
For more information about raw delivery, see [Amazon SNS raw message delivery](sns-large-payload-raw-message-delivery.md).
```
{
"Type": "Notification",
"MessageId": "719a6bbf-f51b-5320-920f-3385b5e9aa56",
"TopicArn": "arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic",
"Subject": "My 1st subject",
"Message": "My 1st body",
"Timestamp": "2020-11-26T23:48:02.032Z",
"UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic:0b410f3c-ee5e-49d8-b59b-3b4aa6d8fcf5",
"MessageAttributes": {
"myKey1": {
"Type": "String",
"Value": "myValue1"
},
"myKey2": {
"Type": "String",
"Value": "myValue2"
}
}
}
```
The following example shows three SNS messages sent through an delivery stream to the same Amazon S3 bucket. Buffering is applied, and line breaks separate each message.
```
{"Type":"Notification","MessageId":"d7d2513e-6126-5d77-bbe2-09042bd0a03a","TopicArn":"arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic","Subject":"My 1st subject","Message":"My 1st body","Timestamp":"2020-11-27T00:30:46.100Z","UnsubscribeURL":"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:313276652360:my-kinesis-test-topic:0b410f3c-ee5e-49d8-b59b-3b4aa6d8fcf5","MessageAttributes":{"myKey1":{"Type":"String","Value":"myValue1"},"myKey2":{"Type":"String","Value":"myValue2"}}}
{"Type":"Notification","MessageId":"0c0696ab-7733-5bfb-b6db-ce913c294d56","TopicArn":"arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic","Subject":"My 2nd subject","Message":"My 2nd body","Timestamp":"2020-11-27T00:31:22.151Z","UnsubscribeURL":"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:313276652360:my-kinesis-test-topic:0b410f3c-ee5e-49d8-b59b-3b4aa6d8fcf5","MessageAttributes":{"myKey1":{"Type":"String","Value":"myValue1"}}}
{"Type":"Notification","MessageId":"816cd54d-8cfa-58ad-91c9-8d77c7d173aa","TopicArn":"arn:aws:sns:us-east-1:333333333333:my-kinesis-test-topic","Subject":"My 3rd subject","Message":"My 3rd body","Timestamp":"2020-11-27T00:31:39.755Z","UnsubscribeURL":"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:313276652360:my-kinesis-test-topic:0b410f3c-ee5e-49d8-b59b-3b4aa6d8fcf5"}
```
# Analyzing Amazon SNS messages stored in Amazon S3 using Athena
This page explains how to analyze Amazon SNS messages that are sent through delivery streams to Amazon Simple Storage Service (Amazon S3) destinations.
**To analyze SNS messages sent through Firehose delivery streams to Amazon S3 destinations**
1. Configure your Amazon S3 resources. For instructions, see [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/CreatingABucket.html) in the *Amazon Simple Storage Service User Guide* and [Working with Amazon S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) in the *Amazon Simple Storage Service User Guide*.
1. Configure your delivery stream. For instructions, see [Choose Amazon S3 for Your Destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-s3) in the *Amazon Data Firehose Developer Guide*.
1. Use [Amazon Athena](https://console.aws.amazon.com/athena) to query the Amazon S3 objects using standard SQL. For more information, see [Getting Started](https://docs.aws.amazon.com/athena/latest/ug/getting-started.html) in the *Amazon Athena User Guide*.
## Example query
For this example query, assume the following:
+ Messages are stored in the `notifications` table in the `default` schema.
+ The `notifications` table includes a `timestamp` column with a type of `string`.
The following query returns all SNS messages received in the specified date range:
```
SELECT *
FROM default.notifications
WHERE from_iso8601_timestamp(timestamp) BETWEEN TIMESTAMP '2020-12-01 00:00:00' AND TIMESTAMP '2020-12-02 00:00:00';
```
# Integrating Amazon SNS messages with Amazon OpenSearch Service destinations
This section explains how delivery streams publish data to Amazon OpenSearch Service (OpenSearch Service).
![\[A publisher sends messages to an Amazon SNS topic, which then distributes these messages to multiple Amazon SQS queues. Messages from these queues can be processed by Lambda functions or sent through an Data Firehose delivery stream to an Amazon OpenSearch Service, creating a searchable message index. This setup demonstrates an advanced message routing and processing scenario using AWS services.\]](http://docs.aws.amazon.com/sns/latest/dg/images/firehose-architecture-es.png)
**Topics**
+ [Archived message format](firehose-archived-message-format-elasticsearch.md)
+ [Analyzing messages](firehose-message-analysis-elasticsearch.md)
# Storing and formatting Amazon SNS Notifications in OpenSearch Service indices
The following example demonstrates an Amazon SNS notification sent to an Amazon OpenSearch Service (OpenSearch Service) index called `my-index`. This index has a time filter field on the `Timestamp` field. The SNS notification is placed in the `_source` property of the payload.
**Note**
In this example, raw message delivery is disabled for the published message. When raw message delivery is disabled, Amazon SNS adds JSON metadata to the message, including these properties:
`Type`
`MessageId`
`TopicArn`
`Subject`
`Timestamp`
`UnsubscribeURL`
`MessageAttributes`
For more information about raw delivery, see [Amazon SNS raw message delivery](sns-large-payload-raw-message-delivery.md).
```
{
"_index": "my-index",
"_type": "_doc",
"_id": "49613100963111323203250405402193283794773886550985932802.0",
"_version": 1,
"_score": null,
"_source": {
"Type": "Notification",
"MessageId": "bf32e294-46e3-5dd5-a6b3-bad65162e136",
"TopicArn": "arn:aws:sns:us-east-1:111111111111:my-topic",
"Subject": "Sample subject",
"Message": "Sample message",
"Timestamp": "2020-12-02T22:29:21.189Z",
"UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:111111111111:my-topic:b5aa9bc1-9c3d-452b-b402-aca2cefc63c9",
"MessageAttributes": {
"my_attribute": {
"Type": "String",
"Value": "my_value"
}
}
},
"fields": {
"Timestamp": [
"2020-12-02T22:29:21.189Z"
]
},
"sort": [
1606948161189
]
}
```
# Analyzing Amazon SNS messages for OpenSearch Service destinations
This topic explains how to analyze Amazon SNS messages sent through delivery streams to Amazon OpenSearch Service (OpenSearch Service) destinations.
**To analyze SNS messages sent through Firehose delivery streams to OpenSearch Service destinations**
1. Configure your OpenSearch Service resources. For instructions, see [Getting Started with Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/es-gsg.html) in the *Amazon OpenSearch Service Developer Guide*.
1. Configure your delivery stream. For instructions, see [Choose OpenSearch Service for Your Destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-elasticsearch) in the *Amazon Data Firehose Developer Guide*.
1. Run a query using OpenSearch Service queries and Kibana. For more information, see [Step 3: Search Documents in an OpenSearch Service Domain](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/es-gsg-search.html) and [Kibana](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/es-kibana.html) in the *Amazon OpenSearch Service Developer Guide*.
## Example query
The following example queries the `my-index` index for all SNS messages received in the specified date range:
```
POST https://search-my-domain.us-east-1.es.amazonaws.com/my-index/_search
{
"query": {
"bool": {
"filter": [
{
"range": {
"Timestamp": {
"gte": "2020-12-08T00:00:00.000Z",
"lte": "2020-12-09T00:00:00.000Z",
"format": "strict_date_optional_time"
}
}
}
]
}
}
}
```
# Configuring Amazon SNS message delivery and analysis in Amazon Redshift destinations
This topic explains how to fan out Amazon SNS notifications to an delivery stream, which then publishes data to Amazon Redshift. With this setup, you can connect to the Amazon Redshift database and use a SQL query tool to retrieve Amazon SNS messages that match specific criteria.
![\[Messages published by a sender to an Amazon SNS topic are distributed to multiple Amazon SQS queues for processing by Lambda functions, and also sent through an Data Firehose delivery stream to an Amazon Redshift cluster for storage and analysis in a message data warehouse. This setup demonstrates a robust message handling and data warehousing architecture using AWS services.\]](http://docs.aws.amazon.com/sns/latest/dg/images/firehose-architecture-rs.png)
**Topics**
+ [Structuring message archives in Amazon Redshift tables](firehose-archive-table-structure-redshift.md)
+ [Analyzing messages stored in Amazon Redshift destinations](firehose-message-analysis-redshift.md)
# Structuring Amazon SNS message archives in Amazon Redshift tables
For Amazon Redshift endpoints, Amazon SNS messages are archived as rows in a table. Here's an example of how the data is stored:
**Note**
In this example, raw message delivery is disabled for the published message. When raw message delivery is disabled, Amazon SNS adds JSON metadata to the message, including these properties:
`Type`
`MessageId`
`TopicArn`
`Subject`
`Message`
`Timestamp`
`UnsubscribeURL`
`MessageAttributes`
For more information about raw delivery, see [Amazon SNS raw message delivery](sns-large-payload-raw-message-delivery.md).
Although Amazon SNS adds properties to the message using the capitalization shown in this list, column names in Amazon Redshift tables appear in all lowercase characters. To transform the JSON metadata for the Amazon Redshift endpoint, you can use the SQL `COPY` command. For more information, see [Copy from JSON examples](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#r_COPY_command_examples-copy-from-json) and [Load from JSON data using the 'auto ignorecase' option](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#copy-from-json-examples-using-auto-ignorecase) in the *Amazon Redshift Database Developer Guide*.
| type | messageid | topicarn | subject | message | timestamp | unsubscribeurl | messageattributes |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Notification | ea544832-a0d8-581d-9275-108243c46103 | arn:aws:sns:us-east-1:111111111111:my-topic | Sample subject | Sample message | 2020-12-02T00:33:32.272Z | https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:111111111111:my-topic:326deeeb-cbf4-45da-b92b-ca77a247813b | \$1\$1"my\$1attribute\$1":\$1\$1"Type\$1":\$1"String\$1",\$1"Value\$1":\$1"my\$1value\$1"\$1\$1 |
| Notification | ab124832-a0d8-581d-9275-108243c46114 | arn:aws:sns:us-east-1:111111111111:my-topic | Sample subject 2 | Sample message 2 | 2020-12-03T00:18:11.129Z | https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:111111111111:my-topic:326deeeb-cbf4-45da-b92b-ca77a247813b | \$1\$1"my\$1attribute2\$1":\$1\$1"Type\$1":\$1"String\$1",\$1"Value\$1":\$1"my\$1value\$1"\$1\$1 |
| Notification | ce644832-a0d8-581d-9275-108243c46125 | arn:aws:sns:us-east-1:111111111111:my-topic | Sample subject 3 | Sample message 3 | 2020-12-09T00:08:44.405Z | https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:111111111111:my-topic:326deeeb-cbf4-45da-b92b-ca77a247813b | \$1\$1"my\$1attribute3\$1":\$1\$1"Type\$1":\$1"String\$1",\$1"Value\$1":\$1"my\$1value\$1"\$1\$1 |
For more information about fanning out notifications to Amazon Redshift endpoints, see [Configuring Amazon SNS message delivery and analysis in Amazon Redshift destinations](firehose-redshift-destinations.md).
# Analyzing Amazon SNS messages stored in Amazon Redshift destinations
This topic describes how to analyze Amazon SNS messages that are sent through delivery streams to Amazon Redshift destinations.
**To analyze SNS messages sent through Firehose delivery streams to Amazon Redshift destinations**
1. Configure your Amazon Redshift resources. For instructions, see [Getting started with Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html) in the *Amazon Redshift Getting Started Guide*.
1. Configure your delivery stream. For instructions, see [Choose Amazon Redshift for Your Destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-redshift) in the *Amazon Data Firehose Developer Guide*.
1. Run a query. For more information, see [Querying a database using the query editor](https://docs.aws.amazon.com/redshift/latest/mgmt/query-editor.html) in the *Amazon Redshift Management Guide*.
## Example query
For this example query, assume the following:
+ Messages are stored in the `notifications` table in the default `public` schema.
+ The `Timestamp` property from the SNS message is stored in the table's `timestamp` column with a column data type of `timestamptz`.
**Note**
To transform the JSON metadata for the Amazon Redshift endpoint, you can use the SQL `COPY` command. For more information, see [Copy from JSON examples](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#r_COPY_command_examples-copy-from-json) and [Load from JSON data using the 'auto ignorecase' option](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#copy-from-json-examples-using-auto-ignorecase) in the *Amazon Redshift Database Developer Guide*.
The following query returns all SNS messages received in the specified date range:
```
SELECT *
FROM public.notifications
WHERE timestamp > '2020-12-01T09:00:00.000Z' AND timestamp < '2020-12-02T09:00:00.000Z';
```
# Configuring Amazon SNS message delivery to HTTP destinations using
This topic explains how delivery streams publish data to HTTP endpoints.
![\[A publisher to an Amazon SNS topic, which then distributes the messages to multiple Amazon SQS queues. These messages are processed by Lambda functions and also sent through an Data Firehose delivery stream to an HTTP endpoint. This setup showcases how AWS services work together to facilitate message handling and integration with external HTTP services.\]](http://docs.aws.amazon.com/sns/latest/dg/images/firehose-architecture-http.png)
**Topics**
+ [Notification format for delivery to HTTP destinations](firehose-delivered-message-format-http.md)
# Amazon SNS notification format for delivery to HTTP destinations
Here’s an example of an HTTP POST request body from Amazon SNS, sent through an delivery stream to an HTTP endpoint. The Amazon SNS notification is encoded as a base64 payload within the records property.
**Note**
In this example, raw message delivery is disabled for the published message. For more information about raw delivery, see [Amazon SNS raw message delivery](sns-large-payload-raw-message-delivery.md).
```
"body": {
"requestId": "ebc9e8b2-fce3-4aef-a8f1-71698bf8175f",
"timestamp": 1606255960435,
"records": [
{
"data": "eyJUeXBlIjoiTm90aWZpY2F0aW9uIiwiTWVzc2FnZUlkIjoiMjFkMmUzOGQtMmNhYi01ZjYxLTliYTItYmJiYWFhYzg0MGY2IiwiVG9waWNBcm4iOiJhcm46YXdzOnNuczp1cy1lYXN0LTE6MTExMTExMTExMTExOm15LXRvcGljIiwiTWVzc2FnZSI6IlNhbXBsZSBtZXNzYWdlIGZvciBBbWF6b24gS2luZXNpcyBEYXRhIEZpcmVob3NlIGVuZHBvaW50cyIsIlRpbWVzdGFtcCI6IjIwMjAtMTEtMjRUMjI6MDc6MzEuNjY3WiIsIlVuc3Vic2NyaWJlVVJMIjoiaHR0cHM6Ly9zbnMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1VbnN1YnNjcmliZSZTdWJzY3JpcHRpb25Bcm49YXJuOmF3czpzbnM6MTExMTExMTExMTExOm15LXRvcGljOjAxYjY5MTJjLTAwNzAtNGQ4Yi04YjEzLTU1NWJmYjc2ZTdkNCJ9"
}
]
}
```
# Amazon SNS message archiving and analytics: An example use case for airline ticketing platforms
This topic provides a tutorial for a common use case of archiving and analyzing Amazon SNS messages.
The setting of this use case is an airline ticketing platform that operates in a regulated environment.
1. The platform is subject to a compliance framework that requires the company to archive all ticket sales for at least five years.
1. To meet the compliance goal on data retention, the company subscribes an delivery stream to an existing Amazon SNS topic.
1. The destination for the delivery stream is an Amazon Simple Storage Service (Amazon S3) bucket. With this configuration, all events published to the SNS topic are archived in the Amazon S3 bucket.
The following diagram shows the architecture of this configuration:
![\[An AWS architecture for an airline ticketing platform, illustrating how ticket sales data is processed and archived. It shows the flow of data from a Lambda function through an Amazon SNS topic, which then distributes messages to Amazon SQS queues for payment processing and fraud detection, handled by respective Lambda functions. The data is also streamed via Data Firehose to an Amazon S3 bucket for long-term archival, supporting compliance with data retention requirements. This setup enables the platform to run detailed analytics on ticket sales data using tools like Amazon Athena.\]](http://docs.aws.amazon.com/sns/latest/dg/images/sns-archiving-use-case.png)
To run analytics and gain insights on ticket sales, the company runs SQL queries using Amazon Athena. For example, the company can query to learn about the most popular destinations and the most frequent flyers.
To create the AWS resources for this use case, you can use the AWS Management Console or an CloudFormation template.
**Topics**
+ [Setting-up initial AWS resources for message archiving and analytics](firehose-example-initial-resources.md)
+ [Setting-up a Firehose delivery stream for message archiving](firehose-example-create-delivery-stream.md)
+ [Subscribing the delivery stream to the topic](firehose-example-subscribe-delivery-stream-to-topic.md)
+ [Testing and querying a configuration for effective data management](firehose-example-test-and-query.md)
+ [Automating message archiving with an CloudFormation template](firehose-example-cfn.md)
# Setting-up initial AWS resources for Amazon SNS message archiving and analytics
This topic describes how to create the resources needed for the [message archiving and analytics example use case](firehose-example-use-case.md):
+ An Amazon Simple Storage Service (Amazon S3) bucket
+ Two Amazon Simple Queue Service (Amazon SQS) queues
+ An Amazon SNS topic
+ Two Amazon SQS subscriptions to the Amazon SNS topic
**To create the initial resources**
1. Create the Amazon S3 bucket:
1. Open the [Amazon S3 console](https://console.aws.amazon.com/s3/home).
1. Choose **Create bucket**.
1. For **Bucket name**, enter a globally unique name. Keep the other fields as the defaults.
1. Choose **Create bucket**.
For more information about Amazon S3 buckets, see [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/CreatingABucket.html) in the *Amazon Simple Storage Service User Guide* and [Working with Amazon S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) in the *Amazon Simple Storage Service User Guide*.
1. Create the two Amazon SQS queues:
1. Open the [Amazon SQS console](https://console.aws.amazon.com/sqs/home).
1. Choose **Create queue**.
1. For **Type**, choose **Standard**.
1. For **Name**, enter **ticketPaymentQueue**.
1. Under **Access policy**, for **Choose method**, choose **Advanced**.
1. In the JSON policy box, paste the following policy:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "sqs:SendMessage",
"Resource": "*",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:ticketTopic"
}
}
}
]
}
```
------
In this access policy, replace the AWS account number (*123456789012*) with your own, and change the AWS Region (*us-east-1*) accordingly.
1. Choose **Create queue**.
1. Repeat these steps to create a second SQS queue named **ticketFraudQueue**.
For more information on creating SQS queues, see [Creating an Amazon SQS queue (console)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-create-queue.html) in the *Amazon Simple Queue Service Developer Guide*.
1. Create the SNS topic:
1. Open the [Topics page](https://console.aws.amazon.com/sns/home#/topics) of the Amazon SNS console.
1. Choose **Create topic**.
1. Under **Details**, for **Type**, choose **Standard**.
1. For **Name**, enter **ticketTopic**.
1. Choose **Create topic**.
For more information on creating SNS topics, see [Creating an Amazon SNS topic](sns-create-topic.md).
1. Subscribe both SQS queues to the SNS topic:
1. In the [Amazon SNS console](https://console.aws.amazon.com/sns/home#/topics), on the **ticketTopic** topic's details page, choose **Create subscription**.
1. Under **Details**, for **Protocol**, choose **Amazon SQS**.
1. For **Endpoint**, choose the Amazon Resource Name (ARN) of the **ticketPaymentQueue** queue.
1. Choose **Create subscription**.
1. Repeat these steps to create a second subscription using the ARN of the **ticketFraudQueue** queue.
For more information on subscribing to SNS topics, see [Creating a subscription to an Amazon SNS topic](sns-create-subscribe-endpoint-to-topic.md). You can also subscribe SQS queues to SNS topics from the Amazon SQS console. For more information, see [Subscribing an Amazon SQS queue to an Amazon SNS topic (console)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-subscribe-queue-sns-topic.html) in the *Amazon Simple Queue Service Developer Guide*.
You've created the initial resources for this example use case. To continue, see [Setting-up a Amazon Data Firehose delivery stream for Amazon SNS message archiving](firehose-example-create-delivery-stream.md).
# Setting-up a Amazon Data Firehose delivery stream for Amazon SNS message archiving
This topic explains how to create the Amazon Data Firehose delivery stream for the [message archiving and analytics example use case](firehose-example-use-case.md).
**To create the Amazon Data Firehose delivery stream**
1. Open the [Amazon Data Firehose services console](https://console.aws.amazon.com/kinesis/home).
1. Choose **Firehose** and then choose **Create delivery stream**.
1. On the **New delivery stream** page, for **Delivery stream name**, enter **ticketUploadStream**, and then choose **Next**.
1. On the **Process records** page, choose **Next**.
1. On the **Choose a destination** page, do the following:
1. For **Destination**, choose **Amazon S3**.
1. Under **S3 destination**, for **S3 bucket**, choose the S3 bucket that you [created initially](firehose-example-initial-resources.md).
1. Choose **Next**.
1. On the **Configure settings** page, for **S3 buffer conditions**, do the following:
+ For **Buffer size**, enter **1**.
+ For **Buffer interval**, enter **60**.
Using these values for the Amazon S3 buffer lets you quickly test the configuration. The first condition that is satisfied triggers data delivery to the S3 bucket.
1. On the **Configure settings** page, for **Permissions**, choose to create an AWS Identity and Access Management (IAM) role with the required permissions assigned automatically. Then choose **Next**.
1. On the **Review** page, choose **Create delivery stream**.
1. From the **Amazon Data Firehose delivery streams page,** choose the delivery stream you just created (**ticketUploadStream**). On the **Details** tab, note the stream's Amazon Resource Name (ARN) for later.
For more information on creating delivery streams, see [Creating an Amazon Data Firehose Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html) in the *Amazon Data Firehose Developer Guide*. For more information on creating IAM roles, see [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) in the *IAM User Guide*.
You've created the Firehose delivery stream with the required permissions. To continue, see [Subscribing the Firehose delivery stream to the Amazon SNS topic](firehose-example-subscribe-delivery-stream-to-topic.md).
# Subscribing the Firehose delivery stream to the Amazon SNS topic
This topic explains how to create the following resources for the [message archiving and analytics example use case](firehose-example-use-case.md):
+ The AWS Identity and Access Management (IAM) role that allows the Amazon SNS subscription to put records on the delivery stream.
+ The Firehose delivery stream subscription to the Amazon SNS topic.
**To create the IAM role for the Amazon SNS subscription**
1. Open the [Roles page](https://console.aws.amazon.com/iam/home?#/roles) of the IAM console.
1. Choose **Create role**.
1. For **Select type of trusted entity**, choose **AWS service**.
1. For **Choose a use case**, choose **SNS**. Then choose **Next: Permissions**.
1. Choose **Next: Tags**.
1. Choose **Next: Review**.
1. On the **Review** page, for **Role name**, enter **ticketUploadStreamSubscriptionRole**. Then choose **Create role**.
1. When the role is created, choose its name (**ticketUploadStreamSubscriptionRole**).
1. On the role's **Summary** page, choose **Add inline policy**.
1. On the **Create policy** page, choose the **JSON** tab, and then paste the following policy into the box:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Action": [
"firehose:DescribeDeliveryStream",
"firehose:ListDeliveryStreams",
"firehose:ListTagsForDeliveryStream",
"firehose:PutRecord",
"firehose:PutRecordBatch"
],
"Resource": [
"arn:aws:firehose:us-east-1:123456789012:deliverystream/ticketUploadStream"
],
"Effect": "Allow"
}
]
}
```
------
In this policy, replace the AWS account number (*123456789012*) with your own, and change the AWS Region (*us-east-1*) accordingly.
1. Choose **Review policy**.
1. On the **Review policy** page, for **Name**, enter **FirehoseSnsPolicy**. Then choose **Create policy**.
1. On the role's **Summary** page, note the **Role ARN** for later.
For more information on creating IAM roles, see [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) in the *IAM User Guide*.
**To subscribe the Firehose delivery stream to the SNS topic**
1. Open the [Topics page](https://console.aws.amazon.com/sns/home#/topics) of the Amazon SNS console.
1. On the **Subscriptions**, tab, choose **Create subscription**.
1. Under **Details**, for **Protocol**, choose ****.
1. For **Endpoint**, enter the Amazon Resource Name (ARN) of the **ticketUploadStream** delivery stream that you created earlier. For example, enter **arn:aws:firehose:us-east-1:123456789012:deliverystream/ticketUploadStream**.
1. For **Subscription role ARN**, enter the ARN of the **ticketUploadStreamSubscriptionRole** IAM role that you created earlier. For example, enter **arn:aws:iam::123456789012:role/ticketUploadStreamSubscriptionRole**.
1. Select the **Enable raw message delivery** check box.
1. Choose **Create subscription**.
You've created the IAM role and SNS topic subscription. To continue, see [Testing and querying an Amazon SNS configuration for effective data management](firehose-example-test-and-query.md).
# Testing and querying an Amazon SNS configuration for effective data management
This topic explains how to test the [message archiving and analytics example use case](firehose-example-use-case.md) by publishing a message to the Amazon SNS topic. The instructions include an example query that you can run and adapt to your own needs.
**To test your configuration**
1. Open the [Topics page](https://console.aws.amazon.com/sns/home#/topics) of the Amazon SNS console.
1. Choose the **ticketTopic** topic.
1. Choose **Publish message**.
1. On the **Publish message to topic** page, enter the following for the message body. Add a newline character at the end of the message.
```
{"BookingDate":"2020-12-15","BookingTime":"2020-12-15 04:15:05","Destination":"Miami","FlyingFrom":"Vancouver","TicketNumber":"abcd1234"}
```
Keep all other options as their defaults.
1. Choose **Publish message**.
For more information on publishing messages, see [Publishing an Amazon SNS message](sns-publishing.md).
1. After the delivery stream interval of 60 seconds, open the [Amazon Simple Storage Service (Amazon S3) console](https://console.aws.amazon.com/s3/home) and choose the Amazon S3 bucket that you [created initially](firehose-example-initial-resources.md).
The published message appears in the bucket.
**To query the data**
1. Open the [Amazon Athena console](https://console.aws.amazon.com/athena/home).
1. Run a query.
For example, assume that the `notifications` table in the `default` schema contains the following data:
```
{"BookingDate":"2020-12-15","BookingTime":"2020-12-15 04:15:05","Destination":"Miami","FlyingFrom":"Vancouver","TicketNumber":"abcd1234"}
{"BookingDate":"2020-12-15","BookingTime":"2020-12-15 11:30:15","Destination":"Miami","FlyingFrom":"Omaha","TicketNumber":"efgh5678"}
{"BookingDate":"2020-12-15","BookingTime":"2020-12-15 3:30:10","Destination":"Miami","FlyingFrom":"NewYork","TicketNumber":"ijkl9012"}
{"BookingDate":"2020-12-15","BookingTime":"2020-12-15 12:30:05","Destination":"Delhi","FlyingFrom":"Omaha","TicketNumber":"mnop3456"}
```
To find the top destination, run the following query:
```
SELECT destination
FROM default.notifications
GROUP BY destination
ORDER BY count(*) desc
LIMIT 1;
```
To query for tickets sold during a specific date and time range, run a query like the following:
```
SELECT *
FROM default.notifications
WHERE bookingtime
BETWEEN TIMESTAMP '2020-12-15 10:00:00'
AND TIMESTAMP '2020-12-15 12:00:00';
```
You can adapt both sample queries for your own needs. For more information on using Athena to run queries, see [Getting Started](https://docs.aws.amazon.com/athena/latest/ug/getting-started.html) in the *Amazon Athena User Guide*.
## Cleaning up
To avoid incurring usage charges after you're done testing, delete the following resources that you created during the tutorial:
+ Amazon SNS subscriptions
+ Amazon SNS topic
+ Amazon Simple Queue Service (Amazon SQS) queues
+ Amazon S3 bucket
+ delivery stream
+ AWS Identity and Access Management (IAM) roles and policies
# Automating Amazon SNS message archiving with an CloudFormation template
To automate the deployment of the Amazon SNS [message archiving and analytics example use case](firehose-example-use-case.md), you can use the following YAML template:
```
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Template for creating an SNS archiving use case
Resources:
ticketUploadStream:
DependsOn:
- ticketUploadStreamRolePolicy
Type: AWS::KinesisFirehose::DeliveryStream
Properties:
S3DestinationConfiguration:
BucketARN: !Sub 'arn:${AWS::Partition}:s3:::${ticketArchiveBucket}'
BufferingHints:
IntervalInSeconds: 60
SizeInMBs: 1
CompressionFormat: UNCOMPRESSED
RoleARN: !GetAtt ticketUploadStreamRole.Arn
ticketArchiveBucket:
Type: AWS::S3::Bucket
ticketTopic:
Type: AWS::SNS::Topic
ticketPaymentQueue:
Type: AWS::SQS::Queue
ticketFraudQueue:
Type: AWS::SQS::Queue
ticketQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Statement:
Effect: Allow
Principal:
Service: sns.amazonaws.com
Action:
- sqs:SendMessage
Resource: '*'
Condition:
ArnEquals:
aws:SourceArn: !Ref ticketTopic
Queues:
- !Ref ticketPaymentQueue
- !Ref ticketFraudQueue
ticketUploadStreamSubscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !Ref ticketTopic
Endpoint: !GetAtt ticketUploadStream.Arn
Protocol: firehose
SubscriptionRoleArn: !GetAtt ticketUploadStreamSubscriptionRole.Arn
ticketPaymentQueueSubscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !Ref ticketTopic
Endpoint: !GetAtt ticketPaymentQueue.Arn
Protocol: sqs
ticketFraudQueueSubscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !Ref ticketTopic
Endpoint: !GetAtt ticketFraudQueue.Arn
Protocol: sqs
ticketUploadStreamRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17 '
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: firehose.amazonaws.com
Action: sts:AssumeRole
ticketUploadStreamRolePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: FirehoseticketUploadStreamRolePolicy
PolicyDocument:
Version: '2012-10-17 '
Statement:
- Effect: Allow
Action:
- s3:AbortMultipartUpload
- s3:GetBucketLocation
- s3:GetObject
- s3:ListBucket
- s3:ListBucketMultipartUploads
- s3:PutObject
Resource:
- !Sub 'arn:aws:s3:::${ticketArchiveBucket}'
- !Sub 'arn:aws:s3:::${ticketArchiveBucket}/*'
Roles:
- !Ref ticketUploadStreamRole
ticketUploadStreamSubscriptionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17 '
Statement:
- Effect: Allow
Principal:
Service:
- sns.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: SNSKinesisFirehoseAccessPolicy
PolicyDocument:
Version: '2012-10-17 '
Statement:
- Action:
- firehose:DescribeDeliveryStream
- firehose:ListDeliveryStreams
- firehose:ListTagsForDeliveryStream
- firehose:PutRecord
- firehose:PutRecordBatch
Effect: Allow
Resource:
- !GetAtt ticketUploadStream.Arn
```
# Fanout Amazon SNS notifications to Lambda functions for automated processing
Amazon SNS integrates with AWS Lambda, allowing you to trigger Lambda functions in response to Amazon SNS notifications. When a message is published to an Amazon SNS topic that has a Lambda function subscribed to it, the Lambda function is invoked with the payload of the published message. The Lambda function receives the message payload as an input parameter and can manipulate the information in the message, publish the message to other Amazon SNS topics, or send the message to other AWS services.
Amazon SNS also supports message delivery status attributes for message notifications sent to Lambda endpoints. For more information, see [Amazon SNS message delivery status](sns-topic-attributes.md).
**Topics**
+ [Prerequisites](lambda-prereq.md)
+ [Subscribing a function to a topic](lambda-console.md)
# Prerequisites for integrating Amazon SNS with Lambda functions across regions
To invoke Lambda functions using Amazon SNS notifications, you need the following:
+ A Lambda function
+ An Amazon SNS topic
For information about creating a Lambda function to use with Amazon SNS, see [Using Lambda with Amazon SNS](https://docs.aws.amazon.com/lambda/latest/dg/with-sns-example.html). For information about creating an Amazon SNS topic, see [Create a topic](https://docs.aws.amazon.com/sns/latest/dg/CreateTopic.html).
When you use Amazon SNS to deliver messages from opt-in regions to regions which are enabled by default, you must alter the policy created in the Lambda function by replacing the principal `sns.amazonaws.com` with `sns..amazonaws.com`.
For example, if you want to subscribe a Lambda function in US East (N. Virginia) to an Amazon SNS topic in Asia Pacific (Hong Kong), change the principal in the Lambda function policy to `sns.ap-east-1.amazonaws.com`. Opt-in regions include any regions launched after March 20, 2019, which includes Asia Pacific (Hong Kong), Middle East (Bahrain), EU (Milano), and Africa (Cape Town). Regions launched prior to March 20, 2019 are enabled by default.
**Note**
AWS doesn't support cross-region delivery to Lambda from a region that is enabled by default to an opt-in region. Also, cross-region forwarding of Amazon SNS messages from opt-in regions to other opt-in regions is not supported.
# Subscribing a Lambda function to an Amazon SNS topic
This topic explains how to subscribe a Lambda function to an Amazon SNS topic, enabling the function to be triggered by published messages.
1. Sign in to the [Amazon SNS console](https://console.aws.amazon.com/sns/home).
1. On the navigation panel, choose **Topics**.
1. On the **Topics** page, choose a topic.
1. In the **Subscriptions** section, choose **Create subscription**.
1. On the **Create subscription** page, in the **Details** section, do the following:
1. Verify the chosen **Topic ARN**.
1. For **Protocol** choose AWS Lambda.
1. For **Endpoint** enter the ARN of a function.
1. Choose **Create subscription**.
When a message is published to an Amazon SNS topic that has a Lambda function subscribed to it, the Lambda function is invoked with the payload of the published message. For information about how to use AWS Lambda with Amazon SNS, including a tutorial, see [ Using AWS Lambda with Amazon SNS](https://docs.aws.amazon.com/lambda/latest/dg/with-sns.html).
# Fanout Amazon SNS notifications to Amazon SQS queues for asynchronous processing
[Amazon SNS](https://aws.amazon.com/sns/) works closely with Amazon Simple Queue Service (Amazon SQS). These services provide different benefits for developers. Amazon SNS allows applications to send time-critical messages to multiple subscribers through a “push” mechanism, eliminating the need to periodically check or “poll” for updates. Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components—without requiring each component to be concurrently available. Using Amazon SNS and Amazon SQS together, messages can be delivered to applications that require immediate notification of an event, and also persisted in an Amazon SQS queue for other applications to process at a later time.
When you subscribe an Amazon SQS queue to an Amazon SNS topic, you can publish a message to the topic and Amazon SNS sends an Amazon SQS message to the subscribed queue. The Amazon SQS message contains the subject and message that were published to the topic along with metadata about the message in a JSON document. The Amazon SQS message will look similar to the following JSON document.
```
{
"Type" : "Notification",
"MessageId" : "63a3f6b6-d533-4a47-aef9-fcf5cf758c76",
"TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
"Subject" : "Testing publish to subscribed queues",
"Message" : "Hello world!",
"Timestamp" : "2012-03-29T05:12:16.901Z",
"SignatureVersion" : "1",
"Signature" : "EXAMPLEnTrFPa3...",
"SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",
"UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:123456789012:MyTopic:c7fe3a54-ab0e-4ec2-88e0-db410a0f2bee"
}
```
# Subscribing an Amazon SQS queue to an Amazon SNS topic
To enable an Amazon SNS topic to send messages to an Amazon SQS queue, choose one of the following:
+ Use the [Amazon SQS console](https://console.aws.amazon.com/sqs/), which simplifies the process. For more information, see [Subscribing an Amazon SQS queue to an Amazon SNS topic](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-subscribe-queue-sns-topic.html) in the *Amazon Simple Queue Service Developer Guide*.
+ Use the following steps:
1. [Get the Amazon Resource Name (ARN) of the queue you want to send messages to and the topic to which you want to subscribe the queue.](#SendMessageToSQS.arn)
1. [Give `sqs:SendMessage` permission to the Amazon SNS topic so that it can send messages to the queue.](#SendMessageToSQS.sqs.permissions)
1. [Subscribe the queue to the Amazon SNS topic.](#SendMessageToSQS.subscribe)
1. [Give IAM users or AWS accounts the appropriate permissions to publish to the Amazon SNS topic and read messages from the Amazon SQS queue.](#SendMessageToSQS.iam.permissions)
1. [Test it out by publishing a message to the topic and reading the message from the queue.](#SendMessageToSQS.test)
To learn about how to set up a topic to send messages to a queue that is in a different AWS-account, see [Sending Amazon SNS messages to an Amazon SQS queue in a different account](sns-send-message-to-sqs-cross-account.md).
To see an CloudFormation template that creates a topic that sends messages to two queues, see [Automate Amazon SNS to Amazon SQS messaging with AWS CloudFormation](SendMessageToSQS.cloudformation.md).
## Step 1: Get the ARN of the queue and topic
When subscribing a queue to your topic, you'll need a copy of the ARN for the queue. Similarly, when giving permission for the topic to send messages to the queue, you'll need a copy of the ARN for the topic.
To get the queue ARN, you can use the Amazon SQS console or the [GetQueueAttributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Query_QueryGetQueueAttributes.html) API action.
**To get the queue ARN from the Amazon SQS console**
1. Sign in to the AWS Management Console and open the Amazon SQS console at [https://console.aws.amazon.com/sqs/](https://console.aws.amazon.com/sqs/).
1. Select the box for the queue whose ARN you want to get.
1. From the **Details** section, copy the ARN value so that you can use it to subscribe to the Amazon SNS topic.
To get the topic ARN, you can use the Amazon SNS console, the `[sns-get-topic-attributes](https://docs.aws.amazon.com/cli/latest/reference/sns/get-topic-attributes.html)` command, or the `[GetQueueAttributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Query_QueryGetQueueAttributes.html)` API action.
**To get the topic ARN from the Amazon SNS console**
1. Sign in to the [Amazon SNS console](https://console.aws.amazon.com/sns/home).
1. On the navigation panel, choose the topic whose ARN you want to get.
1. From the **Details** section, copy the **ARN** value so that you can use it to give permission for the Amazon SNS topic to send messages to the queue.
## Step 2: Give permission to the Amazon SNS topic to send messages to the Amazon SQS queue
For an Amazon SNS topic to be able to send messages to a queue, you must set a policy on the queue that allows the Amazon SNS topic to perform the `sqs:SendMessage` action.
Before you subscribe a queue to a topic, you need a topic and a queue. If you haven't already created a topic or queue, create them now. For more information, see [Creating a topic](sns-create-topic.md), and see [Create a queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/step-create-queue.html) in the *Amazon Simple Queue Service Developer Guide*.
To set a policy on a queue, you can use the Amazon SQS console or the [SetQueueAttributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Query_QuerySetQueueAttributes.html) API action. Before you start, make sure you have the ARN for the topic that you want to allow to send messages to the queue. If you are subscribing a queue to multiple topics, your policy must contain one `Statement` element for each topic.
**To set a SendMessage policy on a queue using the Amazon SQS console**
1. Sign in to the AWS Management Console and open the Amazon SQS console at [https://console.aws.amazon.com/sqs/](https://console.aws.amazon.com/sqs/).
1. Select the box for the queue whose policy you want to set, choose the **Access policy** tab, and then choose **Edit**.
1. In the **Access policy** section, define who can access your queue.
+ Add a condition that allows the action for the topic.
+ Set `Principal` to be the Amazon SNS service, as shown in the example below.
+ Use the [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn) or [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount) global condition keys to protect against the [confused deputy](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html) scenario. To use these condition keys, set the value to the ARN of your topic. If your queue is subscribed to multiple topics, you can use `aws:SourceAccount` instead.
For example, the following policy allows MyTopic to send messages to MyQueue.
```
{
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:sns:us-east-2:123456789012:MyTopic"
}
}
}
]
}
```
## Step 3: Subscribe the queue to the Amazon SNS topic
To send messages to a queue through a topic, you must subscribe the queue to the Amazon SNS topic. You specify the queue by its ARN. To subscribe to a topic, you can use the Amazon SNS console, the `[sns-subscribe](https://docs.aws.amazon.com/cli/latest/reference/sns/subscribe.html)` CLI command, or the `[Subscribe](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html)` API action. Before you start, make sure you have the ARN for the queue that you want to subscribe.
1. Sign in to the [Amazon SNS console](https://console.aws.amazon.com/sns/home).
1. On the navigation panel, choose **Topics**.
1. On the **Topics** page, choose a topic.
1. On the ***MyTopic*** page, in the **Subscriptions** page, choose **Create subscription**.
1. On the **Create subscription** page, in the **Details** section, do the following:
1. Verify the **Topic ARN**.
1. For **Protocol**, choose **Amazon SQS**.
1. For **Endpoint**, enter the ARN of an Amazon SQS queue.
1. Choose **Create Subscription**.
When the subscription is confirmed, your new subscription's **Subscription ID** displays its subscription ID. If the owner of the queue creates the subscription, the subscription is automatically confirmed and the subscription should be active almost immediately.
Usually, you'll be subscribing your own queue to your own topic in your own account. However, you can also subscribe a queue from a different account to your topic. If the user who creates the subscription is not the owner of the queue (for example, if a user from account A subscribes a queue from account B to a topic in account A), the subscription must be confirmed. For more information about subscribing a queue from a different account and confirming the subscription, see [Sending Amazon SNS messages to an Amazon SQS queue in a different account](sns-send-message-to-sqs-cross-account.md).
## Step 4: Give users permissions to the appropriate topic and queue actions
You should use AWS Identity and Access Management (IAM) to allow only appropriate users to publish to the Amazon SNS topic and to read/delete messages from the Amazon SQS queue. For more information about controlling actions on topics and queues for IAM users, see [Using identity-based policies with Amazon SNS](sns-using-identity-based-policies.md), and [Identity and access management in Amazon SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/UsingIAM.html) in the Amazon Simple Queue Service Developer Guide.
There are two ways to control access to a topic or queue:
+ [Add a policy to an IAM user or group](#SendMessageToSQS.iam.permissions.user). The simplest way to give users permissions to topics or queues is to create a group and add the appropriate policy to the group and then add users to that group. It's much easier to add and remove users from a group than to keep track of which policies you set on individual users.
+ [Add a policy to topic or queue](#SendMessageToSQS.iam.permissions.resource). If you want to give permissions to a topic or queue to another AWS account, the only way you can do that is by adding a policy that has as its principal the AWS account you want to give permissions to.
You should use the first method for most cases (apply policies to groups and manage permissions for users by adding or removing the appropriate users to the groups). If you need to give permissions to a user in another account, you should use the second method.
### Adding a policy to an IAM user or group
If you added the following policy to an IAM user or group, you would give that user or members of that group permission to perform the `sns:Publish` action on the topic MyTopic.
```
{
"Statement": [
{
"Effect": "Allow",
"Action": "sns:Publish",
"Resource": "arn:aws:sns:us-east-2:123456789012:MyTopic"
}
]
}
```
If you added the following policy to an IAM user or group, you would give that user or members of that group permission to perform the `sqs:ReceiveMessage` and `sqs:DeleteMessage` actions on the queues MyQueue1 and MyQueue2.
```
{
"Statement": [
{
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:DeleteMessage"
],
"Resource": [
"arn:aws:sqs:us-east-2:123456789012:MyQueue1",
"arn:aws:sqs:us-east-2:123456789012:MyQueue2"
]
}
]
}
```
### Adding a policy to a topic or queue
The following example policies show how to give another account permissions to a topic and queue.
**Note**
When you give another AWS account access to a resource in your account, you are also giving IAM users who have admin-level access (wildcard access) permissions to that resource. All other IAM users in the other account are automatically denied access to your resource. If you want to give specific IAM users in that AWS account access to your resource, the account or an IAM user with admin-level access must delegate permissions for the resource to those IAM users. For more information about cross-account delegation, see [Enabling Cross-Account Access](https://docs.aws.amazon.com/IAM/latest/UserGuide/Delegation.html) in the *Using IAM Guide*.
If you added the following policy to a topic MyTopic in account 123456789012, you would give account 111122223333 permission to perform the `sns:Publish` action on that topic.
```
{
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "111122223333"
},
"Action": "sns:Publish",
"Resource": "arn:aws:sns:us-east-2:123456789012:MyTopic"
}
]
}
```
If you added the following policy to a queue MyQueue in account 123456789012, you would give account 111122223333 permission to perform the `sqs:ReceiveMessage` and `sqs:DeleteMessage` actions on that queue.
```
{
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "111122223333"
},
"Action": [
"sqs:DeleteMessage",
"sqs:ReceiveMessage"
],
"Resource": [
"arn:aws:sqs:us-east-2:123456789012:MyQueue"
]
}
]
}
```
## Step 5: Test the topic's queue subscriptions
You can test a topic's queue subscriptions by publishing to the topic and viewing the message that the topic sends to the queue.
**To publish to a topic using the Amazon SNS console**
1. Using the credentials of the AWS account or IAM user with permission to publish to the topic, sign in to the AWS Management Console and open the Amazon SNS console at [https://console.aws.amazon.com/sns/](https://console.aws.amazon.com/sns/home).
1. On the navigation panel, choose the topic and choose **Publish to Topic**.
1. In the **Subject** box, enter a subject (for example, **Testing publish to queue**) in the **Message** box, enter some text (for example, **Hello world\$1**), and choose **Publish Message**. The following message appears: Your message has been successfully published.
**To view the message from the topic using the Amazon SQS console**
1. Using the credentials of the AWS account or IAM user with permission to view messages in the queue, sign in to the AWS Management Console and open the Amazon SQS console at [https://console.aws.amazon.com/sqs/](https://console.aws.amazon.com/sqs/).
1. Choose a **queue** that is subscribed to the topic.
1. Choose **Send and receive messages**, and then choose **Poll for messages**. A message with a type of **Notification** appears.
1. In the **Body** column, choose **More Details**. The **Message Details** box contains a JSON document that contains the subject and message that you published to the topic. The message looks similar to the following JSON document.
```
{
"Type" : "Notification",
"MessageId" : "63a3f6b6-d533-4a47-aef9-fcf5cf758c76",
"TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
"Subject" : "Testing publish to subscribed queues",
"Message" : "Hello world!",
"Timestamp" : "2012-03-29T05:12:16.901Z",
"SignatureVersion" : "1",
"Signature" : "EXAMPLEnTrFPa3...",
"SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",
"UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:123456789012:MyTopic:c7fe3a54-ab0e-4ec2-88e0-db410a0f2bee"
}
```
1. Choose **Close**. You have successfully published to a topic that sends notification messages to a queue.
# Automate Amazon SNS to Amazon SQS messaging with AWS CloudFormation
CloudFormation enables you to use a template file to create and configure a collection of AWS resources together as a single unit. This section has an example template that makes it easy to deploy topics that publish to queues. The templates take care of the setup steps for you by creating two queues, creating a topic with subscriptions to the queues, adding a policy to the queues so that the topic can send messages to the queues, and creating IAM users and groups to control access to those resources.
For more information about deploying AWS resources using an CloudFormation template, see [Get Started](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/GettingStarted.Walkthrough.html) in the *CloudFormation User Guide*.
## Using an CloudFormation template to set up topics and queues within an AWS account
The example template creates an Amazon SNS topic that can send messages to two Amazon SQS queues with appropriate permissions for members of one IAM group to publish to the topic and another to read messages from the queues. The template also creates IAM users that are added to each group.
You copy the template contents into a file. You can also download the template from the [AWS CloudFormation Templates page](http://aws.amazon.com/cloudformation/aws-cloudformation-templates/). On the templates page, choose **Browse sample templates by AWS service**and then choose **Amazon Simple Queue Service**.
MySNSTopic is set up to publish to two subscribed endpoints, which are two Amazon SQS queues (MyQueue1 and MyQueue2). MyPublishTopicGroup is an IAM group whose members have permission to publish to MySNSTopic using the [Publish](https://docs.aws.amazon.com/sns/latest/api/API_Publish.html) API action or [sns-publish](https://docs.aws.amazon.com/cli/latest/reference/sns/publish.html) command. The template creates the IAM users MyPublishUser and MyQueueUser and gives them login profiles and access keys. The user who creates a stack with this template specifies the passwords for the login profiles as input parameters. The template creates access keys for the two IAM users with MyPublishUserKey and MyQueueUserKey. AddUserToMyPublishTopicGroup adds MyPublishUser to the MyPublishTopicGroup so that the user will have the permissions assigned to the group.
MyRDMessageQueueGroup is an IAM group whose members have permission to read and delete messages from the two Amazon SQS queues using the [ReceiveMessage](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Query_QueryReceiveMessage.html) and [DeleteMessage](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Query_QueryDeleteMessage.html) API actions. AddUserToMyQueueGroup adds MyQueueUser to the MyRDMessageQueueGroup so that the user will have the permissions assigned to the group. MyQueuePolicy assigns permission for MySNSTopic to publish its notifications to the two queues.
The following listing shows the CloudFormation template contents.
```
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "AWS CloudFormation Sample Template SNSToSQS: This Template creates an SNS topic that can send messages to
two SQS queues with appropriate permissions for one IAM user to publish to the topic and another to read messages from the queues.
MySNSTopic is set up to publish to two subscribed endpoints, which are two SQS queues (MyQueue1 and MyQueue2). MyPublishUser is an IAM user
that can publish to MySNSTopic using the Publish API. MyTopicPolicy assigns that permission to MyPublishUser. MyQueueUser is an IAM user
that can read messages from the two SQS queues. MyQueuePolicy assigns those permissions to MyQueueUser. It also assigns permission for
MySNSTopic to publish its notifications to the two queues. The template creates access keys for the two IAM users with MyPublishUserKey
and MyQueueUserKey. ***Warning*** you will be billed for the AWS resources used if you create a stack from this template.",
"Parameters": {
"MyPublishUserPassword": {
"NoEcho": "true",
"Type": "String",
"Description": "Password for the IAM user MyPublishUser",
"MinLength": "1",
"MaxLength": "41",
"AllowedPattern": "[a-zA-Z0-9]*",
"ConstraintDescription": "must contain only alphanumeric characters."
},
"MyQueueUserPassword": {
"NoEcho": "true",
"Type": "String",
"Description": "Password for the IAM user MyQueueUser",
"MinLength": "1",
"MaxLength": "41",
"AllowedPattern": "[a-zA-Z0-9]*",
"ConstraintDescription": "must contain only alphanumeric characters."
}
},
"Resources": {
"MySNSTopic": {
"Type": "AWS::SNS::Topic",
"Properties": {
"Subscription": [{
"Endpoint": {
"Fn::GetAtt": ["MyQueue1", "Arn"]
},
"Protocol": "sqs"
},
{
"Endpoint": {
"Fn::GetAtt": ["MyQueue2", "Arn"]
},
"Protocol": "sqs"
}
]
}
},
"MyQueue1": {
"Type": "AWS::SQS::Queue"
},
"MyQueue2": {
"Type": "AWS::SQS::Queue"
},
"MyPublishUser": {
"Type": "AWS::IAM::User",
"Properties": {
"LoginProfile": {
"Password": {
"Ref": "MyPublishUserPassword"
}
}
}
},
"MyPublishUserKey": {
"Type": "AWS::IAM::AccessKey",
"Properties": {
"UserName": {
"Ref": "MyPublishUser"
}
}
},
"MyPublishTopicGroup": {
"Type": "AWS::IAM::Group",
"Properties": {
"Policies": [{
"PolicyName": "MyTopicGroupPolicy",
"PolicyDocument": {
"Statement": [{
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": {
"Ref": "MySNSTopic"
}
}]
}
}]
}
},
"AddUserToMyPublishTopicGroup": {
"Type": "AWS::IAM::UserToGroupAddition",
"Properties": {
"GroupName": {
"Ref": "MyPublishTopicGroup"
},
"Users": [{
"Ref": "MyPublishUser"
}]
}
},
"MyQueueUser": {
"Type": "AWS::IAM::User",
"Properties": {
"LoginProfile": {
"Password": {
"Ref": "MyQueueUserPassword"
}
}
}
},
"MyQueueUserKey": {
"Type": "AWS::IAM::AccessKey",
"Properties": {
"UserName": {
"Ref": "MyQueueUser"
}
}
},
"MyRDMessageQueueGroup": {
"Type": "AWS::IAM::Group",
"Properties": {
"Policies": [{
"PolicyName": "MyQueueGroupPolicy",
"PolicyDocument": {
"Statement": [{
"Effect": "Allow",
"Action": [
"sqs:DeleteMessage",
"sqs:ReceiveMessage"
],
"Resource": [{
"Fn::GetAtt": ["MyQueue1", "Arn"]
},
{
"Fn::GetAtt": ["MyQueue2", "Arn"]
}
]
}]
}
}]
}
},
"AddUserToMyQueueGroup": {
"Type": "AWS::IAM::UserToGroupAddition",
"Properties": {
"GroupName": {
"Ref": "MyRDMessageQueueGroup"
},
"Users": [{
"Ref": "MyQueueUser"
}]
}
},
"MyQueuePolicy": {
"Type": "AWS::SQS::QueuePolicy",
"Properties": {
"PolicyDocument": {
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": ["sqs:SendMessage"],
"Resource": "*",
"Condition": {
"ArnEquals": {
"aws:SourceArn": {
"Ref": "MySNSTopic"
}
}
}
}]
},
"Queues": [{
"Ref": "MyQueue1"
}, {
"Ref": "MyQueue2"
}]
}
}
},
"Outputs": {
"MySNSTopicTopicARN": {
"Value": {
"Ref": "MySNSTopic"
}
},
"MyQueue1Info": {
"Value": {
"Fn::Join": [
" ",
[
"ARN:",
{
"Fn::GetAtt": ["MyQueue1", "Arn"]
},
"URL:",
{
"Ref": "MyQueue1"
}
]
]
}
},
"MyQueue2Info": {
"Value": {
"Fn::Join": [
" ",
[
"ARN:",
{
"Fn::GetAtt": ["MyQueue2", "Arn"]
},
"URL:",
{
"Ref": "MyQueue2"
}
]
]
}
},
"MyPublishUserInfo": {
"Value": {
"Fn::Join": [
" ",
[
"ARN:",
{
"Fn::GetAtt": ["MyPublishUser", "Arn"]
},
"Access Key:",
{
"Ref": "MyPublishUserKey"
},
"Secret Key:",
{
"Fn::GetAtt": ["MyPublishUserKey", "SecretAccessKey"]
}
]
]
}
},
"MyQueueUserInfo": {
"Value": {
"Fn::Join": [
" ",
[
"ARN:",
{
"Fn::GetAtt": ["MyQueueUser", "Arn"]
},
"Access Key:",
{
"Ref": "MyQueueUserKey"
},
"Secret Key:",
{
"Fn::GetAtt": ["MyQueueUserKey", "SecretAccessKey"]
}
]
]
}
}
}
}
```
# Fanout Amazon SNS notifications to HTTPS endpoints
You can use Amazon SNS to send notification messages to one or more HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic, you can publish a notification to the topic and Amazon SNS sends an HTTP POST request delivering the contents of the notification to the subscribed endpoint. When you subscribe the endpoint, you choose whether Amazon SNS uses HTTP or HTTPS to send the POST request to the endpoint. If you use HTTPS, then you can take advantage of the support in Amazon SNS for the following:
+ **Server Name Indication (SNI)**—This allows Amazon SNS to support HTTPS endpoints that require SNI, such as a server requiring multiple certificates for hosting multiple domains. For more information about SNI, see [Server Name Indication](http://en.wikipedia.org/wiki/Server_Name_Indication).
+ **Basic and Digest Access Authentication**—This allows you to specify a username and password in the HTTPS URL for the HTTP POST request, such as `https://user:password@domain.com` or `https://user@domain.com` The username and password are encrypted over the SSL connection established when using HTTPS. Only the domain name is sent in plaintext. For more information about Basic and Digest Access Authentication, see [RFC-2617](http://www.rfc-editor.org/info/rfc2617).
**Important**
Amazon SNS does not currently support private HTTP(S) endpoints.
HTTPS URLs are only retrievable from the Amazon SNS `GetSubscriptionAttributes` API action, for principals to which you have granted API access.
**Note**
The client service must be able to support the `HTTP/1.1 401 Unauthorized` header response
The request contains the subject and message that were published to the topic along with metadata about the notification in a JSON document. The request will look similar to the following HTTP POST request. For details about the HTTP header and the JSON format of the request body, see [HTTP/HTTPS headers](http-header.md) and [HTTP/HTTPS notification JSON format](http-notification-json.md).
**Note**
Amazon SNS considers all 5XX errors and 429 (too many requests sent) errors as retryable. These errors are subject to the delivery policy. All other errors are considered as permanent failures and retries will not be attempted.
```
POST / HTTP/1.1
x-amz-sns-message-type: Notification
x-amz-sns-message-id: da41e39f-ea4d-435a-b922-c6aae3915ebe
x-amz-sns-topic-arn: arn:aws:sns:us-west-2:123456789012:MyTopic
x-amz-sns-subscription-arn: arn:aws:sns:us-west-2:123456789012:MyTopic:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55
Content-Length: 761
Content-Type: text/plain; charset=UTF-8
Host: ec2-50-17-44-49.compute-1.amazonaws.com
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
{
"Type" : "Notification",
"MessageId" : "da41e39f-ea4d-435a-b922-c6aae3915ebe",
"TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
"Subject" : "test",
"Message" : "test message",
"Timestamp" : "2012-04-25T21:49:25.719Z",
"SignatureVersion" : "1",
"Signature" : "EXAMPLElDMXvB8r9R83tGoNn0ecwd5UjllzsvSvbItzfaMpN2nk5HVSw7XnOn/49IkxDKz8YrlH2qJXj2iZB0Zo2O71c4qQk1fMUDi3LGpij7RCW7AW9vYYsSqIKRnFS94ilu7NFhUzLiieYr4BKHpdTmdD6c0esKEYBpabxDSc=",
"SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",
"UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:123456789012:MyTopic:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55"
}
```
# Subscribing an HTTPS endpoint to an Amazon SNS topic
This topic explains how to subscribe HTTP/S endpoints to Amazon SNS topics.
**Topics**
+ [Step 1: Make sure your endpoint is ready to process Amazon SNS messages](SendMessageToHttp.prepare.md)
+ [Step 2: Subscribe the HTTP/HTTPS endpoint to the Amazon SNS topic](SendMessageToHttp.subscribe.md)
+ [Step 3: Confirm your Amazon SNS subscription](SendMessageToHttp.confirm.md)
+ [Step 4: Optional - Set the delivery policy for the Amazon SNS subscription](SendMessageToHttp.retry.md)
+ [Step 5: Optional - Give users permissions to publish to the Amazon SNS topic](SendMessageToHttp.iam.permissions.md)
+ [Step 6: Send Amazon SNS messages to the HTTP/HTTPS endpoint](SendMessageToHttp.publish.md)
# Step 1: Make sure your endpoint is ready to process Amazon SNS messages
Before you subscribe your HTTP or HTTPS endpoint to a topic, you must make sure that the HTTP or HTTPS endpoint has the capability to handle the HTTP POST requests that Amazon SNS uses to send the subscription confirmation and notification messages. Usually, this means creating and deploying a web application (for example, a Java servlet if your endpoint host is running Linux with Apache and Tomcat) that processes the HTTP requests from Amazon SNS. When you subscribe an HTTP endpoint, Amazon SNS sends it a subscription confirmation request. Your endpoint must be prepared to receive and process this request when you create the subscription because Amazon SNS sends this request at that time. Amazon SNS will not send notifications to the endpoint until you confirm the subscription. Once you confirm the subscription, Amazon SNS will send notifications to the endpoint when a publish action is performed on the subscribed topic.
**To set up your endpoint to process subscription confirmation and notification messages**
1. Your code should read the HTTP headers of the HTTP POST requests that Amazon SNS sends to your endpoint. Your code should look for the header field `x-amz-sns-message-type`, which tells you the type of message that Amazon SNS has sent to you. By looking at the header, you can determine the message type without having to parse the body of the HTTP request. There are two types that you need to handle: `SubscriptionConfirmation` and `Notification`. The `UnsubscribeConfirmation` message is used only when the subscription is deleted from the topic.
For details about the HTTP header, see [HTTP/HTTPS headers](http-header.md). The following HTTP POST request is an example of a subscription confirmation message.
```
POST / HTTP/1.1
x-amz-sns-message-type: SubscriptionConfirmation
x-amz-sns-message-id: 165545c9-2a5c-472c-8df2-7ff2be2b3b1b
x-amz-sns-topic-arn: arn:aws:sns:us-west-2:123456789012:MyTopic
Content-Length: 1336
Content-Type: text/plain; charset=UTF-8
Host: example.com
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
{
"Type" : "SubscriptionConfirmation",
"MessageId" : "165545c9-2a5c-472c-8df2-7ff2be2b3b1b",
"Token" : "2336412f37f...",
"TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
"Message" : "You have chosen to subscribe to the topic arn:aws:sns:us-west-2:123456789012:MyTopic.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
"SubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-west-2:123456789012:MyTopic&Token=2336412f37...",
"Timestamp" : "2012-04-26T20:45:04.751Z",
"SignatureVersion" : "1",
"Signature" : "EXAMPLEpH+...",
"SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem"
}
```
1. Your code should parse the JSON document in the body of the HTTP POST request and content-type text/plain to read the name-value pairs that make up the Amazon SNS message. Use a JSON parser that handles converting the escaped representation of control characters back to their ASCII character values (for example, converting \$1n to a newline character). You can use an existing JSON parser such as the [Jackson JSON Processor](https://github.com/FasterXML/jackson) or write your own. In order to send the text in the subject and message fields as valid JSON, Amazon SNS must convert some control characters to escaped representations that can be included in the JSON document. When you receive the JSON document in the body of the POST request sent to your endpoint, you must convert the escaped characters back to their original character values if you want an exact representation of the original subject and messages published to the topic. This is critical if you want to verify the signature of a notification because the signature uses the message and subject in their original forms as part of the string to sign.
1. Your code should verify the authenticity of a notification, subscription confirmation, or unsubscribe confirmation message sent by Amazon SNS. Using information contained in the Amazon SNS message, your endpoint can recreate the signature so that you can verify the contents of the message by matching your signature with the signature that Amazon SNS sent with the message. For more information about verifying the signature of a message, see [Verifying the signatures of Amazon SNS messages](sns-verify-signature-of-message.md).
1. Based on the type specified by the header field `x-amz-sns-message-type`, your code should read the JSON document contained in the body of the HTTP request and process the message. Here are the guidelines for handling the two primary types of messages:
**SubscriptionConfirmation**
Read the value for `SubscribeURL` and visit that URL. To confirm the subscription and start receiving notifications at the endpoint, you must visit the `SubscribeURL`URL (for example, by sending an HTTP GET request to the URL). See the example HTTP request in the previous step to see what the `SubscribeURL` looks like. For more information about the format of the `SubscriptionConfirmation` message, see [HTTP/HTTPS subscription confirmation JSON format](http-subscription-confirmation-json.md). When you visit the URL, you will get back a response that looks like the following XML document. The document returns the subscription ARN for the endpoint within the `ConfirmSubscriptionResult` element.
```
arn:aws:sns:us-west-2:123456789012:MyTopic:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55
075ecce8-8dac-11e1-bf80-f781d96e9307
```
As an alternative to visiting the `SubscribeURL`, you can confirm the subscription using the [ConfirmSubscription](https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html) action with the `Token` set to its corresponding value in the `SubscriptionConfirmation` message. If you want to allow only the topic owner and subscription owner to be able to unsubscribe the endpoint, you call the `ConfirmSubscription` action with an AWS signature.
**Notification**
Read the values for `Subject` and `Message` to get the notification information that was published to the topic.
For details about the format of the `Notification` message, see [HTTP/HTTPS headers](http-header.md). The following HTTP POST request is an example of a notification message sent to the endpoint example.com.
```
POST / HTTP/1.1
x-amz-sns-message-type: Notification
x-amz-sns-message-id: 22b80b92-fdea-4c2c-8f9d-bdfb0c7bf324
x-amz-sns-topic-arn: arn:aws:sns:us-west-2:123456789012:MyTopic
x-amz-sns-subscription-arn: arn:aws:sns:us-west-2:123456789012:MyTopic:c9135db0-26c4-47ec-8998-413945fb5a96
Content-Length: 773
Content-Type: text/plain; charset=UTF-8
Host: example.com
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
{
"Type" : "Notification",
"MessageId" : "22b80b92-fdea-4c2c-8f9d-bdfb0c7bf324",
"TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
"Subject" : "My First Message",
"Message" : "Hello world!",
"Timestamp" : "2012-05-02T00:54:06.655Z",
"SignatureVersion" : "1",
"Signature" : "EXAMPLEw6JRN...",
"SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",
"UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:123456789012:MyTopic:c9135db0-26c4-47ec-8998-413945fb5a96"
}
```
1. Make sure that your endpoint responds to the HTTP POST message from Amazon SNS with the appropriate status code. The connection will time out in approximately 15 seconds. If your endpoint does not respond before the connection times out, or if your endpoint returns a status code outside the range of 200–4*xx*, Amazon SNS will consider the delivery of the message as a failed attempt.
1. Make sure that your code can handle message delivery retries from Amazon SNS. If Amazon SNS doesn't receive a successful response from your endpoint, it attempts to deliver the message again. This applies to all messages, including the subscription confirmation message. By default, if the initial delivery of the message fails, Amazon SNS attempts up to three retries with a delay between failed attempts set at 20 seconds.
**Note**
The message request times out after approximately 15 seconds. This means that, if the message delivery failure is caused by a timeout, Amazon SNS retries for approximately 35 seconds after the previous delivery attempt. You can set a different delivery policy for the endpoint.
Amazon SNS uses the `x-amz-sns-message-id` header field to uniquely identify each message published to an Amazon SNS topic. By comparing the IDs of the messages you have processed with incoming messages, you can determine whether the message is a retry attempt.
1. If you are subscribing an HTTPS endpoint, make sure that your endpoint has a server certificate from a trusted Certificate Authority (CA). Amazon SNS will only send messages to HTTPS endpoints that have a server certificate signed by a CA trusted by Amazon SNS.
1. Deploy the code that you have created to receive Amazon SNS messages. When you subscribe the endpoint, the endpoint must be ready to receive at least the subscription confirmation message.
# Step 2: Subscribe the HTTP/HTTPS endpoint to the Amazon SNS topic
To send messages to an HTTP or HTTPS endpoint through a topic, you must subscribe the endpoint to the Amazon SNS topic. You specify the endpoint using its URL. To subscribe to a topic, you can use the Amazon SNS console, the [sns-subscribe](https://docs.aws.amazon.com/cli/latest/reference/sns/subscribe.html) command, or the [Subscribe](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html) API action. Before you start, make sure you have the URL for the endpoint that you want to subscribe and that your endpoint is prepared to receive the confirmation and notification messages as described in Step 1.
**To subscribe an HTTP or HTTPS endpoint to a topic using the Amazon SNS console**
1. Sign in to the [Amazon SNS console](https://console.aws.amazon.com/sns/home).
1. On the navigation panel, choose **Subscriptions**.
1. Choose the **Create subscription**.
1. In the **Protocol** drop-down list, select **HTTP** or **HTTPS**.
1. In the **Endpoint** box, paste in the URL for the endpoint that you want the topic to send messages to and then choose **Create subscription**.
1. The confirmation message is displayed. Choose **Close**.
Your new subscription's **Subscription ID** displays PendingConfirmation. When you confirm the subscription, **Subscription ID** will display the subscription ID.
# Step 3: Confirm your Amazon SNS subscription
To confirm your Amazon SNS subscription, follow these steps to ensure your endpoint can successfully receive messages. This process involves setting up your endpoint to handle incoming confirmation messages, retrieving the confirmation URL, and confirming the subscription. You can confirm the subscription either automatically or manually, depending on your setup.
1. After subscribing to an Amazon SNS topic, Amazon SNS sends a confirmation message to your endpoint. This message contains a `SubscribeURL` that you must use to confirm the subscription.
1. Your endpoint must be set up to listen for incoming messages from Amazon SNS. When the confirmation message arrives, extract the **`SubscribeURL`** from the message.
1. Once you have the `SubscribeURL`, you can confirm the subscription in one of two ways:
+ **Automatic confirmation** – Your endpoint can automatically confirm the subscription by sending an **HTTP GET request** to the `SubscribeURL`.
This method does not require manual intervention.
+ **Manual confirmation** – If automatic confirmation is not set up, **copy** the **`SubscribeURL`** from the confirmation message and **paste** it into your browser’s address bar.
This will confirm the subscription manually.
1. You can also verify the **subscription status** using the Amazon SNS console:
1. Sign in to the [Amazon SNS console](https://console.aws.amazon.com/sns/home).
1. In the navigation pane, choose **Subscriptions**.
1. Find your **subscription** in the list.
+ If confirmed, the `SubscriptionArn` will be displayed.
+ If still unconfirmed, it will show as `PendingConfirmation`.
# Step 4: Optional - Set the delivery policy for the Amazon SNS subscription
By default, if the initial delivery of the message fails, Amazon SNS attempts up to three retries with a delay between failed attempts set at 20 seconds. As discussed in [Step 1](SendMessageToHttp.prepare.md), your endpoint should have code that can handle retried messages. By setting the delivery policy on a topic or subscription, you can control the frequency and interval that Amazon SNS will retry failed messages. You can also specify the content type for your HTTP/S notifications in `DeliveryPolicy`. For more information, see [Creating an HTTP/S delivery policy](sns-message-delivery-retries.md#creating-delivery-policy).
# Step 5: Optional - Give users permissions to publish to the Amazon SNS topic
By default, the topic owner has permissions to publish the topic. To enable other users or applications to publish to the topic, you should use AWS Identity and Access Management (IAM) to give publish permission to the topic. For more information about giving permissions for Amazon SNS actions to IAM users, see [Using identity-based policies with Amazon SNS](sns-using-identity-based-policies.md).
There are two ways to control access to a topic:
+ Add a policy to an IAM user or group. The simplest way to give users permissions to topics is to create a group and add the appropriate policy to the group and then add users to that group. It's much easier to add and remove users from a group than to keep track of which policies you set on individual users.
+ Add a policy to the topic. If you want to give permissions to a topic to another AWS account, the only way you can do that is by adding a policy that has as its principal the AWS account you want to give permissions to.
You should use the first method for most cases (apply policies to groups and manage permissions for users by adding or removing the appropriate users to the groups). If you need to give permissions to a user in another account, use the second method.
If you added the following policy to an IAM user or group, you would give that user or members of that group permission to perform the `sns:Publish` action on the topic MyTopic.
```
{
"Statement":[{
"Sid":"AllowPublishToMyTopic",
"Effect":"Allow",
"Action":"sns:Publish",
"Resource":"arn:aws:sns:us-east-2:123456789012:MyTopic"
}]
}
```
The following example policy shows how to give another account permissions to a topic.
**Note**
When you give another AWS account access to a resource in your account, you are also giving IAM users who have admin-level access (wildcard access) permissions to that resource. All other IAM users in the other account are automatically denied access to your resource. If you want to give specific IAM users in that AWS account access to your resource, the account or an IAM user with admin-level access must delegate permissions for the resource to those IAM users. For more information about cross-account delegation, see [Enabling Cross-Account Access](https://docs.aws.amazon.com/IAM/latest/UserGuide/Delegation.html) in the *Using IAM Guide*.
If you added the following policy to a topic MyTopic in account 123456789012, you would give account 111122223333 permission to perform the `sns:Publish` action on that topic.
```
{
"Statement":[{
"Sid":"Allow-publish-to-topic",
"Effect":"Allow",
"Principal":{
"AWS":"111122223333"
},
"Action":"sns:Publish",
"Resource":"arn:aws:sns:us-east-2:123456789012:MyTopic"
}]
}
```
# Step 6: Send Amazon SNS messages to the HTTP/HTTPS endpoint
You can send a message to a topic's subscriptions by publishing to the topic. To publish to a topic, you can use the Amazon SNS console, the `[sns-publish](https://docs.aws.amazon.com/cli/latest/reference/sns/publish.html)` CLI command, or the `[Publish](https://docs.aws.amazon.com/sns/latest/api/API_Publish.html)` API.
If you followed [Step 1](SendMessageToHttp.prepare.md), the code that you deployed at your endpoint should process the notification.
**To publish to a topic using the Amazon SNS console**
1. Using the credentials of the AWS account or IAM user with permission to publish to the topic, sign in to the AWS Management Console and open the Amazon SNS console at [https://console.aws.amazon.com/sns/](https://console.aws.amazon.com/sns/home).
1. On the navigation panel, choose **Topics** and then choose a topic.
1. Choose the **Publish message** button.
1. In the **Subject** box, enter a subject (for example, **Testing publish to my endpoint**).
1. In the **Message** box, enter some text (for example, **Hello world\$1**), and choose **Publish message**.
The following message appears: Your message has been successfully published.
# Verifying the signatures of Amazon SNS messages
Amazon SNS uses message signatures to confirm the authenticity of messages sent to your HTTP endpoint. To ensure message integrity and prevent spoofing, you **must** verify the signature before processing any Amazon SNS messages.
**When should you verify Amazon SNS signatures?**
You should verify Amazon SNS message signatures in the following scenarios:
+ When Amazon SNS sends a notification message to your HTTP(S) endpoint.
+ When Amazon SNS sends a confirmation message to your endpoint after a [https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html) or [https://docs.aws.amazon.com/sns/latest/api/API_Unsubscribe.html](https://docs.aws.amazon.com/sns/latest/api/API_Unsubscribe.html) API call.
**Amazon SNS supports two signature versions:**
+ SignatureVersion1 – Uses an SHA1 hash of the message.
+ SignatureVersion2 – Uses an SHA256 hash of the message. This provides stronger security and is the recommended option.
**To correctly verify SNS message signatures, follow these best practices:**
+ Always retrieve the signing certificate using HTTPS to prevent unauthorized interception attacks.
+ Check that the certificate is issued by Amazon SNS.
+ Confirm that the certificate’s chain of trust is valid.
+ The certificate should come from an SNS-signed URL.
+ Don't trust any certificates provided in the message without validation.
+ Reject any message with an unexpected `TopicArn` to prevent spoofing.
+ The AWS SDKs for Amazon SNS provide built-in validation logic, reducing the risk of misimplementation.
# Configuring the message signature version on Amazon SNS topics
Configuring the message signature version on Amazon SNS topics allows you to enhance the security and compatibility of your message verification process.
Select between `SignatureVersion`**1** (SHA1) and `SignatureVersion`**2** (SHA256) to control the hashing algorithm used for signing messages. Amazon SNS topics default to `SignatureVersion`**1**. You can configure this setting using the [https://docs.aws.amazon.com/sns/latest/api/API_SetTopicAttributes.html](https://docs.aws.amazon.com/sns/latest/api/API_SetTopicAttributes.html) API action.
Use the following example to set the topic attribute `SignatureVersion` using the AWS CLI:
```
aws sns set-topic-attributes \
--topic-arn arn:aws:sns:us-east-2:123456789012:MyTopic \
--attribute-name SignatureVersion \
--attribute-value 2
```
# Verifying the signature of an Amazon SNS message when using HTTP query-based requests
Verifying the signature of an Amazon SNS message when using HTTP query-based requests ensures the message's authenticity and integrity. This process confirms that the message originates from Amazon SNS and has not been tampered with during transit. By parsing the message, constructing the correct string to sign, and validating the signature against a trusted public key, you safeguard your system against spoofing and unauthorized message alterations.
1. Extract **key-value pairs** from the JSON document in the HTTP POST request body sent by Amazon SNS. These fields are required to construct the **string to sign**.
+ `Message`
+ `Subject` (if present)
+ `MessageId`
+ `Timestamp`
+ `TopicArn`
+ `Type`
For example:
```
MESSAGE_FILE="message.json"
FIELDS=("Message" "MessageId" "Subject" "Timestamp" "TopicArn" "Type")
```
**Note**
If any field contains escaped characters (for example, `\n`), convert them to their **original form** to ensure an exact match.
1. Locate the `SigningCertURL` field in the Amazon SNS message. This certificate contains the public key needed to verify the message signature. For example:
```
SIGNING_CERT_URL=$(jq -r '.SigningCertURL' "$MESSAGE_FILE")
```
1. Ensure the `SigningCertURL` is from a trusted AWS domain (for example, https://sns.us-east-1.amazonaws.com). Reject any URLs **outside AWS domains** for security reasons.
1. Download the **X.509 certificate **from the provided URL. For example:
```
curl -s "$SIGNING_CERT_URL" -o signing_cert.pem
```
1. Extract the **public key** from the downloaded X.509 certificate. The public key allows you to decrypt the message's signature and verify its integrity. For example:
```
openssl x509 -pubkey -noout -in signing_cert.pem > public_key.pem
```
1. Different message types require different key-value pairs in the string to sign. Identify the **message type** (`Type` field in the Amazon SNS message) to determine which **key-value pairs** to include:
+ **Notification message** – Includes `Message`, `MessageId`, `Subject` (if present), `Timestamp`, `TopicArn`, and `Type`.
+ **SubscriptionConfirmation** or **UnsubscribeConfirmation message** – Includes `Message`, `MessageId`, `SubscribeURL`, `Timestamp`, `Token`, `TopicArn`, and `Type`.
1. Amazon SNS requires the string to sign to follow a strict, fixed field order for verification. **Only the explicitly required fields must be included**—no extra fields can be added. Optional fields, such as `Subject`, must be included only if present in the message and must appear in the exact position defined by the required field order. For example:
```
KeyNameOne\nValueOne\nKeyNameTwo\nValueTwo
```
**Important**
Do not add a newline character at the end of the string.
1. Arrange the** key-value pairs** in byte-sort order (alphabetical by key name).
1. Construct the **string to sign **using the following format example:
```
STRING_TO_SIGN=""
for FIELD in "${FIELDS[@]}"; do
VALUE=$(jq -r --arg field "$FIELD" '.[$field]' "$MESSAGE_FILE")
STRING_TO_SIGN+="$FIELD\n$VALUE"
# Append a newline after each field except the last one
if [[ "$FIELD" != "Type" ]]; then
STRING_TO_SIGN+="\n"
fi
done
```
**Notification message example:**
```
Message
My Test Message
MessageId
4d4dc071-ddbf-465d-bba8-08f81c89da64
Subject
My subject
Timestamp
2019-01-31T04:37:04.321Z
TopicArn
arn:aws:sns:us-east-2:123456789012:s4-MySNSTopic-1G1WEFCOXTC0P
Type
Notification
```
**SubscriptionConfirmation example:**
```
Message
Please confirm your subscription
MessageId
3d891288-136d-417f-bc05-901c108273ee
SubscribeURL
https://sns.us-east-2.amazonaws.com/...
Timestamp
2024-01-01T00:00:00.000Z
Token
abc123...
TopicArn
arn:aws:sns:us-east-2:123456789012:MyTopic
Type
SubscriptionConfirmation
```
1. The `Signature` field in the message is Base64-encoded. You need to **decode** it to compare its **raw binary form** with the **derived hash**. For example:
```
SIGNATURE=$(jq -r '.Signature' "$MESSAGE_FILE")
echo "$SIGNATURE" | base64 -d > signature.bin
```
1. Use the `SignatureVersion` field to select the hash algorithm:
+ For `SignatureVersion`**1**, use **SHA1** (for example, `-sha1`).
+ For `SignatureVersion`**2**, use **SHA256** (for example, `-sha256`).
1. To confirm the authenticity of the Amazon SNS message, generate a **hash** of the constructed string and verify the signature using the **public key**.
```
openssl dgst -sha256 -verify public_key.pem -signature signature.bin <<< "$STRING_TO_SIGN"
```
If the signature is valid, the output is `Verified OK`. Otherwise, the output is `Verification Failure`.
## Example script with error handling
The following example script automates the verification process:
```
#!/bin/bash
# Path to the local message file
MESSAGE_FILE="message.json"
# Extract the SigningCertURL and Signature from the message
SIGNING_CERT_URL=$(jq -r '.SigningCertURL' "$MESSAGE_FILE")
SIGNATURE=$(jq -r '.Signature' "$MESSAGE_FILE")
# Fetch the X.509 certificate
curl -s "$SIGNING_CERT_URL" -o signing_cert.pem
# Extract the public key from the certificate
openssl x509 -pubkey -noout -in signing_cert.pem > public_key.pem
# Define the fields to include in the string to sign
FIELDS=("Message" "MessageId" "Subject" "Timestamp" "TopicArn" "Type")
# Initialize the string to sign
STRING_TO_SIGN=""
# Iterate over the fields to construct the string to sign
for FIELD in "${FIELDS[@]}"; do
VALUE=$(jq -r --arg field "$FIELD" '.[$field]' "$MESSAGE_FILE")
STRING_TO_SIGN+="$FIELD\n$VALUE"
# Append a newline after each field except the last one
if [[ "$FIELD" != "Type" ]]; then
STRING_TO_SIGN+="\n"
fi
done
# Verify the signature
echo -e "$STRING_TO_SIGN" | openssl dgst -sha256 -verify public_key.pem -signature <(echo "$SIGNATURE" | base64 -d)
```
# Parsing Amazon SNS message formats
When Amazon SNS sends messages to HTTP/HTTPS endpoints, they contain both HTTP headers and a JSON message body. These messages follow a structured format that includes metadata such as the message type, topic ARN, timestamps, and digital signatures. By correctly parsing Amazon SNS messages, you can determine whether a message is a subscription confirmation, notification, or unsubscribe confirmation, extract relevant data, and verify authenticity using signature validation.
# HTTP/HTTPS headers
When Amazon SNS sends a subscription confirmation, notification, or unsubscribe confirmation message to HTTP/HTTPS endpoints, it sends a POST message with a number of Amazon SNS-specific header values. You can use header values for such tasks as identifying the message type without having to parse the JSON message body to read the `Type` value. By default, Amazon SNS sends all the notification to HTTP/S endpoints with `Content-Type` set to `text/plain; charset=UTF-8`. To choose a `Content-Type` other than text/plain (default), see `headerContentType` in [Creating an HTTP/S delivery policy](sns-message-delivery-retries.md#creating-delivery-policy).
**`x-amz-sns-message-type`**
The type of message. The possible values are `SubscriptionConfirmation`, `Notification`, and `UnsubscribeConfirmation`.
**`x-amz-sns-message-id`**
A Universally Unique Identifier (UUID), unique for each message published. For a notification that Amazon SNS resends during a retry, the message ID of the original message is used.
**`x-amz-sns-topic-arn`**
The Amazon Resource Name (ARN) for the topic that this message was published to.
**`x-amz-sns-subscription-arn`**
The ARN for the subscription to this endpoint.
The following HTTP POST header is an example of a header for a `Notification` message to an HTTP endpoint.
```
POST / HTTP/1.1
x-amz-sns-message-type: Notification
x-amz-sns-message-id: 165545c9-2a5c-472c-8df2-7ff2be2b3b1b
x-amz-sns-topic-arn: arn:aws:sns:us-west-2:123456789012:MyTopic
x-amz-sns-subscription-arn: arn:aws:sns:us-west-2:123456789012:MyTopic:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55
Content-Length: 1336
Content-Type: text/plain; charset=UTF-8
Host: myhost.example.com
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
```
# HTTP/HTTPS subscription confirmation JSON format
After you subscribe an HTTP/HTTPS endpoint, Amazon SNS sends a subscription confirmation message to the HTTP/HTTPS endpoint. This message contains a `SubscribeURL` value that you must visit to confirm the subscription (alternatively, you can use the `Token` value with the [https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html](https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html)).
**Note**
Amazon SNS doesn't send notifications to this endpoint until the subscription is confirmed
The subscription confirmation message is a POST message with a message body that contains a JSON document with the following name-value pairs.
**`Type`**
The type of message. For a subscription confirmation, the type is `SubscriptionConfirmation`.
**`MessageId`**
A Universally Unique Identifier (UUID), unique for each message published. For a message that Amazon SNS resends during a retry, the message ID of the original message is used.
**`Token`**
A value you can use with the [https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html](https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html) action to confirm the subscription. Alternatively, you can simply visit the `SubscribeURL`.
**`TopicArn`**
The Amazon Resource Name (ARN) for the topic that this endpoint is subscribed to.
**`Message`**
A string that describes the message. For subscription confirmation, this string looks like this:
```
You have chosen to subscribe to the topic arn:aws:sns:us-east-2:123456789012:MyTopic.\nTo confirm the subscription, visit the SubscribeURL included in this message.
```
**`SubscribeURL`**
The URL that you must visit in order to confirm the subscription. Alternatively, you can instead use the `Token` with the [https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html](https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html) action to confirm the subscription.
**`Timestamp`**
The time (GMT) when the subscription confirmation was sent.
**`SignatureVersion`**
Version of the Amazon SNS signature used.
+ If the `SignatureVersion` is **1**, `Signature` is a Base64-encoded `SHA1withRSA` signature of the `Message`, `MessageId`, `Type`, `Timestamp`, and `TopicArn` values.
+ If the `SignatureVersion` is **2**, `Signature` is a Base64-encoded `SHA256withRSA` signature of the `Message`, `MessageId`, `Type`, `Timestamp`, and `TopicArn` values.
**`Signature`**
Base64-encoded `SHA1withRSA` or `SHA256withRSA` signature of the `Message`, `MessageId`, `Type`, `Timestamp`, and `TopicArn` values.
**`SigningCertURL`**
The URL to the certificate that was used to sign the message.
The following HTTP POST message is an example of a `SubscriptionConfirmation` message to an HTTP endpoint.
```
POST / HTTP/1.1
x-amz-sns-message-type: SubscriptionConfirmation
x-amz-sns-message-id: 165545c9-2a5c-472c-8df2-7ff2be2b3b1b
x-amz-sns-topic-arn: arn:aws:sns:us-west-2:123456789012:MyTopic
Content-Length: 1336
Content-Type: text/plain; charset=UTF-8
Host: myhost.example.com
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
{
"Type" : "SubscriptionConfirmation",
"MessageId" : "165545c9-2a5c-472c-8df2-7ff2be2b3b1b",
"Token" : "2336412f37...",
"TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
"Message" : "You have chosen to subscribe to the topic arn:aws:sns:us-west-2:123456789012:MyTopic.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
"SubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-west-2:123456789012:MyTopic&Token=2336412f37...",
"Timestamp" : "2012-04-26T20:45:04.751Z",
"SignatureVersion" : "1",
"Signature" : "EXAMPLEpH+DcEwjAPg8O9mY8dReBSwksfg2S7WKQcikcNKWLQjwu6A4VbeS0QHVCkhRS7fUQvi2egU3N858fiTDN6bkkOxYDVrY0Ad8L10Hs3zH81mtnPk5uvvolIC1CXGu43obcgFxeL3khZl8IKvO61GWB6jI9b5+gLPoBc1Q=",
"SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem"
}
```
# HTTP/HTTPS notification JSON format
When Amazon SNS sends a notification to a subscribed HTTP or HTTPS endpoint, the POST message sent to the endpoint has a message body that contains a JSON document with the following name-value pairs.
**`Type`**
The type of message. For a notification, the type is `Notification`.
**`MessageId`**
A Universally Unique Identifier (UUID), unique for each message published. For a notification that Amazon SNS resends during a retry, the message ID of the original message is used.
**`TopicArn`**
The Amazon Resource Name (ARN) for the topic that this message was published to.
**`Subject`**
The `Subject` parameter specified when the notification was published to the topic.
This is an optional parameter. If no `Subject` was specified, then this name-value pair does not appear in this JSON document.
**`Message`**
The `Message` value specified when the notification was published to the topic.
**`Timestamp`**
The time (GMT) when the notification was published.
**`SignatureVersion`**
Version of the Amazon SNS signature used.
+ If the `SignatureVersion` is **1**, `Signature` is a Base64-encoded `SHA1withRSA` signature of the `Message`, `MessageId`, `Subject` (if present), `Type`, `Timestamp`, and `TopicArn` values.
+ If the `SignatureVersion` is **2**, `Signature` is a Base64-encoded `SHA256withRSA` signature of the `Message`, `MessageId`, `Subject` (if present), `Type`, `Timestamp`, and `TopicArn` values.
**`Signature`**
Base64-encoded `SHA1withRSA` or `SHA256withRSA` signature of the `Message`, `MessageId`, `Subject` (if present), `Type`, `Timestamp`, and `TopicArn` values.
**`SigningCertURL`**
The URL to the certificate that was used to sign the message.
**`UnsubscribeURL`**
A URL that you can use to unsubscribe the endpoint from this topic. If you visit this URL, Amazon SNS unsubscribes the endpoint and stops sending notifications to this endpoint.
The following HTTP POST message is an example of a `Notification` message to an HTTP endpoint.
```
POST / HTTP/1.1
x-amz-sns-message-type: Notification
x-amz-sns-message-id: 22b80b92-fdea-4c2c-8f9d-bdfb0c7bf324
x-amz-sns-topic-arn: arn:aws:sns:us-west-2:123456789012:MyTopic
x-amz-sns-subscription-arn: arn:aws:sns:us-west-2:123456789012:MyTopic:c9135db0-26c4-47ec-8998-413945fb5a96
Content-Length: 773
Content-Type: text/plain; charset=UTF-8
Host: myhost.example.com
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
{
"Type" : "Notification",
"MessageId" : "22b80b92-fdea-4c2c-8f9d-bdfb0c7bf324",
"TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
"Subject" : "My First Message",
"Message" : "Hello world!",
"Timestamp" : "2012-05-02T00:54:06.655Z",
"SignatureVersion" : "1",
"Signature" : "EXAMPLEw6JRN...",
"SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",
"UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:123456789012:MyTopic:c9135db0-26c4-47ec-8998-413945fb5a96"
}
```
# HTTP/HTTPS unsubscribe confirmation JSON format
After an HTTP/HTTPS endpoint is unsubscribed from a topic, Amazon SNS sends an unsubscribe confirmation message to the endpoint.
The unsubscribe confirmation message is a POST message with a message body that contains a JSON document with the following name-value pairs.
**`Type`**
The type of message. For a unsubscribe confirmation, the type is `UnsubscribeConfirmation`.
**`MessageId`**
A Universally Unique Identifier (UUID), unique for each message published. For a message that Amazon SNS resends during a retry, the message ID of the original message is used.
**`Token`**
A value you can use with the [https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html](https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html) action to re-confirm the subscription. Alternatively, you can simply visit the `SubscribeURL`.
**`TopicArn`**
The Amazon Resource Name (ARN) for the topic that this endpoint has been unsubscribed from.
**`Message`**
A string that describes the message. For unsubscribe confirmation, this string looks like this:
```
You have chosen to deactivate subscription arn:aws:sns:us-east-2:123456789012:MyTopic:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55.\nTo cancel this operation and restore the subscription, visit the SubscribeURL included in this message.
```
**`SubscribeURL`**
The URL that you must visit in order to re-confirm the subscription. Alternatively, you can instead use the `Token` with the [https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html](https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html) action to re-confirm the subscription.
**`Timestamp`**
The time (GMT) when the unsubscribe confirmation was sent.
**`SignatureVersion`**
Version of the Amazon SNS signature used.
+ If the `SignatureVersion` is **1**, `Signature` is a Base64-encoded `SHA1withRSA` signature of the `Message`, `MessageId`, `Type`, `Timestamp`, and `TopicArn` values.
+ If the `SignatureVersion` is **2**, `Signature` is a Base64-encoded `SHA256withRSA` signature of the `Message`, `MessageId`, `Type`, `Timestamp`, and `TopicArn` values.
**`Signature`**
Base64-encoded `SHA1withRSA` or `SHA256withRSA` signature of the `Message`, `MessageId`, `Type`, `Timestamp`, and `TopicArn` values.
**`SigningCertURL`**
The URL to the certificate that was used to sign the message.
The following HTTP POST message is an example of a `UnsubscribeConfirmation` message to an HTTP endpoint.
```
POST / HTTP/1.1
x-amz-sns-message-type: UnsubscribeConfirmation
x-amz-sns-message-id: 47138184-6831-46b8-8f7c-afc488602d7d
x-amz-sns-topic-arn: arn:aws:sns:us-west-2:123456789012:MyTopic
x-amz-sns-subscription-arn: arn:aws:sns:us-west-2:123456789012:MyTopic:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55
Content-Length: 1399
Content-Type: text/plain; charset=UTF-8
Host: myhost.example.com
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
{
"Type" : "UnsubscribeConfirmation",
"MessageId" : "47138184-6831-46b8-8f7c-afc488602d7d",
"Token" : "2336412f37...",
"TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
"Message" : "You have chosen to deactivate subscription arn:aws:sns:us-west-2:123456789012:MyTopic:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55.\nTo cancel this operation and restore the subscription, visit the SubscribeURL included in this message.",
"SubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-west-2:123456789012:MyTopic&Token=2336412f37fb6...",
"Timestamp" : "2012-04-26T20:06:41.581Z",
"SignatureVersion" : "1",
"Signature" : "EXAMPLEHXgJm...",
"SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem"
}
```
# SetSubscriptionAttributes delivery policy JSON format
If you send a request to the `SetSubscriptionAttributes` action and set the `AttributeName` parameter to a value of `DeliveryPolicy`, the value of the `AttributeValue` parameter must be a valid JSON object. For example, the following example sets the delivery policy to 5 total retries.
```
http://sns.us-east-2.amazonaws.com/
?Action=SetSubscriptionAttributes
&SubscriptionArn=arn%3Aaws%3Asns%3Aus-east-2%3A123456789012%3AMy-Topic%3A80289ba6-0fd4-4079-afb4-ce8c8260f0ca
&AttributeName=DeliveryPolicy
&AttributeValue={"healthyRetryPolicy":{"numRetries":5}}
...
```
Use the following JSON format for the value of the `AttributeValue` parameter.
```
{
"healthyRetryPolicy" : {
"minDelayTarget" : int,
"maxDelayTarget" : int,
"numRetries" : int,
"numMaxDelayRetries" : int,
"backoffFunction" : "linear|arithmetic|geometric|exponential"
},
"throttlePolicy" : {
"maxReceivesPerSecond" : int
},
"requestPolicy" : {
"headerContentType" : "text/plain | application/json | application/xml"
}
}
```
For more information about the `SetSubscriptionAttribute` action, go to [SetSubscriptionAttributes](https://docs.aws.amazon.com/sns/latest/api/API_SetSubscriptionAttributes.html) in the *Amazon Simple Notification Service API Reference*. For more information on the supported HTTP content-type headers, see [Creating an HTTP/S delivery policy](sns-message-delivery-retries.md#creating-delivery-policy).
# SetTopicAttributes delivery policy JSON format
If you send a request to the `SetTopicAttributes` action and set the `AttributeName` parameter to a value of `DeliveryPolicy`, the value of the `AttributeValue` parameter must be a valid JSON object. For example, the following example sets the delivery policy to 5 total retries.
```
http://sns.us-east-2.amazonaws.com/
?Action=SetTopicAttributes
&TopicArn=arn%3Aaws%3Asns%3Aus-east-2%3A123456789012%3AMy-Topic
&AttributeName=DeliveryPolicy
&AttributeValue={"http":{"defaultHealthyRetryPolicy":{"numRetries":5}}}
...
```
Use the following JSON format for the value of the `AttributeValue` parameter.
```
{
"http" : {
"defaultHealthyRetryPolicy" : {
"minDelayTarget": int,
"maxDelayTarget": int,
"numRetries": int,
"numMaxDelayRetries": int,
"backoffFunction": "linear|arithmetic|geometric|exponential"
},
"disableSubscriptionOverrides" : Boolean,
"defaultThrottlePolicy" : {
"maxReceivesPerSecond" : int
},
"defaultRequestPolicy" : {
"headerContentType" : "text/plain | application/json | application/xml"
}
}
}
```
For more information about the `SetTopicAttribute` action, go to [SetTopicAttributes](https://docs.aws.amazon.com/sns/latest/api/API_SetTopicAttributes.html) in the *Amazon Simple Notification Service API Reference*. For more information on the supported HTTP content-type headers, see [Creating an HTTP/S delivery policy](sns-message-delivery-retries.md#creating-delivery-policy).
# Fanout Amazon SNS events to AWS Event Fork Pipelines
| |
| --- |
| For event archiving and analytics, Amazon SNS now recommends using its native integration with Amazon Data Firehose. You can subscribe Firehose delivery streams to SNS topics, which allows you to send notifications to archiving and analytics endpoints such as Amazon Simple Storage Service (Amazon S3) buckets, Amazon Redshift tables, Amazon OpenSearch Service (OpenSearch Service), and more. Using Amazon SNS with Firehose delivery streams is a fully-managed and codeless solution that doesn't require you to use AWS Lambda functions. For more information, see [Fanout to Firehose delivery streams](sns-firehose-as-subscriber.md). |
You can use Amazon SNS to build event-driven applications which use subscriber services to perform work automatically in response to events triggered by publisher services. This architectural pattern can make services more reusable, interoperable, and scalable. However, it can be labor-intensive to fork the processing of events into pipelines that address common event handling requirements, such as event storage, backup, search, analytics, and replay.
To accelerate the development of your event-driven applications, you can subscribe event-handling pipelines—powered by AWS Event Fork Pipelines—to Amazon SNS topics. AWS Event Fork Pipelines is a suite of open-source [nested applications](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-nested-applications.html), based on the [AWS Serverless Application Model](https://aws.amazon.com/serverless/sam/) (AWS SAM), which you can deploy directly from the [AWS Event Fork Pipelines suite](https://serverlessrepo.aws.amazon.com/applications?query=aws-event-fork-pipelines) (choose **Show apps that create custom IAM roles or resource policies**) into your AWS account.
For an AWS Event Fork Pipelines use case, see [Deploying and testing the Amazon SNS event fork pipelines sample application](sns-deploy-test-fork-pipelines-sample-application.md).
**Topics**
+ [How AWS Event Fork Pipelines works](#how-sns-fork-works)
+ [Deploying AWS Event Fork Pipelines](#deploying-sns-fork-pipelines)
+ [Deploying and testing the Amazon SNS event fork pipelines sample application](sns-deploy-test-fork-pipelines-sample-application.md)
+ [Subscribing AWS Event Fork Pipelines to an Amazon SNS topic](sns-subscribe-event-fork-pipelines.md)
## How AWS Event Fork Pipelines works
AWS Event Fork Pipelines is a serverless design pattern. However, it is also a suite of nested serverless applications based on AWS SAM (which you can deploy directly from the AWS Serverless Application Repository (AWS SAR) to your AWS account in order to enrich your event-driven platforms). You can deploy these nested applications individually, as your architecture requires.
**Topics**
+ [The event storage and backup pipeline](#sns-fork-event-storage-and-backup-pipeline)
+ [The event search and analytics pipeline](#sns-fork-event-search-and-analytics-pipeline)
+ [The event replay pipeline](#sns-fork-event-replay-pipeline)
The following diagram shows an AWS Event Fork Pipelines application supplemented by three nested applications. You can deploy any of the pipelines from the AWS Event Fork Pipelines suite on the AWS SAR independently, as your architecture requires.
![\[The the AWS Event Fork Pipelines architecture, showing how events from an Amazon SNS topic are filtered and processed through three distinct pipelines: Event Storage and Backup, Event Search and Analytics, and Event Replay. These pipelines are depicted as vertically stacked boxes, each independently processing events in parallel from the same Amazon SNS topic.\]](http://docs.aws.amazon.com/sns/latest/dg/images/sns-fork-pipeline-as-subscriber-how-it-works.png)
Each pipeline is subscribed to the same Amazon SNS topic, allowing itself to process events in parallel as these events are published to the topic. Each pipeline is independent and can set its own [Subscription Filter Policy](sns-subscription-filter-policies.md). This allows a pipeline to process only a subset of the events that it is interested in (rather than all events published to the topic).
**Note**
Because you place the three AWS Event Fork Pipelines alongside your regular event processing pipelines (possibly already subscribed to your Amazon SNS topic), you don’t need to change any portion of your current message publisher to take advantage of AWS Event Fork Pipelines in your existing workloads.
### The event storage and backup pipeline
The following diagram shows the [Event Storage and Backup Pipeline](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:077246666028:applications~fork-event-storage-backup-pipeline). You can subscribe this pipeline to your Amazon SNS topic to automatically back up the events flowing through your system.
This pipeline is comprised of an Amazon SQS queue that buffers the events delivered by the Amazon SNS topic, an AWS Lambda function that automatically polls for these events in the queue and pushes them into an stream, and an Amazon S3 bucket that durably backs up the events loaded by the stream.
![\[The Fork-Event-Storage-Backup-Pipeline, which is designed to process and back up events from an Amazon SNS topic. The flow starts with an Amazon SNS topic from which events are fanned out to an Amazon SQS queue. These filtered events are then processed by an Lambda function, which forwards them to an Data Firehose. The Firehose stream is responsible for buffering, transforming, and compressing the events before loading them into an Amazon S3 backup bucket. Finally, Amazon Athena can be used to query the stored data. The diagram uses a series of icons and arrows to illustrate the flow from one service to the next, clearly labeling each component of the pipeline.\]](http://docs.aws.amazon.com/sns/latest/dg/images/sns-fork-event-storage-and-backup-pipeline.png)
To fine-tune the behavior of your Firehose stream, you can configure it to buffer, transform, and compress your events prior to loading them into the bucket. As events are loaded, you can use Amazon Athena to query the bucket using standard SQL queries. You can also configure the pipeline to reuse an existing Amazon S3 bucket or create a new one.
### The event search and analytics pipeline
The following diagram shows the [Event Search and Analytics Pipeline](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:077246666028:applications~fork-event-search-analytics-pipeline). You can subscribe this pipeline to your Amazon SNS topic to index the events that flow through your system in a search domain and then run analytics on them.
This pipeline is comprised of an Amazon SQS queue that buffers the events delivered by the Amazon SNS topic, an AWS Lambda function that polls events from the queue and pushes them into an stream, an Amazon OpenSearch Service domain that indexes the events loaded by the Firehose stream, and an Amazon S3 bucket that stores the dead-letter events that can’t be indexed in the search domain.
![\[The Event Search and Analytics Pipeline within an AWS architecture. It starts on the left with the Amazon SNS topic receiving all events. These events are then funneled through a dashed line representing "fan out filtered events" into an Amazon SQS queue. From the queue, events are processed by an Lambda function which then forwards them to an Data Firehose stream. The Data Firehose directs the events into two destinations: one route leads to an Amazon Elasticsearch Service for indexing, and the other route sends unprocessable or "dead-letter" events to an Amazon S3 dead-letter bucket. On the far right, the output from the Elasticsearch Service feeds into a Kibana dashboard for analytics and visualization. The entire flow is laid out horizontally and each component is connected by lines showing the direction of data flow.\]](http://docs.aws.amazon.com/sns/latest/dg/images/sns-fork-event-search-and-analytics-pipeline.png)
To fine-tune your Firehose stream in terms of event buffering, transformation, and compression, you can configure this pipeline.
You can also configure whether the pipeline should reuse an existing OpenSearch domain in your AWS account or create a new one for you. As events are indexed in the search domain, you can use Kibana to run analytics on your events and update visual dashboards in real-time.
### The event replay pipeline
The following diagram shows the [Event Replay Pipeline](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:077246666028:applications~fork-event-replay-pipeline). To record the events that have been processed by your system for the past 14 days (for example when your platform needs to recover from failure), you can subscribe this pipeline to your Amazon SNS topic and then reprocess the events.
This pipeline is comprised of an Amazon SQS queue that buffers the events delivered by the Amazon SNS topic, and an AWS Lambda function that polls events from the queue and redrives them into your regular event processing pipeline, which is also subscribed to your topic.
![\[The Event Replay Pipeline in a flowchart format. From left to right, it begins with an Amazon SNS topic that distributes filtered events to two parallel processes. The upper flow represents your regular event processing pipeline, which includes an Amazon SQS queue that processes events. The lower flow, labeled as the "fork-event-replay-pipeline," includes an Amazon SQS replay queue where events are temporarily stored before being processed by a Lambda replay function. This Lambda function has the capability to re-drive events into your regular event processing pipeline or hold them for replay, based on whether the replay feature is enabled or disabled. The diagram also indicates that operators have control over enabling or disabling the event replay functionality.\]](http://docs.aws.amazon.com/sns/latest/dg/images/sns-fork-event-replay-pipeline.png)
**Note**
By default, the replay function is disabled, not redriving your events. If you need to reprocess events, you must enable the Amazon SQS replay queue as an event source for the AWS Lambda replay function.
## Deploying AWS Event Fork Pipelines
The [AWS Event Fork Pipelines suite](https://serverlessrepo.aws.amazon.com/applications?query=aws-event-fork-pipelines) (choose **Show apps that create custom IAM roles or resource policies**) is available as a group of public applications in the AWS Serverless Application Repository, from where you can deploy and test them manually using the [AWS Lambda console](https://console.aws.amazon.com/lambda/). For information about deploying pipelines using the AWS Lambda console, see [Subscribing AWS Event Fork Pipelines to an Amazon SNS topic](sns-subscribe-event-fork-pipelines.md).
In a production scenario, we recommend embedding AWS Event Fork Pipelines within your overall application's AWS SAM template. The nested-application feature lets you do this by adding the resource `[AWS::Serverless::Application](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template.html#serverless-sam-template-application)` to your AWS SAM template, referencing the AWS SAR `ApplicationId` and the `SemanticVersion` of the nested application.
For example, you can use the Event Storage and Backup Pipeline as a nested application by adding the following YAML snippet to the `Resources` section of your AWS SAM template.
```
Backup:
Type: AWS::Serverless::Application
Properties:
Location:
ApplicationId: arn:aws:serverlessrepo:us-east-2:123456789012:applications/fork-event-storage-backup-pipeline
SemanticVersion: 1.0.0
Parameters:
#The ARN of the Amazon SNS topic whose messages should be backed up to the Amazon S3 bucket.
TopicArn: !Ref MySNSTopic
```
When you specify parameter values, you can use AWS CloudFormation intrinsic functions to reference other resources in your template. For example, in the YAML snippet above, the `TopicArn` parameter references the `[AWS::SNS::Topic](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sns-topic.html)` resource `MySNSTopic`, defined elsewhere in the AWS SAM template. For more information, see the [Intrinsic Function Reference](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html) in the *AWS CloudFormation User Guide*.
**Note**
The AWS Lambda console page for your AWS SAR application includes the **Copy as SAM Resource** button, which copies the YAML required for nesting an AWS SAR application to the clipboard.
# Deploying and testing the Amazon SNS event fork pipelines sample application
To accelerate the development of your event-driven applications, you can subscribe event-handling pipelines—powered by AWS Event Fork Pipelines—to Amazon SNS topics. AWS Event Fork Pipelines is a suite of open-source [nested applications](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-nested-applications.html), based on the [AWS Serverless Application Model](https://aws.amazon.com/serverless/sam/) (AWS SAM), which you can deploy directly from the [AWS Event Fork Pipelines suite](https://serverlessrepo.aws.amazon.com/applications?query=aws-event-fork-pipelines) (choose **Show apps that create custom IAM roles or resource policies**) into your AWS account. For more information, see [How AWS Event Fork Pipelines works](sns-fork-pipeline-as-subscriber.md#how-sns-fork-works).
This page shows how you can use the AWS Management Console to deploy and test the AWS Event Fork Pipelines sample application.
**Important**
To avoid incurring unwanted costs after you finish deploying the AWS Event Fork Pipelines sample application, delete its CloudFormation stack. For more information, see [Deleting a Stack on the CloudFormation Console](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-delete-stack.html) in the *AWS CloudFormation User Guide*.
# AWS Event Fork Pipelines use case example
The following scenario describes an event-driven, serverless e-commerce application that uses AWS Event Fork Pipelines. You can use this [example e-commerce application](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:077246666028:applications~fork-example-ecommerce-checkout-api) in the AWS Serverless Application Repository and then deploy it in your AWS account using the AWS Lambda console, where you can test it and examine its source code in GitHub.
![\[The architecture of a serverless e-commerce application that integrates AWS services. It depicts the flow from ecommerce users placing orders via an API Gateway to different processing pipelines including order storage, search analytics, and replay, showcasing how events are managed and analyzed through Amazon SNS, Lambda, Amazon SQS, DynamoDB, and Kibana.\]](http://docs.aws.amazon.com/sns/latest/dg/images/sns-fork-example-use-case.png)
This e-commerce application takes orders from buyers through a RESTful API hosted by API Gateway and backed by the AWS Lambda function `CheckoutApiBackendFunction`. This function publishes all received orders to an Amazon SNS topic named `CheckoutEventsTopic` which, in turn, fans out the orders to four different pipelines.
The first pipeline is the regular checkout-processing pipeline designed and implemented by the owner of the e-commerce application. This pipeline has the Amazon SQS queue `CheckoutQueue` that buffers all received orders, an AWS Lambda function named `CheckoutFunction` that polls the queue to process these orders, and the DynamoDB table `CheckoutTable` that securely saves all placed orders.
## Applying AWS Event Fork Pipelines
The components of the e-commerce application handle the core business logic. However, the e-commerce application owner also needs to address the following:
+ **Compliance**—secure, compressed backups encrypted at rest and sanitization of sensitive information
+ **Resiliency**—replay of most recent orders in case of the disruption of the fulfillment process
+ **Searchability**—running analytics and generating metrics on placed orders
Instead of implementing this event processing logic, the application owner can subscribe AWS Event Fork Pipelines to the `CheckoutEventsTopic` Amazon SNS topic
+ [The event storage and backup pipeline](sns-fork-pipeline-as-subscriber.md#sns-fork-event-storage-and-backup-pipeline) is configured to transform data to remove credit card details, buffer data for 60 seconds, compress it using GZIP, and encrypt it using the default customer managed key for Amazon S3. This key is managed by AWS and powered by the AWS Key Management Service (AWS KMS).
For more information, see [Choose Amazon S3 For Your Destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-s3), [Amazon Data Firehose Data Transformation](https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html), and [Configure Settings](https://docs.aws.amazon.com/firehose/latest/dev/create-configure.html) in the *Amazon Data Firehose Developer Guide*.
+ [The event search and analytics pipeline](sns-fork-pipeline-as-subscriber.md#sns-fork-event-search-and-analytics-pipeline) is configured with an index retry duration of 30 seconds, a bucket for storing orders that fail to be indexed in the search domain, and a filter policy to restrict the set of indexed orders.
For more information, see [Choose OpenSearch Service for your Destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-elasticsearch) in the *Amazon Data Firehose Developer Guide*.
+ [The event replay pipeline](sns-fork-pipeline-as-subscriber.md#sns-fork-event-replay-pipeline) is configured with the Amazon SQS queue part of the regular order-processing pipeline designed and implemented by the e-commerce application owner.
For more information, see [Queue Name and URL](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-general-identifiers.html#queue-name-url) in the *Amazon Simple Queue Service Developer Guide*.
The following JSON filter policy is set in the configuration for the Event Search and Analytics Pipeline. It matches only incoming orders in which the total amount is \$1100 or higher. For more information, see [Amazon SNS message filtering](sns-message-filtering.md).
```
{
"amount": [{ "numeric": [ ">=", 100 ] }]
}
```
Using the AWS Event Fork Pipelines pattern, the e-commerce application owner can avoid the development overhead that often follows coding undifferentiating logic for event handling. Instead, she can deploy AWS Event Fork Pipelines directly from the AWS Serverless Application Repository into her AWS account.
# Step 1: Deploying the sample Amazon SNS application
1. Sign in to the [AWS Lambda console](https://console.aws.amazon.com/lambda/).
1. On the navigation panel, choose **Functions** and then choose **Create function**.
1. On the **Create function** page, do the following:
1. Choose **Browse serverless app repository**, **Public applications**, **Show apps that create custom IAM roles or resource policies**.
1. Search for `fork-example-ecommerce-checkout-api` and then choose the application.
1. On the **fork-example-ecommerce-checkout-api** page, do the following:
1. In the **Application settings** section, enter an **Application name** (for example, `fork-example-ecommerce-my-app`).
**Note**
To find your resources easily later, keep the prefix `fork-example-ecommerce`.
For each deployment, the application name must be unique. If you reuse an application name, the deployment will update only the previously deployed CloudFormation stack (rather than create a new one).
1. (Optional) Enter one of the following **LogLevel** settings for the execution of your application's Lambda function:
+ `DEBUG`
+ `ERROR`
+ `INFO` (default)
+ `WARNING`
1. Choose **I acknowledge that this app creates custom IAM roles, resource policies and deploys nested applications.** and then, at the bottom of the page, choose **Deploy**.
On the **Deployment status for fork-example-ecommerce-*my-app*** page, Lambda displays the **Your application is being deployed** status.
In the **Resources** section, CloudFormation begins to create the stack and displays the **CREATE\$1IN\$1PROGRESS** status for each resource. When the process is complete, CloudFormation displays the **CREATE\$1COMPLETE** status.
**Note**
It might take 20-30 minutes for all resources to be deployed.
When the deployment is complete, Lambda displays the **Your application has been deployed** status.
# Step 2: Executing the SNS-linked sample application
1. In the AWS Lambda console, on the navigation panel, choose **Applications**.
1. On the **Applications** page, in the search field, search for `serverlessrepo-fork-example-ecommerce-my-app` and then choose the application.
1. In the **Resources** section, do the following:
1. To find the resource whose type is **ApiGateway RestApi**, sort the resources by **Type**, for example `ServerlessRestApi`, and then expand the resource.
1. Two nested resources are displayed, of types **ApiGateway Deployment** and **ApiGateway Stage**.
1. Copy the link **Prod API endpoint** and append `/checkout` to it, for example:
```
https://abcdefghij.execute-api.us-east-2.amazonaws.com/Prod/checkout
```
1. Copy the following JSON to a file named `test_event.json`.
```
{
"id": 15311,
"date": "2019-03-25T23:41:11-08:00",
"status": "confirmed",
"customer": {
"id": 65144,
"quantity": 2,
"price": 25.00,
"subtotal": 50.00
}]
}
```
1. To send an HTTPS request to your API endpoint, pass the sample event payload as input by executing a `curl` command, for example:
```
curl -d "$(cat test_event.json)" https://abcdefghij.execute-api.us-east-2.amazonaws.com/Prod/checkout
```
The API returns the following empty response, indicating a successful execution:
```
{ }
```
# Step 3: Verifying Amazon SNS application and pipeline performance
## Step 1: Verifying the execution of the sample checkout pipeline
1. Sign in to the [Amazon DynamoDB console](https://console.aws.amazon.com/dynamodb/).
1. On the navigation panel, choose **Tables**.
1. Search for `serverlessrepo-fork-example` and choose `CheckoutTable`.
1. On the table details page, choose **Items** and then choose the created item.
The stored attributes are displayed.
## Step 2: Verifying the execution of the event storage and backup pipeline
1. Sign in to the [Amazon S3 console](https://console.aws.amazon.com/s3/).
1. On the navigation panel, choose **Buckets**.
1. Search for `serverlessrepo-fork-example` and then choose `CheckoutBucket`.
1. Navigate the directory hierarchy until you find a file with the extension `.gz`.
1. To download the file, choose **Actions**, **Open**.
1. The pipeline is configured with a Lambda function that sanitizes credit card information for compliance reasons.
To verify that the stored JSON payload doesn't contain any credit card information, decompress the file.
## Step 3: Verifying the execution of the event search and analytics pipeline
1. Sign in to the [OpenSearch Service console](https://console.aws.amazon.com/aos/).
1. On the navigation panel, under **My domains**, choose the domain prefixed with `serverl-analyt`.
1. The pipeline is configured with an Amazon SNS subscription filter policy that sets a numeric matching condition.
To verify that the event is indexed because it refers to an order whose value is higher than USD \$1100, on the **serverl-analyt-*abcdefgh1ijk*** page, choose **Indices**, **checkout\$1events**.
## Step 4: Verifying the execution of the event replay pipeline
1. Sign in to the [Amazon SQS console](https://console.aws.amazon.com/sqs/).
1. In the list of queues, search for `serverlessrepo-fork-example` and choose `ReplayQueue`.
1. Choose **Send and receive messages**.
1. In the **Send and receive messages in fork-example-ecommerce-*my-app*...ReplayP-ReplayQueue-*123ABCD4E5F6*** dialog box, choose **Poll for messages**.
1. To verify that the event is enqueued, choose **More Details** next to the message that appears in the queue.
# Step 4: Simulating an issue and replay events for recovery
## Step 1: Enable the simulated issue and send a second API request
1. Sign in to the [AWS Lambda console](https://console.aws.amazon.com/lambda/).
1. On the navigation panel, choose **Functions**.
1. Search for `serverlessrepo-fork-example` and choose `CheckoutFunction`.
1. On the **fork-example-ecommerce-*my-app*-CheckoutFunction-*ABCDEF*...** page, in the **Environment variables** section, set the **BUG\$1ENABLED** variable to **true** and then choose **Save**.
1. Copy the following JSON to a file named `test_event_2.json`.
```
{
"id": 9917,
"date": "2019-03-26T21:11:10-08:00",
"status": "confirmed",
"customer": {
"id": 56999,
"quantity": 1,
"price": 75.00,
"subtotal": 75.00
}]
}
```
1. To send an HTTPS request to your API endpoint, pass the sample event payload as input by executing a `curl` command, for example:
```
curl -d "$(cat test_event_2.json)" https://abcdefghij.execute-api.us-east-2.amazonaws.com/Prod/checkout
```
The API returns the following empty response, indicating a successful execution:
```
{ }
```
## Step 2: Verify simulated data corruption
1. Sign in to the [Amazon DynamoDB console](https://console.aws.amazon.com/dynamodb/).
1. On the navigation panel, choose **Tables**.
1. Search for `serverlessrepo-fork-example` and choose `CheckoutTable`.
1. On the table details page, choose **Items** and then choose the created item.
The stored attributes are displayed, some marked as **CORRUPTED\$1**
## Step 3: Disable the simulated issue
1. Sign in to the [AWS Lambda console](https://console.aws.amazon.com/lambda/).
1. On the navigation panel, choose **Functions**.
1. Search for `serverlessrepo-fork-example` and choose `CheckoutFunction`.
1. On the **fork-example-ecommerce-*my-app*-CheckoutFunction-*ABCDEF*...** page, in the **Environment variables** section, set the **BUG\$1ENABLED** variable to **false** and then choose **Save**.
## Step 4: Enable replay to recover from the issue
1. In the AWS Lambda console, on the navigation panel, choose **Functions**.
1. Search for `serverlessrepo-fork-example` and choose `ReplayFunction`.
1. Expand the **Designer** section, choose the **SQS** tile and then, in the **SQS** section, choose **Enabled**.
**Note**
It takes approximately 1 minute for the Amazon SQS event source trigger to become enabled.
1. Choose **Save**.
1. To view the recovered attributes, return to the Amazon DynamoDB console.
1. To disable replay, return to the AWS Lambda console and disable the Amazon SQS event source trigger for `ReplayFunction`.
# Subscribing AWS Event Fork Pipelines to an Amazon SNS topic
To accelerate the development of your event-driven applications, you can subscribe event-handling pipelines—powered by AWS Event Fork Pipelines—to Amazon SNS topics. AWS Event Fork Pipelines is a suite of open-source [nested applications](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-nested-applications.html), based on the [AWS Serverless Application Model](https://aws.amazon.com/serverless/sam/) (AWS SAM), which you can deploy directly from the [AWS Event Fork Pipelines suite](https://serverlessrepo.aws.amazon.com/applications?query=aws-event-fork-pipelines) (choose **Show apps that create custom IAM roles or resource policies**) into your AWS account. For more information, see [How AWS Event Fork Pipelines works](sns-fork-pipeline-as-subscriber.md#how-sns-fork-works).
This section show how you can use the AWS Management Console to deploy a pipeline and then subscribe AWS Event Fork Pipelines to an Amazon SNS topic. Before you begin, [create an Amazon SNS topic](sns-create-topic.md).
To delete the resources that comprise a pipeline, find the pipeline on the **Applications** page of on the AWS Lambda console, expand the **SAM template section**, choose **CloudFormation stack**, and then choose **Other Actions**, **Delete Stack**.
# Deploying and subscribing the Event Storage and Backup Pipeline to Amazon SNS
| |
| --- |
| For event archiving and analytics, Amazon SNS now recommends using its native integration with Amazon Data Firehose. You can subscribe Firehose delivery streams to SNS topics, which allows you to send notifications to archiving and analytics endpoints such as Amazon Simple Storage Service (Amazon S3) buckets, Amazon Redshift tables, Amazon OpenSearch Service (OpenSearch Service), and more. Using Amazon SNS with Firehose delivery streams is a fully-managed and codeless solution that doesn't require you to use AWS Lambda functions. For more information, see [Fanout to Firehose delivery streams](sns-firehose-as-subscriber.md). |
This page shows how to deploy the [Event Storage and Backup Pipeline](sns-fork-pipeline-as-subscriber.md#sns-fork-event-storage-and-backup-pipeline) and subscribe it to an Amazon SNS topic. This process automatically turns the AWS SAM template associated with the pipeline into an CloudFormation stack, and then deploys the stack into your AWS account. This process also creates and configures the set of resources that comprise the Event Storage and Backup Pipeline, including the following:
+ Amazon SQS queue
+ Lambda function
+ Firehose delivery stream
+ Amazon S3 backup bucket
For more information about configuring a stream with an Amazon S3 bucket as a destination, see `[S3DestinationConfiguration](https://docs.aws.amazon.com/firehose/latest/APIReference/API_S3DestinationConfiguration.html)` in the *Amazon Data Firehose API Reference*.
For more information about transforming events and about configuring event buffering, event compression, and event encryption, see [Creating an Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html) in the *Amazon Data Firehose Developer Guide*.
For more information about filtering events, see [Amazon SNS subscription filter policies](sns-subscription-filter-policies.md) in this guide.
1. Sign in to the [AWS Lambda console](https://console.aws.amazon.com/lambda/).
1. On the navigation panel, choose **Functions** and then choose **Create function**.
1. On the **Create function** page, do the following:
1. Choose **Browse serverless app repository**, **Public applications**, **Show apps that create custom IAM roles or resource policies**.
1. Search for `fork-event-storage-backup-pipeline` and then choose the application.
1. On the **fork-event-storage-backup-pipeline** page, do the following:
1. In the **Application settings** section, enter an **Application name** (for example, `my-app-backup`).
**Note**
For each deployment, the application name must be unique. If you reuse an application name, the deployment will update only the previously deployed CloudFormation stack (rather than create a new one).
1. (Optional) For **BucketArn**, enter the ARN of the Amazon S3 bucket into which incoming events are loaded. If you don't enter a value, a new Amazon S3 bucket is created in your AWS account.
1. (Optional) For **DataTransformationFunctionArn**, enter the ARN of the Lambda function through which the incoming events are transformed. If you don't enter a value, data transformation is disabled.
1. (Optional) Enter one of the following **LogLevel** settings for the execution of your application's Lambda function:
+ `DEBUG`
+ `ERROR`
+ `INFO` (default)
+ `WARNING`
1. For **TopicArn**, enter the ARN of the Amazon SNS topic to which this instance of the fork pipeline is to be subscribed.
1. (Optional) For **StreamBufferingIntervalInSeconds** and **StreamBufferingSizeInMBs**, enter the values for configuring the buffering of incoming events. If you don't enter any values, 300 seconds and 5 MB are used.
1. (Optional) Enter one of the following **StreamCompressionFormat** settings for compressing incoming events:
+ `GZIP`
+ `SNAPPY`
+ `UNCOMPRESSED` (default)
+ `ZIP`
1. (Optional) For **StreamPrefix**, enter the string prefix to name files stored in the Amazon S3 backup bucket. If you don't enter a value, no prefix is used.
1. (Optional) For **SubscriptionFilterPolicy**, enter the Amazon SNS subscription filter policy, in JSON format, to be used for filtering incoming events. The filter policy decides which events are indexed in the OpenSearch Service index. If you don't enter a value, no filtering is used (all events are indexed).
1. (Optional) For **SubscriptionFilterPolicyScope**, enter the string `MessageBody` or `MessageAttributes` to enable payload-based or attribute-based message filtering.
1. Choose **I acknowledge that this app creates custom IAM roles, resource policies and deploys nested applications.** and then choose **Deploy**.
On the **Deployment status for *my-app*** page, Lambda displays the **Your application is being deployed** status.
In the **Resources** section, CloudFormation begins to create the stack and displays the **CREATE\$1IN\$1PROGRESS** status for each resource. When the process is complete, CloudFormation displays the **CREATE\$1COMPLETE** status.
When the deployment is complete, Lambda displays the **Your application has been deployed** status.
Messages published to your Amazon SNS topic are stored in the Amazon S3 backup bucket provisioned by the Event Storage and Backup pipeline automatically.
# Deploying and subscribing the Event Search and Analytics Pipeline to Amazon SNS
| |
| --- |
| For event archiving and analytics, Amazon SNS now recommends using its native integration with Amazon Data Firehose. You can subscribe Firehose delivery streams to SNS topics, which allows you to send notifications to archiving and analytics endpoints such as Amazon Simple Storage Service (Amazon S3) buckets, Amazon Redshift tables, Amazon OpenSearch Service (OpenSearch Service), and more. Using Amazon SNS with Firehose delivery streams is a fully-managed and codeless solution that doesn't require you to use AWS Lambda functions. For more information, see [Fanout to Firehose delivery streams](sns-firehose-as-subscriber.md). |
This page shows how to deploy the [Event Search and Analytics Pipeline](sns-fork-pipeline-as-subscriber.md#sns-fork-event-search-and-analytics-pipeline) and subscribe it to an Amazon SNS topic. This process automatically turns the AWS SAM template associated with the pipeline into an CloudFormation stack, and then deploys the stack into your AWS account. This process also creates and configures the set of resources that comprise the Event Search and Analytics Pipeline, including the following:
+ Amazon SQS queue
+ Lambda function
+ Firehose delivery stream
+ Amazon OpenSearch Service domain
+ Amazon S3 dead-letter bucket
For more information about configuring a stream with an index as a destination, see `[ElasticsearchDestinationConfiguration](https://docs.aws.amazon.com/firehose/latest/APIReference/API_ElasticsearchDestinationConfiguration.html)` in the *Amazon Data Firehose API Reference*.
For more information about transforming events and about configuring event buffering, event compression, and event encryption, see [Creating an Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html) in the *Amazon Data Firehose Developer Guide*.
For more information about filtering events, see [Amazon SNS subscription filter policies](sns-subscription-filter-policies.md) in this guide.
1. Sign in to the [AWS Lambda console](https://console.aws.amazon.com/lambda/).
1. On the navigation panel, choose **Functions** and then choose **Create function**.
1. On the **Create function** page, do the following:
1. Choose **Browse serverless app repository**, **Public applications**, **Show apps that create custom IAM roles or resource policies**.
1. Search for `fork-event-search-analytics-pipeline` and then choose the application.
1. On the **fork-event-search-analytics-pipeline** page, do the following:
1. In the **Application settings** section, enter an **Application name** (for example, `my-app-search`).
**Note**
For each deployment, the application name must be unique. If you reuse an application name, the deployment will update only the previously deployed CloudFormation stack (rather than create a new one).
1. (Optional) For **DataTransformationFunctionArn**, enter the ARN of the Lambda function used for transforming incoming events. If you don't enter a value, data transformation is disabled.
1. (Optional) Enter one of the following **LogLevel** settings for the execution of your application's Lambda function:
+ `DEBUG`
+ `ERROR`
+ `INFO` (default)
+ `WARNING`
1. (Optional) For **SearchDomainArn**, enter the ARN of the OpenSearch Service domain, a cluster that configures the needed compute and storage functionality. If you don't enter a value, a new domain is created with the default configuration.
1. For **TopicArn**, enter the ARN of the Amazon SNS topic to which this instance of the fork pipeline is to be subscribed.
1. For **SearchIndexName**, enter the name of the OpenSearch Service index for event search and analytics.
**Note**
The following quotas apply to index names:
Can't include uppercase letters
Can't include the following characters: `\ / * ? " < > | ` , #`
Can't begin with the following characters: `- + _`
Can't be the following: `. ..`
Can't be longer than 80 characters
Can't be longer than 255 bytes
Can't contain a colon (from OpenSearch Service 7.0)
1. (Optional) Enter one of the following **SearchIndexRotationPeriod** settings for the rotation period of the OpenSearch Service index:
+ `NoRotation` (default)
+ `OneDay`
+ `OneHour`
+ `OneMonth`
+ `OneWeek`
Index rotation appends a timestamp to the index name, facilitating the expiration of old data.
1. For **SearchTypeName**, enter the name of the OpenSearch Service type for organizing the events in an index.
**Note**
OpenSearch Service type names can contain any character (except null bytes) but can't begin with `_`.
For OpenSearch Service 6.x, there can be only one type per index. If you specify a new type for an existing index that already has another type, Firehose returns a runtime error.
1. (Optional) For **StreamBufferingIntervalInSeconds** and **StreamBufferingSizeInMBs**, enter the values for configuring the buffering of incoming events. If you don't enter any values, 300 seconds and 5 MB are used.
1. (Optional) Enter one of the following **StreamCompressionFormat** settings for compressing incoming events:
+ `GZIP`
+ `SNAPPY`
+ `UNCOMPRESSED` (default)
+ `ZIP`
1. (Optional) For **StreamPrefix**, enter the string prefix to name files stored in the Amazon S3 dead-letter bucket. If you don't enter a value, no prefix is used.
1. (Optional) For **StreamRetryDurationInSecons**, enter the retry duration for cases when Firehose can't index events in the OpenSearch Service index. If you don't enter a value, then 300 seconds is used.
1. (Optional) For **SubscriptionFilterPolicy**, enter the Amazon SNS subscription filter policy, in JSON format, to be used for filtering incoming events. The filter policy decides which events are indexed in the OpenSearch Service index. If you don't enter a value, no filtering is used (all events are indexed).
1. Choose **I acknowledge that this app creates custom IAM roles, resource policies and deploys nested applications.** and then choose **Deploy**.
On the **Deployment status for *my-app-search*** page, Lambda displays the **Your application is being deployed** status.
In the **Resources** section, CloudFormation begins to create the stack and displays the **CREATE\$1IN\$1PROGRESS** status for each resource. When the process is complete, CloudFormation displays the **CREATE\$1COMPLETE** status.
When the deployment is complete, Lambda displays the **Your application has been deployed** status.
Messages published to your Amazon SNS topic are indexed in the OpenSearch Service index provisioned by the Event Search and Analytics pipeline automatically. If the pipeline can't index an event, it stores it in a Amazon S3 dead-letter bucket.
# Deploying the Event Replay Pipeline with Amazon SNS integration
This page shows how to deploy the [Event Replay Pipeline](sns-fork-pipeline-as-subscriber.md#sns-fork-event-replay-pipeline) and subscribe it to an Amazon SNS topic. This process automatically turns the AWS SAM template associated with the pipeline into an CloudFormation stack, and then deploys the stack into your AWS account. This process also creates and configures the set of resources that comprise the Event Replay Pipeline, including an Amazon SQS queue and a Lambda function.
For more information about filtering events, see [Amazon SNS subscription filter policies](sns-subscription-filter-policies.md) in this guide.
1. Sign in to the [AWS Lambda console](https://console.aws.amazon.com/lambda/).
1. On the navigation panel, choose **Functions** and then choose **Create function**.
1. On the **Create function** page, do the following:
1. Choose **Browse serverless app repository**, **Public applications**, **Show apps that create custom IAM roles or resource policies**.
1. Search for `fork-event-replay-pipeline` and then choose the application.
1. On the **fork-event-replay-pipeline** page, do the following:
1. In the **Application settings** section, enter an **Application name** (for example, `my-app-replay`).
**Note**
For each deployment, the application name must be unique. If you reuse an application name, the deployment will update only the previously deployed CloudFormation stack (rather than create a new one).
1. (Optional) Enter one of the following **LogLevel** settings for the execution of your application's Lambda function:
+ `DEBUG`
+ `ERROR`
+ `INFO` (default)
+ `WARNING`
1. (Optional) For **ReplayQueueRetentionPeriodInSeconds**, enter the amount of time, in seconds, for which the Amazon SQS replay queue keeps the message. If you don't enter a value, 1,209,600 seconds (14 days) is used.
1. For **TopicArn**, enter the ARN of the Amazon SNS topic to which this instance of the fork pipeline is to be subscribed.
1. For **DestinationQueueName**, enter the name of the Amazon SQS queue to which the Lambda replay function forwards messages.
1. (Optional) For **SubscriptionFilterPolicy**, enter the Amazon SNS subscription filter policy, in JSON format, to be used for filtering incoming events. The filter policy decides which events are buffered for replay. If you don't enter a value, no filtering is used (all events are buffered for replay).
1. Choose **I acknowledge that this app creates custom IAM roles, resource policies and deploys nested applications.** and then choose **Deploy**.
On the **Deployment status for *my-app-replay*** page, Lambda displays the **Your application is being deployed** status.
In the **Resources** section, CloudFormation begins to create the stack and displays the **CREATE\$1IN\$1PROGRESS** status for each resource. When the process is complete, CloudFormation displays the **CREATE\$1COMPLETE** status.
When the deployment is complete, Lambda displays the **Your application has been deployed** status.
Messages published to your Amazon SNS topic are buffered for replay in the Amazon SQS queue provisioned by the Event Replay Pipeline automatically.
**Note**
By default, replay is disabled. To enable replay, navigate to the function's page on the Lambda console, expand the **Designer** section, choose the **SQS** tile and then, in the **SQS** section, choose **Enabled**.
# Using Amazon EventBridge Scheduler with Amazon SNS
[Amazon EventBridge Scheduler](https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html) is a serverless scheduler that allows you to create, run, and manage tasks from one central, managed service. With EventBridge Scheduler, you can create schedules using Cron and rate expressions for recurring patterns, or configure one-time invocations. You can set up flexible time windows for delivery, define retry limits, and set the maximum retention time for failed API invocations.
This page explains how to use EventBridge Scheduler to publish a message from an Amazon SNS topic on a schedule.
## Setting-up the execution role
When you create a new schedule, EventBridge Scheduler must have permission to invoke its target API operation on your behalf. You grant these permissions to EventBridge Scheduler using an *execution role*. The permission policy you attach to your schedule's execution role defines the required permissions. These permissions depend on the target API you want EventBridge Scheduler to invoke.
When you use the EventBridge Scheduler console to create a schedule, as in the following procedure, EventBridge Scheduler automatically sets up an execution role based on your selected target. If you want to create a schedule using one of the EventBridge Scheduler SDKs, the AWS CLI, or CloudFormation, you must have an existing execution role that grants the permissions EventBridge Scheduler requires to invoke a target. For more information about manually setting-up an execution role for your schedule, see [Setting-up an execution role](https://docs.aws.amazon.com/scheduler/latest/UserGuide/setting-up.html#setting-up-execution-role) in the *EventBridge Scheduler User Guide*.
## Create a schedule
**To create a schedule by using the console**
1. Open the Amazon EventBridge Scheduler console at [https://console.aws.amazon.com/scheduler/home](https://console.aws.amazon.com/scheduler/home/).
1. On the **Schedules** page, choose **Create schedule**.
1. On the **Specify schedule detail** page, in the **Schedule name and description** section, do the following:
1. For **Schedule name**, enter a name for your schedule. For example, **MyTestSchedule**.
1. (Optional) For **Description**, enter a description for your schedule. For example, **My first schedule**.
1. For **Schedule group**, choose a schedule group from the dropdown list. If you don't have a group, choose **default**. To create a schedule group, choose **create your own schedule**.
You use schedule groups to add tags to groups of schedules.
1.
1. Choose your schedule options.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sns/latest/dg/using-eventbridge-scheduler.html)
1. (Optional) If you chose **Recurring schedule** in the previous step, in the **Timeframe** section, do the following:
1. For **Timezone**, choose a timezone.
1. For **Start date and time**, enter a valid date in `YYYY/MM/DD` format, and then specify a timestamp in 24-hour `hh:mm` format.
1. For **End date and time**, enter a valid date in `YYYY/MM/DD` format, and then specify a timestamp in 24-hour `hh:mm` format.
1. Choose **Next**.
1. On the **Select target** page, choose the AWS API operation that EventBridge Scheduler invokes:
1. Choose **Amazon SNS Publish**.
1. In the **Publish** section, select an SNS topic or choose **Create new SNS topic**.
1. (Optional) Enter a JSON payload. If you don't enter a payload, EventBridge Scheduler uses an empty event to invoke the function.
1. Choose **Next**.
1. On the **Settings** page, do the following:
1. To turn on the schedule, under **Schedule state**, toggle **Enable schedule**.
1. To configure a retry policy for your schedule, under **Retry policy and dead-letter queue (DLQ)**, do the following:
+ Toggle **Retry**.
+ For **Maximum age of event**, enter the maximum **hour(s)** and **min(s)** that EventBridge Scheduler must keep an unprocessed event.
+ The maximum time is 24 hours.
+ For **Maximum retries**, enter the maximum number of times EventBridge Scheduler retries the schedule if the target returns an error.
The maximum value is 185 retries.
With retry policies, if a schedule fails to invoke its target, EventBridge Scheduler re-runs the schedule. If configured, you must set the maximum retention time and retries for the schedule.
1. Choose where EventBridge Scheduler stores undelivered events.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sns/latest/dg/using-eventbridge-scheduler.html)
1. To use a customer managed key to encrypt your target input, under **Encryption**, choose **Customize encryption settings (advanced)**.
If you choose this option, enter an existing KMS key ARN or choose **Create an AWS KMS key** to navigate to the AWS KMS console. For more information about how EventBridge Scheduler encrypts your data at rest, see [Encryption at rest](https://docs.aws.amazon.com/scheduler/latest/UserGuide/encryption-rest.html) in the *Amazon EventBridge Scheduler User Guide*.
1. To have EventBridge Scheduler create a new execution role for you, choose **Create new role for this schedule**. Then, enter a name for **Role name**. If you choose this option, EventBridge Scheduler attaches the required permissions necessary for your templated target to the role.
1. Choose **Next**.
1. In the **Review and create schedule** page, review the details of your schedule. In each section, choose **Edit** to go back to that step and edit its details.
1. Choose **Create schedule**.
You can view a list of your new and existing schedules on the **Schedules** page. Under the **Status** column, verify that your new schedule is **Enabled**.
## Related resources
For more information about EventBridge Scheduler, see the following:
+ [EventBridge Scheduler User Guide](https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html)
+ [EventBridge Scheduler API Reference](https://docs.aws.amazon.com/scheduler/latest/APIReference/Welcome.html)
+ [EventBridge Scheduler Pricing](https://aws.amazon.com/eventbridge/pricing/#Scheduler)