# Amazon SQS best practices Best practices Amazon SQS manages and processes message queues, enabling different parts of an application to exchange messages reliably and at scale. This topic covers key operational best practices, including using long polling to reduce empty responses, implementing dead-letter queues to handle processing errors, and optimizing queue permissions for security. ****Topics**** + [Error handling and problematic messages](best-practices-error-handling.md) + [Message deduplication and grouping](best-practices-message-deduplication.md) + [Message processing and timing](best-practices-message-processing.md) # Amazon SQS error handling and problematic messages Error handling and problematic messages This topic provides detailed instructions on managing and mitigating errors in Amazon SQS, including techniques for handling request errors, capturing problematic messages, and configuring dead-letter queue retention to ensure message reliability. ****Topics**** + [ # Handling request errors in Amazon SQS ](handling-request-errors.md) + [ # Capturing problematic messages in Amazon SQS ](capturing-problematic-messages.md) + [ # Setting-up dead-letter queue retention in Amazon SQS ](setting-up-dead-letter-queue-retention.md) # Handling request errors in Amazon SQS To handle request errors, use one of the following strategies: + If you use an AWS SDK, you already have automatic *retry and backoff* logic at your disposal. For more information, see [Error Retries and Exponential Backoff in AWS](https://docs.aws.amazon.com/general/latest/gr/api-retries.html) in the *Amazon Web Services General Reference*. + If you don't use the AWS SDK features for retry and backoff, allow a pause (for example, 200 ms) before retrying the [ReceiveMessage](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html) action after receiving no messages, a timeout, or an error message from Amazon SQS. For subsequent use of `ReceiveMessage` that gives the same results, allow a longer pause (for example, 400 ms). # Capturing problematic messages in Amazon SQS To capture all messages that can't be processed, and to collect accurate CloudWatch metrics, configure a [dead-letter queue](sqs-dead-letter-queues.md). + The redrive policy redirects messages to a dead-letter queue after the source queue fails to process a message a specified number of times. + Using a dead-letter queue decreases the number of messages and reduces the possibility of exposing you to *poison pill* messages (messages that are received but can't be processed). + Including a poison pill message in a queue can distort the [`ApproximateAgeOfOldestMessage`](sqs-available-cloudwatch-metrics.md) CloudWatch metric by giving an incorrect age of the poison pill message. Configuring a dead-letter queue helps avoid false alarms when using this metric. # Setting-up dead-letter queue retention in Amazon SQS For standard queues, the expiration of a message is always based on its original enqueue timestamp. When a message is moved to a dead-letter queue, the enqueue timestamp is unchanged. The `ApproximateAgeOfOldestMessage` metric indicates when the message moved to the dead-letter queue, *not* when the message was originally sent. For example, assume that a message spends 1 day in the original queue before it's moved to a dead-letter queue. If the dead-letter queue's retention period is 4 days, the message is deleted from the dead-letter queue after 3 days and the `ApproximateAgeOfOldestMessage` is 3 days. Thus, it is a best practice to always set the retention period of a dead-letter queue to be longer than the retention period of the original queue. For FIFO queues, the enqueue timestamp resets when the message is moved to a dead-letter queue. The `ApproximateAgeOfOldestMessage` metric indicates when the message moved to the dead-letter queue. In the same example above, the message is deleted from the dead-letter queue after 4 days and the `ApproximateAgeOfOldestMessage` is 4 days. # Amazon SQS message deduplication and grouping Message deduplication and grouping This topic provides best practices for ensuring consistent message processing in Amazon SQS. It explains how to use: + [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html#API_SendMessage_RequestSyntax](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html#API_SendMessage_RequestSyntax) to prevent duplicate messages in FIFO queues. + [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) to manage message ordering within distinct message groups. ****Topics**** + [ # Avoiding inconsistent message processing in Amazon SQS ](avoiding-inconsistent-message-processing.md) + [Using the message deduplication ID](using-messagededuplicationid-property.md) + [Using the message group ID](using-messagegroupid-property.md) + [Using the receive request attempt ID](using-receiverequestattemptid-request-parameter.md) # Avoiding inconsistent message processing in Amazon SQS Because Amazon SQS is a distributed system, it is possible for a consumer to not receive a message even when Amazon SQS marks the message as delivered while returning successfully from a [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html) API method call. In this case, Amazon SQS records the message as delivered at least once, although the consumer has never received it. Because no additional attempts to deliver messages are made under these conditions, we don't recommend setting the number of maximum receives to 1 for a [dead-letter queue](sqs-dead-letter-queues.md). # Using the message deduplication ID in Amazon SQS Using the message deduplication ID [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) is a token used only in Amazon SQS FIFO queues to prevent duplicate message delivery. It ensures that within a 5-minute deduplication window, only one instance of a message with the same deduplication ID is processed and delivered. If Amazon SQS has already accepted a message with a specific deduplication ID, any subsequent messages with the same ID will be acknowledged but not delivered to consumers. **Note** Amazon SQS continues tracking the deduplication ID even after the message has been received and deleted. **Topics** + [ # When to provide a message deduplication ID in Amazon SQS ](providing-message-deduplication-id.md) + [ # Enabling deduplication for a single-producer/consumer system in Amazon SQS ](single-producer-single-consumer.md) + [ # Outage recovery scenarios in Amazon SQS ](designing-for-outage-recovery-scenarios.md) + [ # Configuring visibility timeouts in Amazon SQS ](working-with-visibility-timeouts.md) # When to provide a message deduplication ID in Amazon SQS A producer should specify a message deduplication ID in the following scenarios: + When sending identical message bodies that must be treated as unique. + When sending messages with the same content but different message attributes, ensuring each message is processed separately. + When sending messages with different content (for example, a retry counter in the message body) but requiring Amazon SQS to recognize them as duplicates. # Enabling deduplication for a single-producer/consumer system in Amazon SQS If you have a single producer and a single consumer, and messages are unique because they include an application-specific message ID in the body, follow these best practices: + Enable content-based deduplication for the queue (each of your messages has a unique body). The producer can omit the message deduplication ID. + When content-based deduplication is enabled for an Amazon SQS FIFO queue, and a message is sent with a deduplication ID, the [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) deduplication ID overrides the generated content-based deduplication ID. + Although the consumer isn't required to provide a receive request attempt ID for each request, it's a best practice because it allows fail-retry sequences to execute faster. + You can retry send or receive requests because they don't interfere with the ordering of messages in FIFO queues. # Outage recovery scenarios in Amazon SQS The deduplication process in FIFO queues is time-sensitive. When designing your application, ensure that both the producer and consumer can recover from client or network outages without introducing duplicates or processing failures. Producer considerations + Amazon SQS enforces a 5-minute deduplication window. + If a producer retries a [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) request after 5-minutes, Amazon SQS treats it as a new message, potentially creating duplicates. Consumer considerations + If a consumer fails to process a message before the visibility timeout expires, another consumer may receive and process it, leading to duplicate processing. + Adjust the visibility timeout based on your application's processing time. + Use the [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html) API to extend the timeout while a message is still being processed. + If a message repeatedly fails to process, route it to a [dead-letter queue (DLQ)](sqs-dead-letter-queues.md) instead of allowing it to be reprocessed indefinitely. + The producer must be aware of the deduplication interval of the queue. Amazon SQS has a deduplication interval of 5 minutes. Retrying `SendMessage` requests after the deduplication interval expires can introduce duplicate messages into the queue. For example, a mobile device in a car sends messages whose order is important. If the car loses cellular connectivity for a period of time before receiving an acknowledgement, retrying the request after regaining cellular connectivity can create a duplicate. + The consumer must have a visibility timeout that minimizes the risk of being unable to process messages before the visibility timeout expires. You can extend the visibility timeout while the messages are being processed by calling the `ChangeMessageVisibility` action. However, if the visibility timeout expires, another consumer can immediately begin to process the messages, causing a message to be processed multiple times. To avoid this scenario, configure a [dead-letter queue](sqs-dead-letter-queues.md). # Configuring visibility timeouts in Amazon SQS To ensure reliable message processing, set the visibility timeout to be longer than the AWS SDK read timeout. This applies when using the [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html) API with both short polling and long polling. A longer visibility timeout prevents messages from becoming available to other consumers before the original request completes, reducing the risk of duplicate processing. # Using the message group ID with Amazon SQS FIFO Queues Using the message group ID In FIFO (First-In-First-Out) queues, [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) is an attribute that organizes messages into distinct groups. Messages within the same message group are always processed one at a time, in strict order, ensuring that no two messages from the same group are processed simultaneously. In standard queues, using `MessageGroupId` enables [fair queues](sqs-fair-queues.md). If strict ordering is required, use a FIFO queue. **Topics** + [ # Interleaving multiple ordered message groups in Amazon SQS ](interleaving-multiple-ordered-message-groups.md) + [ # Preventing duplicate processing in a multiple-producer/consumer system in Amazon SQS ](avoding-processing-duplicates-in-multiple-producer-consumer-system.md) + [ # Avoid large message backlogs with the same message group ID in Amazon SQS ](avoid-backlog-with-the-same-message-group-id.md) + [ # Avoid reusing the same message group ID with virtual queues in Amazon SQS ](avoiding-reusing-message-group-id-with-virtual-queues.md) # Interleaving multiple ordered message groups in Amazon SQS To interleave multiple ordered message groups within a single FIFO queue, assign a [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) to each group (for example, session data for different users). This allows multiple consumers to read from the queue simultaneously while ensuring that messages within the same group are processed in order. When a message with a specific `MessageGroupId` is being processed and is invisible, no other consumer can process messages from that same group until the visibility timeout expires or the message is deleted. # Preventing duplicate processing in a multiple-producer/consumer system in Amazon SQS In a high-throughput, low-latency system where message ordering is not a priority, producers can assign a unique [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) to each message. This ensures that Amazon SQS FIFO queues eliminate duplicates, even in a multiple-producer/multiple-consumer setup. While this approach prevents duplicate messages, it does not guarantee message ordering since each message is treated as its own independent group. In any system with multiple producers and consumers, there is always a risk of duplicate delivery. If a consumer fails to process a message before the visibility timeout expires, Amazon SQS makes the message available again, potentially allowing another consumer to pick it up. To mitigate this, ensure proper message acknowledgment and visibility timeout settings based on processing time. # Avoid large message backlogs with the same message group ID in Amazon SQS FIFO queues support a maximum of 120,000 in-flight messages (messages received by a consumer but not yet deleted). If this limit is reached, Amazon SQS does not return an error, but processing may be impacted. You can request an increase beyond this limit by contacting [AWS Support](https://docs.aws.amazon.com/awssupport/latest/user/create-service-quota-increase.html). FIFO queues scan the first 120,000 messages to determine available message groups. If a large backlog builds up in a single message group, messages from other groups sent later will remain blocked until the backlog is processed. **Note** A message backlog can occur when a consumer repeatedly fails to process a message. This could be due to message content issues or consumer-side failures. To prevent message processing delays, configure a [dead-letter queue](sqs-dead-letter-queues.md) to move unprocessed messages after multiple failed attempts. This ensures that other messages in the same message group can be processed, preventing system bottlenecks. # Avoid reusing the same message group ID with virtual queues in Amazon SQS When using virtual queues with a shared host queue, avoid reusing the same [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) across different virtual queues. If multiple virtual queues share the same host queue and contain messages with the same `MessageGroupId`, those messages can block each other, preventing efficient processing. To ensure smooth message processing, assign unique `MessageGroupId` values for messages in different virtual queues. # Using the Amazon SQS receive request attempt ID Using the receive request attempt ID The receive request attempt ID is a unique token used to deduplicate [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html) calls in Amazon SQS. During a network outage or connectivity issue between your application and Amazon SQS, it is best practice to: + Provide a receive request attempt ID when making a `ReceiveMessage` call. + Retry using the same receive request attempt ID if the operation fails. # Amazon SQS message processing and timing Message processing and timing This topic provides a comprehensive guidance on optimizing the speed and efficiency of message handling in Amazon SQS, including strategies for timely message processing, selecting the best polling mode, and configuring long polling for improved performance. ****Topics**** + [ # Processing messages in a timely manner in Amazon SQS ](best-practices-processing-messages-timely-manner.md) + [ # Setting-up long polling in Amazon SQS ](best-practices-setting-up-long-polling.md) + [ # Using the appropriate polling mode in Amazon SQS ](best-practices-using-appropriate-polling-mode.md) # Processing messages in a timely manner in Amazon SQS Setting the visibility timeout depends on how long it takes your application to process and delete a message. For example, if your application requires 10 seconds to process a message and you set the visibility timeout to 15 minutes, you must wait for a relatively long time to attempt to process the message again if the previous processing attempt fails. Alternatively, if your application requires 10 seconds to process a message but you set the visibility timeout to only 2 seconds, a duplicate message is received by another consumer while the original consumer is still working on the message. To make sure that there is sufficient time to process messages, use one of the following strategies: + If you know (or can reasonably estimate) how long it takes to process a message, extend the message's *visibility timeout* to the maximum time it takes to process and delete the message. For more information, see [Configuring the Visibility Timeout](sqs-visibility-timeout.md#configuring-visibility-timeout). + If you don't know how long it takes to process a message, create a *heartbeat* for your consumer process: Specify the initial visibility timeout (for example, 2 minutes) and then—as long as your consumer still works on the message—keep extending the visibility timeout by 2 minutes every minute. **Important** The maximum visibility timeout is 12 hours from the time that Amazon SQS receives the `ReceiveMessage` request. Extending the visibility timeout does not reset the 12 hour maximum. Additionally, you may be unable to set the timeout on an individual message to the full 12 hours (e.g. 43,200 seconds) since the `ReceiveMessage` request initiates the timer. For example, if you receive a message and immediately set the 12 hour maximum by sending a `ChangeMessageVisibility` call with `VisibilityTimeout` equal to 43,200 seconds, it will likely fail. However, using a value of 43,195 seconds will work unless there is a significant delay between requesting the message via `ReceiveMessage` and updating the visibility timeout. If your consumer needs longer than 12 hours, consider using Step Functions. # Setting-up long polling in Amazon SQS When the wait time for the `[ReceiveMessage](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html)` API action is greater than 0, *long polling* is in effect. The maximum long polling wait time is 20 seconds. Long polling helps reduce the cost of using Amazon SQS by reducing the number of empty responses (when there are no messages available for a `ReceiveMessage` request) and false empty responses (when messages are available but aren't included in a response). For more information, see [Amazon SQS short and long polling](sqs-short-and-long-polling.md). For optimal message processing, use the following strategies: + In most cases, you can set the `ReceiveMessage` wait time to 20 seconds. If 20 seconds is too long for your application, set a shorter `ReceiveMessage` wait time (1 second minimum). If you don't use an AWS SDK to access Amazon SQS, or if you configure an AWS SDK to have a shorter wait time, you might have to modify your Amazon SQS client to either allow longer requests or use a shorter wait time for long polling. + If you implement long polling for multiple queues, use one thread for each queue instead of a single thread for all queues. Using a single thread for each queue allows your application to process the messages in each of the queues as they become available, while using a single thread for polling multiple queues might cause your application to become unable to process messages available in other queues while the application waits (up to 20 seconds) for the queue which doesn't have any available messages. **Important** To avoid HTTP errors, make sure that the HTTP response timeout for `ReceiveMessage` requests is longer than the `WaitTimeSeconds` parameter. For more information, see [ReceiveMessage](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html). # Using the appropriate polling mode in Amazon SQS + Long polling lets you consume messages from your Amazon SQS queue as soon as they become available. + To reduce the cost of using Amazon SQS and to decrease the number of empty receives to an empty queue (responses to the `ReceiveMessage` action which return no messages), enable long polling. For more information, see [Amazon SQS Long Polling](sqs-short-and-long-polling.md). + To increase efficiency when polling for multiple threads with multiple receives, decrease the number of threads. + Long polling is preferable over short polling in most cases. + Short polling returns responses immediately, even if the polled Amazon SQS queue is empty. + To satisfy the requirements of an application that expects immediate responses to the `ReceiveMessage` request, use short polling. + Short polling is billed the same as long polling.