AWS SQS

Amazon SQS

SQS is a managed queue. A producer writes a message, a consumer reads it, and the two never need to be available at the same time, which is the whole point. A traffic spike becomes a longer queue rather than a stack of failed requests, and a consumer you take down for deployment simply resumes where it left off.

Most of the trouble people have with SQS comes down to three things: choosing Standard when they needed FIFO, setting a visibility timeout shorter than their processing time, and writing consumers that break when a message arrives twice. All three are covered below.

Standard or FIFO

aws sqs create-queue --queue-name orders
aws sqs create-queue --queue-name orders.fifo \
  --attributes FifoQueue=true,ContentBasedDeduplication=true

A FIFO queue name must end in .fifo, and you cannot convert a queue from one type to the other. The choice is permanent for the life of the queue.

Default to Standard. Its throughput is unbounded and its ordering is good enough for most work. An image to resize or an email to send does not care about sequence. Choose FIFO only when out-of-order processing is genuinely incorrect, such as applying a sequence of updates to the same record.

FIFO ordering is per message group, not per queue, and this is the part that gets missed:

aws sqs send-message --queue-url "$Q" \
  --message-body '{"event":"updated"}' \
  --message-group-id "customer-4821"

Messages sharing a group id are strictly ordered and processed one at a time; different groups proceed in parallel. Using a single group id for everything serialises the entire queue and throws away your concurrency, which is the usual cause of "FIFO is too slow".

Visibility timeout

When a consumer receives a message, SQS hides it from other consumers for the visibility timeout rather than deleting it. The consumer must delete it explicitly once the work is done:

aws sqs receive-message --queue-url "$Q" --max-number-of-messages 1

{
  "Messages": [
    {
      "MessageId": "8f3c1e2a-4b7d-11ef-9c2a-0242ac120002",
      "ReceiptHandle": "AQEBzbVv...",
      "Body": "{\"orderId\":\"A-4821\"}"
    }
  ]
}

aws sqs delete-message --queue-url "$Q" --receipt-handle "AQEBzbVv..."

That design is what makes SQS reliable: a consumer that crashes mid-work never deletes the message, so it reappears after the timeout and someone else picks it up.

It is also the most common misconfiguration. The default visibility timeout is 30 seconds. If your handler takes 45, the message becomes visible again while you are still working on it, a second consumer starts the same job, and both finish, so the work happens twice and you may see it a third time. The symptom is duplicate processing that no amount of Standard-queue reasoning explains.

aws sqs set-queue-attributes --queue-url "$Q" \
  --attributes VisibilityTimeout=300

Set it to comfortably more than your worst-case processing time. For long or variable jobs, extend the timeout while working instead of guessing high:

aws sqs change-message-visibility --queue-url "$Q" \
  --receipt-handle "$RH" --visibility-timeout 600

With a Lambda trigger, AWS requires the queue's visibility timeout to be at least six times the function timeout, which accounts for retries within a batch.

Long polling

aws sqs set-queue-attributes --queue-url "$Q" \
  --attributes ReceiveMessageWaitTimeSeconds=20

The default of 0 is short polling: every receive-message returns immediately, often empty, and you pay for the request. A consumer loop doing this spends most of its budget asking an empty queue whether anything has arrived.

Long polling holds the connection open for up to 20 seconds and returns as soon as a message appears. It cuts both request cost and latency, and there is essentially no reason not to set it. Note that short polling also samples only a subset of servers, so it can return empty even when messages exist.

Write idempotent consumers

Standard queues guarantee at-least-once delivery. Duplicates are not an edge case to be prevented. They are documented behaviour, and they also arise from visibility timeouts, retries and network failures on FIFO queues.

The practical requirement is that processing a message twice has the same effect as processing it once. Usually that means a unique key and a conditional write:

try:
    table.put_item(
        Item={"orderId": order_id, "status": "processed"},
        ConditionExpression="attribute_not_exists(orderId)",
    )
except ClientError as e:
    if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
        return          # already handled; delete the message and move on
    raise

"We will just make sure it never delivers twice" is not an available option.

Dead-letter queues

A message that keeps failing will keep being retried forever, consuming capacity and often blocking a FIFO message group entirely. A redrive policy moves it aside after a set number of attempts:

aws sqs set-queue-attributes --queue-url "$Q" --attributes '{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:123456789012:orders-dlq\",\"maxReceiveCount\":\"5\"}"
}'

The DLQ must be the same type as the source queue. FIFO source, FIFO DLQ. A DLQ nobody watches is worse than none at all, because failures are now silent; alarm on its depth from the moment you create it. The dead-letter queues page covers redrive and replay in detail.

What to monitor

aws sqs get-queue-attributes --queue-url "$Q" \
  --attribute-names ApproximateNumberOfMessages \
  ApproximateNumberOfMessagesNotVisible ApproximateAgeOfOldestMessage

Queue depth alone is a poor alarm, because a healthy queue can be deep during a burst and drain fine. Age of the oldest message is the honest signal: if it climbs steadily, consumers cannot keep up. Alarm on DLQ depth above zero separately.

Common errors

The receipt handle "AQEBzb..." is not valid for the current visibility timeout

The visibility timeout expired before you called delete-message, so the handle is stale. The work already ran twice. Raise the timeout.

InvalidParameterValue: Value ... for parameter MessageGroupId is invalid

You sent a message to a FIFO queue without a MessageGroupId, which is mandatory there and meaningless on Standard.

The queue name must end with .fifo

FIFO queues are identified by the suffix, and there is no attribute-only way to create one.

Quick reference

---

Key takeaways

• Default to Standard. Use FIFO only when out-of-order processing is genuinely wrong.
• FIFO ordering is per message group: one group id for everything serialises the whole queue.
• Visibility timeout must exceed your worst-case processing time, and the 30-second default rarely does.
• Set ReceiveMessageWaitTimeSeconds=20. Short polling costs more and adds latency.
• Consumers must be idempotent. At-least-once delivery is a guarantee, not a bug.
• Attach a DLQ and alarm on it, or failures become invisible rather than absent.
• Alarm on age of the oldest message, not queue depth. Depth is normal during a burst.

---

Check your understanding

---

Frequently Asked Questions

What is the difference between SQS and SNS?

SQS is a queue: a message is stored until one consumer reads and deletes it, which suits distributing work among workers. SNS is publish-subscribe: a message is pushed immediately to every subscriber and not retained for a subscriber that is absent, which suits fan-out notification. They are frequently combined. SNS publishes to several SQS queues, giving each consumer its own durable copy, and that fan-out pattern is the standard way to let multiple services react to the same event without coupling them.

Should I use a Standard or FIFO queue?

Standard unless out-of-order processing would actually be wrong. Standard gives effectively unlimited throughput and best-effort ordering, which is fine for independent jobs like resizing an image or sending an email. FIFO caps at 300 messages per second (3,000 with batching) and guarantees ordering within a message group, so it earns its place for sequences of updates to the same entity. The choice is permanent: a queue cannot be converted, and FIFO names must end in .fifo.

Why is my SQS message processed more than once?

Two causes, and both are normal behaviour rather than faults. Standard queues guarantee at-least-once delivery, so occasional duplicates are documented. Far more often, though, the visibility timeout is shorter than the processing time, with the 30-second default and a 45-second handler, the message reappears mid-flight and a second consumer starts the same work. Raise VisibilityTimeout above your worst case, and make the consumer idempotent with a conditional write on a unique key so a duplicate is harmless.

What should the SQS visibility timeout be set to?

Comfortably longer than your worst-case processing time, if p99 handling takes 40 seconds, 120 is a reasonable starting point rather than 45. For jobs with unpredictable duration, call change-message-visibility periodically while working instead of setting one very high value, which would delay retries after a genuine crash. When the consumer is a Lambda function, AWS requires the queue timeout to be at least six times the function timeout to allow for in-batch retries.

How does an SQS dead-letter queue work?

You attach a redrive policy naming a second queue and a maxReceiveCount. Each time a message is received without being deleted, its receive count increases, and once it exceeds the limit SQS moves it to the DLQ instead of making it visible again. That stops one broken message consuming capacity indefinitely or blocking a FIFO group. The DLQ must match the source queue's type, and it needs an alarm on depth above zero. Otherwise you have converted loud failures into silent ones.

Why does my consumer keep returning no messages when the queue is not empty?

Most likely short polling, which is the default. ReceiveMessageWaitTimeSeconds=0 samples a subset of SQS's servers and returns immediately, so it can legitimately come back empty while messages exist elsewhere in the queue. Setting it to 20 enables long polling, which queries all servers and waits for a message to arrive. Lower latency and fewer billable requests. The other explanation is that the messages are in flight rather than available: compare ApproximateNumberOfMessagesVisible with ApproximateNumberOfMessagesNotVisible.