FIFO and queues inside AWS and Azure

Nowadays, most of the systems that are running inside a cloud provider like AWS or Microsoft Azure are using in one way or another a messaging system. Many people forget that not all the time a queue guarantee FIFO.

Things are becoming more sophisticated, at the moment in time when you integrate the queue with other systems. You might realise that even if your queue is supporting FIFO, the integration with other cloud services will not enable you to use queue with FIFO guarantee. Because of this, I decided to write this short blog post that highlights things that might go wrong when you need a FIFO inside AWS or Microsoft Azure and how you could tackle them.

AWS and FIFO

At this moment in time (January 2020) the AWS services that manage messages are:

• AWS SQS
• AWS MQ
• AWS SNS (simple event/message routing solution)

NOTE: We don’t take into consideration AWS Kinesis and AWS IoT Message Broker because they are more specific for events handling and IoT solutions.

From the above list, AWS SQS has the option to have FIFO SQS, and AWS MQ is a full message broker solution, where FIFO is ‘by design’. Be careful with AWS SQS because the default configuration is not with FIFO guarantee.

There is another surprise for AWS FIFO SQS if you want to use it in combination with AWS SNS. You cannot send messages from AWS SNS to FIFO SQS. Yes, this is true. At this moment in time, we don’t have support for it. So, if you need a simple message broker and you hopped that you can use AWS SNS and AWS FIFO SQS, well – you can’t. Just use AWS MQ from the beginning for something like this. The cause of this lack of integration can be from the communication protocol (HTTP/S). By default, the protocol cannot guaranty us that if we send to the same endpoint 3 different requests, they will arrive and be consumed in the same order.

Microsoft Azure and FIFO

In January 2020, on Azure, we can find the following services that can manage our messages:

• Azure Queue Storage
• Azure Service Bus

NOTE: Event Hub is not on the list because of the primary purpose of it is to handle events and not messages. The Azure equivalent for AWS SNS is Azure Event Grid, but inside Azure, in general, you don’t need to use similar service with SNS when you handle messages because of the strong integration (connectors) between services.

Things are more inside Microsoft Azure. First of all, Azure Queue Storage does not have support for FIFO. Simple like this, if you need to guarantee the message order than Azure Queue Storage is not for you. Azure Service Bus is the full ESB offer from Azure that comes with strong support for FIFO.

Beside canonical services from both providers, you can find other 3^rd parties that offering messaging solutions that can run on top of AWS EC2 / Azure VMs or inside AWS ECS / Azure ACI. The real challenge when you need ordering of the messages to be guarantee is how you write the consumer. In the next post, we will talk about this.
Part 2 -  ow to guarantee the order in which the messages are processed inside AWS MQ and Azure Service Bus (FIFO on the consumer)