Nowadays, most of the systems that are running inside a
cloud provider like AWS or Microsoft Azure are using in one way or another a
messaging system. Many people forget that not all the time a queue guarantee FIFO.
Things are becoming more sophisticated, at the moment in time when
you integrate the queue with other systems. You might realise that even if your
queue is supporting FIFO, the integration with other cloud services will not
enable you to use queue with FIFO guarantee. Because of this, I decided to
write this short blog post that highlights things that might go wrong when you
need a FIFO inside AWS or Microsoft Azure and how you could tackle them.
AWS and FIFO
At this moment in time (January 2020) the AWS services that manage
messages are:
- AWS SQS
- AWS MQ
- AWS SNS (simple event/message routing solution)
NOTE: We don’t take into consideration AWS Kinesis and AWS
IoT Message Broker because they are more specific for events handling and IoT
solutions.
From the above list, AWS SQS has the option to have FIFO SQS, and AWS MQ is a full message broker solution, where FIFO is ‘by design’. Be careful
with AWS SQS because the default configuration is not with FIFO guarantee.
There is another surprise for AWS FIFO SQS if you want to
use it in combination with AWS SNS. You cannot send messages from AWS SNS to FIFO
SQS. Yes, this is true. At this moment in time, we don’t have support for it.
So, if you need a simple message broker and you hopped that you can use AWS SNS
and AWS FIFO SQS, well – you can’t. Just use AWS MQ from the beginning for something
like this. The cause of this lack of integration can be from the communication
protocol (HTTP/S). By default, the protocol cannot guaranty us that if we send
to the same endpoint 3 different requests, they will arrive and be consumed in
the same order.
Microsoft Azure and FIFO
In January 2020, on Azure, we can find the following services
that can manage our messages:
- Azure Queue Storage
- Azure Service Bus
NOTE: Event Hub is not on the list because of the primary purpose
of it is to handle events and not messages. The Azure equivalent for AWS SNS is
Azure Event Grid, but inside Azure, in general, you don’t need to use similar
service with SNS when you handle messages because of the strong integration (connectors)
between services.
Things are more inside Microsoft Azure. First of all, Azure
Queue Storage does not have support for FIFO. Simple like this, if you need to guarantee
the message order than Azure Queue Storage is not for you. Azure Service Bus is
the full ESB offer from Azure that comes with strong support for FIFO.
Beside canonical services from both providers, you can find
other 3rd parties that offering messaging solutions that can run on
top of AWS EC2 / Azure VMs or inside AWS ECS / Azure ACI. The real challenge when
you need ordering of the messages to be guarantee is how you write the
consumer. In the next post, we will talk about this.
Part 2 - ow to guarantee the order in which the messages are processed inside AWS MQ and Azure Service Bus (FIFO on the consumer)
Part 2 - ow to guarantee the order in which the messages are processed inside AWS MQ and Azure Service Bus (FIFO on the consumer)
Comments
Post a Comment