Skip to main content

FIFO and queues inside AWS and Azure

Nowadays, most of the systems that are running inside a cloud provider like AWS or Microsoft Azure are using in one way or another a messaging system. Many people forget that not all the time a queue guarantee FIFO.
Things are becoming more sophisticated, at the moment in time when you integrate the queue with other systems. You might realise that even if your queue is supporting FIFO, the integration with other cloud services will not enable you to use queue with FIFO guarantee. Because of this, I decided to write this short blog post that highlights things that might go wrong when you need a FIFO inside AWS or Microsoft Azure and how you could tackle them.

AWS and FIFO
At this moment in time (January 2020) the AWS services that manage messages are:
  • AWS SQS
  • AWS MQ
  • AWS SNS (simple event/message routing solution)

NOTE: We don’t take into consideration AWS Kinesis and AWS IoT Message Broker because they are more specific for events handling and IoT solutions.
From the above list, AWS SQS has the option to have FIFO SQS, and AWS MQ is a full message broker solution, where FIFO is ‘by design’. Be careful with AWS SQS because the default configuration is not with FIFO guarantee.
There is another surprise for AWS FIFO SQS if you want to use it in combination with AWS SNS. You cannot send messages from AWS SNS to FIFO SQS. Yes, this is true. At this moment in time, we don’t have support for it. So, if you need a simple message broker and you hopped that you can use AWS SNS and AWS FIFO SQS, well – you can’t. Just use AWS MQ from the beginning for something like this. The cause of this lack of integration can be from the communication protocol (HTTP/S). By default, the protocol cannot guaranty us that if we send to the same endpoint 3 different requests, they will arrive and be consumed in the same order.


Microsoft Azure and FIFO
In January 2020, on Azure, we can find the following services that can manage our messages:
  • Azure Queue Storage
  • Azure Service Bus

NOTE: Event Hub is not on the list because of the primary purpose of it is to handle events and not messages. The Azure equivalent for AWS SNS is Azure Event Grid, but inside Azure, in general, you don’t need to use similar service with SNS when you handle messages because of the strong integration (connectors) between services.
Things are more inside Microsoft Azure. First of all, Azure Queue Storage does not have support for FIFO. Simple like this, if you need to guarantee the message order than Azure Queue Storage is not for you. Azure Service Bus is the full ESB offer from Azure that comes with strong support for FIFO.

Beside canonical services from both providers, you can find other 3rd parties that offering messaging solutions that can run on top of AWS EC2 / Azure VMs or inside AWS ECS / Azure ACI. The real challenge when you need ordering of the messages to be guarantee is how you write the consumer. In the next post, we will talk about this.
Part 2 -  ow to guarantee the order in which the messages are processed inside AWS MQ and Azure Service Bus (FIFO on the consumer)

Comments

Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Entity Framework (EF) TransactionScope vs Database.BeginTransaction

In today blog post we will talk a little about a new feature that is available on EF6+ related to Transactions.
Until now, when we had to use transaction we used ‘TransactionScope’. It works great and I would say that is something that is now in our blood.
using (var scope = new TransactionScope(TransactionScopeOption.Required)) { using (SqlConnection conn = new SqlConnection("...")) { conn.Open(); SqlCommand sqlCommand = new SqlCommand(); sqlCommand.Connection = conn; sqlCommand.CommandText = ... sqlCommand.ExecuteNonQuery(); ... } scope.Complete(); } Starting with EF6.0 we have a new way to work with transactions. The new approach is based on Database.BeginTransaction(), Database.Rollback(), Database.Commit(). Yes, no more TransactionScope.
In the followi…

GET call of REST API that contains '/'-slash character in the value of a parameter

Let’s assume that we have the following scenario: I have a public HTTP endpoint and I need to post some content using GET command. One of the parameters contains special characters like “\” and “/”. If the endpoint is an ApiController than you may have problems if you encode the parameter using the http encoder.
using (var httpClient = new HttpClient()) { httpClient.BaseAddress = baseUrl; Task<HttpResponseMessage> response = httpClient.GetAsync(string.Format("api/foo/{0}", "qwert/qwerqwer"))); response.Wait(); response.Result.EnsureSuccessStatusCode(); } One possible solution would be to encode the query parameter using UrlTokenEncode method of HttpServerUtility class and GetBytes method ofUTF8. In this way you would get the array of bytes of the parameter and encode them as a url token.
The following code show to you how you could write the encode and decode methods.
publ…