Skip to main content

Comparison of different Azure messaging systems

In theory sending a message over a wire to another device it is a simple task. But sending a message in a reliable and secure way can be a pretty hard job. In IoT era, where every day the number of devices that are connected to the internet increase drastically we need to find different communication mechanisms.
Because we cannot control when a device it is connected to the internet and ready to receive our package we need to find different ways to communicate with it.
In this post we will look over different messaging system that are offer by Microsoft Azure. For each messaging system we will try to identify what are strengths and when we should use this messaging system. Once we understand each messaging system we will compare them one by one.
In the end of this post we will not be able to find a perfect messaging system that can be used in any situations. For different use cases we may need to use different messaging systems, based on our needs.
The messaging solutions that will be discussed in this post are:
Azure Storage Queues
Azure Service Bus Queues
Azure Service Bus Topics
Azure Event Hub

Azure Storage Queues
This messaging system is part of Azure Storage and allows us to store a lot of messages in the same queue. When I say a lot, imagine queues that can reach not 1 GB, not 1TB and not 10TB. Using Azure Storage Queues we can have a queue that has 200 TB. Because of this, we have the ability to store large amounts of data in queues without thinking about the size of the queue.
Another benefit of this type of queue is the number of concurrent clients that in theory is unlimited. The only limits in this case is the bandwidth that can limits the number of concurrent clients. The maximum size of a message is 64 KB, but in combination with blobs we can have messages that reach 200 GB.
Of course there are also some limitations that we need to take into account. First of all, even if the size of queue can be very large, the maximum TTL (Time To Leave) of a message is 7 days. This means that a message needs to be consumed in 7 days or renewed, otherwise the message will be removed.
Even if we have support for base messaging capabilities like scheduler delivery and batch support, we don’t have support for state management, duplicate detection or transaction support.
In interesting feature of Azure Storage Queues is the logging capabilities. Users have the ability to activate the loggings mechanism and track all the actions that are happening on the queue. Tracking information like client IP are tracked and store as an out of the box solution.
Clients has the ability to peek the messages from the queue, without removing or locking them. This means that if a client peek a specific message, other clients will be able to access the same message from the queue.
Because of this, Azure Storage Queues are great when you need a messaging system that is able to track all the actions that are happening on it. It can be a good solutions for use cases when you know that the size of the queue will be bigger than 80-100 GB. For large queues this can be the best queue mechanisms.

Azure Service Bus Queues
This messaging system is part of a more complex messaging infrastructure offered by Microsoft and supports more complex scenarios. Because of this we will discover that the size of a queue is limited to 80 GB (using partitioning capabilities), but the features offered by Service Bus Queues are more complex.
It is important to know from the starting, that this two messaging systems are constructed over different services and have nothing in common. The size of a message can be 256 KB, larger in comparison with Azure Storage Queues and also a message will be persisted in Service Bus for an unlimited period of time. On top of this, there is full support for AMPQ protocol, feature that can be very useful for embedded devices.
There is support for dead lettering that allow us to move automatically a message in a secondary queue if message expires of clients cannot consume a specific message. There is full support from transactions, handling a specific number of messages in the same transaction. On top of this there is support to group multiple messages in the same session – in this way we can ensure that a client will consume all messages that are part of a specific session. If we know the ID of the session we can consumer messages from that specific session.
An interesting feature of Azure Service Bus Queues is duplicate detection. Once it activated, when we want to add a message that already exists in the system, the message will not be added to it. This is great when we want to ensure to have unique messages in the queue.
Messages can be consumed from the queue in two different ways – Peek and Lock or Receive and Delete. We can peek a message from a queue and make it unavailable for the rest of the clients until we confirm that we consumed it with success or abroad the action (we can specify also a timeout).
The security capabilities of Azure Service Bus Queues are more complex, we have the ability to control more deep the access to the messages.
Based on this features, Azure Service Bus Queues are great when we need duplicate detection, transaction support or to store messages for an unlimited period of time.

Azure Service Bus Topics
In contrast with Azure Service Bus Queues that allow us to deliver one-to-one, Azure Service Bus Topics allow us to deliver messages one-to-many. This mean that we can deliver the same message to multiple clients that are called subscribers. This messaging system is basically an ESB system (Enterprise Service Bus), allowing to have a “publish/subscribe” communication.

From features perspective, they are very similar with Azure Service Bus Queues with some additional features and capabilities. The similarity of this two services is because both messaging system are constructed over the same broker messaging infrastructure system.
Each topic that is used to send messages can have maximum 2.000 subscribers. This means that the same message can be received by 2.000 messages. This can be very useful when we need to distribute messages to different systems. Also a subscription can be added at runtime. We don’t need to stop the system or recreate the topic. Once a new subscription is created, all the new messages that are send to that topic will be received by the new subscription also.
An interesting feature is the filter support. At each subscription we can attach a filter to it. That filter will allow only the messages that respect the filter rule to reach that specific subscription. In this way each subscription can listen to specific messages.
Both messaging systems, Azure Service Bus Topics and Queues have automatic forward capabilities, but for Topics are more interesting. This means that we can forward a message from a subscription automatically to another Service Bus Topic of Queue.
Each message that is send over a Service Bus can have a collection of properties. Properties that are used when a custom filter per subscription is applied. Also each subscription can have a custom action that is executed when a message is received by it. Even if the actions that can be executed are very simple, can be very useful in some situations – for example we are allowed to change the name of the value of the property.
Based on this capabilities, Azure Service Bus Topics are perfect to use when we need to distribute a messages to multiple listeners. In systems where the number of clients that should receive a message can change dynamically, Azure Service Bus Topics can be useful.

Azure Event Hub
Even if in theory it is part of Azure Service Bus, Azure Event Hub is a special messaging system that is used to ingest large amounts of data in a short period of time. This system is capable to ingest more than 1 million messages per second without any kind of problem.

It is constructed around streaming concept. Because of this, all messages that flows in the system are seen like a stream of data. The latency is very low and even if the quantity of data that flows is very high the system is reliable and stable.
An interesting feature of this system is the capacity to navigate between messages that we already received. We have a concept similar with a cursor and we can iterate over old messages also – reply capability. On top of this the stream of messages can reach multiple consumers in the same time using the consumer group concept.
An important feature that can be found to Azure Service Bus is the message order preserving. It means that the order of messages is persisted. Consumers will be able to consume messages in the order they were send.
The Event Hub capabilities can scale pretty interesting, by adding multiple TUs (Throughput Units). Each TU allow us 1 MB/s for ingress and 2 MB/s for egress. The retention of message is 1 day, being able to store them up to 7 days.
Of course this comes with some costs. Features like death letter queue, transaction support or TTL options cannot be found on Azure Event Hub.
This solution is perfect for high volumes of messaging processing, like telemetry or in IoT use cases.

What is the best solution?
We don’t have a unique response for this question. Based on our needs and requirements we may use different messaging system. Azure Storage Queues are perfect when we need to store and manage large amounts of messages but when we need more control Azure Service Bus Queues can be a better solution. For use cases when we need to distribute messages to multiple listeners, Azure Service Bus Topic is the best. But, for large amounts of messaging processing, nothing can beat Azure Event Hub.

Conclusion
In this post we looked at different messaging systems offered by Azure and identify the most important features of each of them. We saw the most important use cases when we can use this systems and what are the weak points. At the end I would like to invite you to go on Azure web site and discover more about this great services.

Azure Service Bus Topics vs Azure Service Bus vs Azure Event Hub vs Azure Storage Queues

Comments

Popular posts from this blog

How to check in AngularJS if a service was register or not

There are cases when you need to check in a service or a controller was register in AngularJS.
For example a valid use case is when you have the same implementation running on multiple application. In this case, you may want to intercept the HTTP provider and add a custom step there. This step don’t needs to run on all the application, only in the one where the service exist and register.
A solution for this case would be to have a flag in the configuration that specify this. In the core you would have an IF that would check the value of this flag.
Another solution is to check if a specific service was register in AngularJS or not. If the service was register that you would execute your own logic.
To check if a service was register or not in AngularJS container you need to call the ‘has’ method of ‘inhector’. It will return TRUE if the service was register.
if ($injector.has('httpInterceptorService')) { $httpProvider.interceptors.push('httpInterceptorService&#…

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Run native .NET application in Docker (.NET Framework 4.6.2)

Scope
The main scope of this post is to see how we can run a legacy application written in .NET Framework in Docker.

Context
First of all, let’s define what is a legacy application in our context. By a legacy application we understand an application that runs .NET Framework 3.5 or higher in a production environment where we don’t have any more the people or documentation that would help us to understand what is happening behind the scene.
In this scenarios, you might want to migrate the current solution from a standard environment to Docker. There are many advantages for such a migration, like:

Continuous DeploymentTestingIsolationSecurity at container levelVersioning ControlEnvironment Standardization
Until now, we didn’t had the possibility to run a .NET application in Docker. With .NET Core, there was support for .NET Core in Docker, but migration from a full .NET framework to .NET Core can be costly and even impossible. Not only because of lack of features, but also because once you…