In theory sending a message over a wire to another device it is a simple task. But sending a message in a reliable and secure way can be a pretty hard job. In IoT era, where every day the number of devices that are connected to the internet increase drastically we need to find different communication mechanisms.
Because we cannot control when a device it is connected to the internet and ready to receive our package we need to find different ways to communicate with it.
In this post we will look over different messaging system that are offer by Microsoft Azure. For each messaging system we will try to identify what are strengths and when we should use this messaging system. Once we understand each messaging system we will compare them one by one.
In the end of this post we will not be able to find a perfect messaging system that can be used in any situations. For different use cases we may need to use different messaging systems, based on our needs.
The messaging solutions that will be discussed in this post are:
• Azure Storage Queues
• Azure Service Bus Queues
• Azure Service Bus Topics
• Azure Event Hub
Azure Storage Queues
This messaging system is part of Azure Storage and allows us to store a lot of messages in the same queue. When I say a lot, imagine queues that can reach not 1 GB, not 1TB and not 10TB. Using Azure Storage Queues we can have a queue that has 200 TB. Because of this, we have the ability to store large amounts of data in queues without thinking about the size of the queue.
Another benefit of this type of queue is the number of concurrent clients that in theory is unlimited. The only limits in this case is the bandwidth that can limits the number of concurrent clients. The maximum size of a message is 64 KB, but in combination with blobs we can have messages that reach 200 GB.
Of course there are also some limitations that we need to take into account. First of all, even if the size of queue can be very large, the maximum TTL (Time To Leave) of a message is 7 days. This means that a message needs to be consumed in 7 days or renewed, otherwise the message will be removed.
Even if we have support for base messaging capabilities like scheduler delivery and batch support, we don’t have support for state management, duplicate detection or transaction support.
In interesting feature of Azure Storage Queues is the logging capabilities. Users have the ability to activate the loggings mechanism and track all the actions that are happening on the queue. Tracking information like client IP are tracked and store as an out of the box solution.
Clients has the ability to peek the messages from the queue, without removing or locking them. This means that if a client peek a specific message, other clients will be able to access the same message from the queue.
Because of this, Azure Storage Queues are great when you need a messaging system that is able to track all the actions that are happening on it. It can be a good solutions for use cases when you know that the size of the queue will be bigger than 80-100 GB. For large queues this can be the best queue mechanisms.
Azure Service Bus Queues
This messaging system is part of a more complex messaging infrastructure offered by Microsoft and supports more complex scenarios. Because of this we will discover that the size of a queue is limited to 80 GB (using partitioning capabilities), but the features offered by Service Bus Queues are more complex.
It is important to know from the starting, that this two messaging systems are constructed over different services and have nothing in common. The size of a message can be 256 KB, larger in comparison with Azure Storage Queues and also a message will be persisted in Service Bus for an unlimited period of time. On top of this, there is full support for AMPQ protocol, feature that can be very useful for embedded devices.
There is support for dead lettering that allow us to move automatically a message in a secondary queue if message expires of clients cannot consume a specific message. There is full support from transactions, handling a specific number of messages in the same transaction. On top of this there is support to group multiple messages in the same session – in this way we can ensure that a client will consume all messages that are part of a specific session. If we know the ID of the session we can consumer messages from that specific session.
An interesting feature of Azure Service Bus Queues is duplicate detection. Once it activated, when we want to add a message that already exists in the system, the message will not be added to it. This is great when we want to ensure to have unique messages in the queue.
Messages can be consumed from the queue in two different ways – Peek and Lock or Receive and Delete. We can peek a message from a queue and make it unavailable for the rest of the clients until we confirm that we consumed it with success or abroad the action (we can specify also a timeout).
The security capabilities of Azure Service Bus Queues are more complex, we have the ability to control more deep the access to the messages.
Based on this features, Azure Service Bus Queues are great when we need duplicate detection, transaction support or to store messages for an unlimited period of time.
Azure Service Bus Topics
In contrast with Azure Service Bus Queues that allow us to deliver one-to-one, Azure Service Bus Topics allow us to deliver messages one-to-many. This mean that we can deliver the same message to multiple clients that are called subscribers. This messaging system is basically an ESB system (Enterprise Service Bus), allowing to have a “publish/subscribe” communication.
From features perspective, they are very similar with Azure Service Bus Queues with some additional features and capabilities. The similarity of this two services is because both messaging system are constructed over the same broker messaging infrastructure system.
Each topic that is used to send messages can have maximum 2.000 subscribers. This means that the same message can be received by 2.000 messages. This can be very useful when we need to distribute messages to different systems. Also a subscription can be added at runtime. We don’t need to stop the system or recreate the topic. Once a new subscription is created, all the new messages that are send to that topic will be received by the new subscription also.
An interesting feature is the filter support. At each subscription we can attach a filter to it. That filter will allow only the messages that respect the filter rule to reach that specific subscription. In this way each subscription can listen to specific messages.
Both messaging systems, Azure Service Bus Topics and Queues have automatic forward capabilities, but for Topics are more interesting. This means that we can forward a message from a subscription automatically to another Service Bus Topic of Queue.
Each message that is send over a Service Bus can have a collection of properties. Properties that are used when a custom filter per subscription is applied. Also each subscription can have a custom action that is executed when a message is received by it. Even if the actions that can be executed are very simple, can be very useful in some situations – for example we are allowed to change the name of the value of the property.
Based on this capabilities, Azure Service Bus Topics are perfect to use when we need to distribute a messages to multiple listeners. In systems where the number of clients that should receive a message can change dynamically, Azure Service Bus Topics can be useful.
Azure Event Hub
Even if in theory it is part of Azure Service Bus, Azure Event Hub is a special messaging system that is used to ingest large amounts of data in a short period of time. This system is capable to ingest more than 1 million messages per second without any kind of problem.
It is constructed around streaming concept. Because of this, all messages that flows in the system are seen like a stream of data. The latency is very low and even if the quantity of data that flows is very high the system is reliable and stable.
An interesting feature of this system is the capacity to navigate between messages that we already received. We have a concept similar with a cursor and we can iterate over old messages also – reply capability. On top of this the stream of messages can reach multiple consumers in the same time using the consumer group concept.
An important feature that can be found to Azure Service Bus is the message order preserving. It means that the order of messages is persisted. Consumers will be able to consume messages in the order they were send.
The Event Hub capabilities can scale pretty interesting, by adding multiple TUs (Throughput Units). Each TU allow us 1 MB/s for ingress and 2 MB/s for egress. The retention of message is 1 day, being able to store them up to 7 days.
Of course this comes with some costs. Features like death letter queue, transaction support or TTL options cannot be found on Azure Event Hub.
This solution is perfect for high volumes of messaging processing, like telemetry or in IoT use cases.
What is the best solution?
We don’t have a unique response for this question. Based on our needs and requirements we may use different messaging system. Azure Storage Queues are perfect when we need to store and manage large amounts of messages but when we need more control Azure Service Bus Queues can be a better solution. For use cases when we need to distribute messages to multiple listeners, Azure Service Bus Topic is the best. But, for large amounts of messaging processing, nothing can beat Azure Event Hub.
Conclusion
In this post we looked at different messaging systems offered by Azure and identify the most important features of each of them. We saw the most important use cases when we can use this systems and what are the weak points. At the end I would like to invite you to go on Azure web site and discover more about this great services.
Azure Service Bus Topics vs Azure Service Bus vs Azure Event Hub vs Azure Storage Queues
Because we cannot control when a device it is connected to the internet and ready to receive our package we need to find different ways to communicate with it.
In this post we will look over different messaging system that are offer by Microsoft Azure. For each messaging system we will try to identify what are strengths and when we should use this messaging system. Once we understand each messaging system we will compare them one by one.
In the end of this post we will not be able to find a perfect messaging system that can be used in any situations. For different use cases we may need to use different messaging systems, based on our needs.
The messaging solutions that will be discussed in this post are:
• Azure Storage Queues
• Azure Service Bus Queues
• Azure Service Bus Topics
• Azure Event Hub
Azure Storage Queues
This messaging system is part of Azure Storage and allows us to store a lot of messages in the same queue. When I say a lot, imagine queues that can reach not 1 GB, not 1TB and not 10TB. Using Azure Storage Queues we can have a queue that has 200 TB. Because of this, we have the ability to store large amounts of data in queues without thinking about the size of the queue.
Another benefit of this type of queue is the number of concurrent clients that in theory is unlimited. The only limits in this case is the bandwidth that can limits the number of concurrent clients. The maximum size of a message is 64 KB, but in combination with blobs we can have messages that reach 200 GB.
Of course there are also some limitations that we need to take into account. First of all, even if the size of queue can be very large, the maximum TTL (Time To Leave) of a message is 7 days. This means that a message needs to be consumed in 7 days or renewed, otherwise the message will be removed.
Even if we have support for base messaging capabilities like scheduler delivery and batch support, we don’t have support for state management, duplicate detection or transaction support.
In interesting feature of Azure Storage Queues is the logging capabilities. Users have the ability to activate the loggings mechanism and track all the actions that are happening on the queue. Tracking information like client IP are tracked and store as an out of the box solution.
Clients has the ability to peek the messages from the queue, without removing or locking them. This means that if a client peek a specific message, other clients will be able to access the same message from the queue.
Because of this, Azure Storage Queues are great when you need a messaging system that is able to track all the actions that are happening on it. It can be a good solutions for use cases when you know that the size of the queue will be bigger than 80-100 GB. For large queues this can be the best queue mechanisms.
Azure Service Bus Queues
This messaging system is part of a more complex messaging infrastructure offered by Microsoft and supports more complex scenarios. Because of this we will discover that the size of a queue is limited to 80 GB (using partitioning capabilities), but the features offered by Service Bus Queues are more complex.
It is important to know from the starting, that this two messaging systems are constructed over different services and have nothing in common. The size of a message can be 256 KB, larger in comparison with Azure Storage Queues and also a message will be persisted in Service Bus for an unlimited period of time. On top of this, there is full support for AMPQ protocol, feature that can be very useful for embedded devices.
There is support for dead lettering that allow us to move automatically a message in a secondary queue if message expires of clients cannot consume a specific message. There is full support from transactions, handling a specific number of messages in the same transaction. On top of this there is support to group multiple messages in the same session – in this way we can ensure that a client will consume all messages that are part of a specific session. If we know the ID of the session we can consumer messages from that specific session.
An interesting feature of Azure Service Bus Queues is duplicate detection. Once it activated, when we want to add a message that already exists in the system, the message will not be added to it. This is great when we want to ensure to have unique messages in the queue.
Messages can be consumed from the queue in two different ways – Peek and Lock or Receive and Delete. We can peek a message from a queue and make it unavailable for the rest of the clients until we confirm that we consumed it with success or abroad the action (we can specify also a timeout).
The security capabilities of Azure Service Bus Queues are more complex, we have the ability to control more deep the access to the messages.
Based on this features, Azure Service Bus Queues are great when we need duplicate detection, transaction support or to store messages for an unlimited period of time.
Azure Service Bus Topics
In contrast with Azure Service Bus Queues that allow us to deliver one-to-one, Azure Service Bus Topics allow us to deliver messages one-to-many. This mean that we can deliver the same message to multiple clients that are called subscribers. This messaging system is basically an ESB system (Enterprise Service Bus), allowing to have a “publish/subscribe” communication.
From features perspective, they are very similar with Azure Service Bus Queues with some additional features and capabilities. The similarity of this two services is because both messaging system are constructed over the same broker messaging infrastructure system.
Each topic that is used to send messages can have maximum 2.000 subscribers. This means that the same message can be received by 2.000 messages. This can be very useful when we need to distribute messages to different systems. Also a subscription can be added at runtime. We don’t need to stop the system or recreate the topic. Once a new subscription is created, all the new messages that are send to that topic will be received by the new subscription also.
An interesting feature is the filter support. At each subscription we can attach a filter to it. That filter will allow only the messages that respect the filter rule to reach that specific subscription. In this way each subscription can listen to specific messages.
Both messaging systems, Azure Service Bus Topics and Queues have automatic forward capabilities, but for Topics are more interesting. This means that we can forward a message from a subscription automatically to another Service Bus Topic of Queue.
Each message that is send over a Service Bus can have a collection of properties. Properties that are used when a custom filter per subscription is applied. Also each subscription can have a custom action that is executed when a message is received by it. Even if the actions that can be executed are very simple, can be very useful in some situations – for example we are allowed to change the name of the value of the property.
Based on this capabilities, Azure Service Bus Topics are perfect to use when we need to distribute a messages to multiple listeners. In systems where the number of clients that should receive a message can change dynamically, Azure Service Bus Topics can be useful.
Azure Event Hub
Even if in theory it is part of Azure Service Bus, Azure Event Hub is a special messaging system that is used to ingest large amounts of data in a short period of time. This system is capable to ingest more than 1 million messages per second without any kind of problem.
It is constructed around streaming concept. Because of this, all messages that flows in the system are seen like a stream of data. The latency is very low and even if the quantity of data that flows is very high the system is reliable and stable.
An interesting feature of this system is the capacity to navigate between messages that we already received. We have a concept similar with a cursor and we can iterate over old messages also – reply capability. On top of this the stream of messages can reach multiple consumers in the same time using the consumer group concept.
An important feature that can be found to Azure Service Bus is the message order preserving. It means that the order of messages is persisted. Consumers will be able to consume messages in the order they were send.
The Event Hub capabilities can scale pretty interesting, by adding multiple TUs (Throughput Units). Each TU allow us 1 MB/s for ingress and 2 MB/s for egress. The retention of message is 1 day, being able to store them up to 7 days.
Of course this comes with some costs. Features like death letter queue, transaction support or TTL options cannot be found on Azure Event Hub.
This solution is perfect for high volumes of messaging processing, like telemetry or in IoT use cases.
What is the best solution?
We don’t have a unique response for this question. Based on our needs and requirements we may use different messaging system. Azure Storage Queues are perfect when we need to store and manage large amounts of messages but when we need more control Azure Service Bus Queues can be a better solution. For use cases when we need to distribute messages to multiple listeners, Azure Service Bus Topic is the best. But, for large amounts of messaging processing, nothing can beat Azure Event Hub.
Conclusion
In this post we looked at different messaging systems offered by Azure and identify the most important features of each of them. We saw the most important use cases when we can use this systems and what are the weak points. At the end I would like to invite you to go on Azure web site and discover more about this great services.
Azure Service Bus Topics vs Azure Service Bus vs Azure Event Hub vs Azure Storage Queues
Comments
Post a Comment