Skip to main content

Events and Message based system comparison

I heard very often discussion related to messaging and event systems. People are trying to compare them and decide on one solution that suits all their needs on a platform. Comparison of an Enterprise Service Bus (ESB) and an Event base messaging system it’s wrong from the beginning.

Even if both solutions are great, they resolve different problems that we might have on a platform. ESB and Events are able to transport data between two or multiple systems, but this is where the similarities end. From the way how they are design, core features and what we can achieve with each of them, we will notify that their capabilities are limited and they don’t compete between each other.

As we will discover, later on, a complex solution might need to use both approaches to offer a consistent and competitive solution.
To be able to give hands-on examples, we will use for comparison two services that Microsoft Azure is offering.

  • Azure Service Bus – an ESB solution hosted inside Microsoft Azure, that gives us the ability to work with Queues and Topics/Subscribers. 
  • Azure Event Hub – an event-based solution hosted inside Microsoft Azure, that gives us the ability to push and consumer millions of events in just a few seconds.

One of the causes why we see events and messages solutions similar it causes by how both look like. Each of them has a header and a payload and their main mission is to transport content from one location to one or multiple destinations. This concept is similar to the one provided by a Datagram that is the basic transfer unit associated with a packet-switched network and well documented in the TCP/IP protocol - RFC 1594.
I like the analogy with a datagram because of how it is defined: self-contained, no dependencies of previous content exchanged and connectionless (without support/requiring of fixed connection). This can be translated in our case of supporting data exchange between two parties if fixed block sizes without requiring a stable and open connection.

Messages
When we are working on a communication channel where we need to ensure that the content is received with success a message based solution is recommended. A solution like the Azure Service Bus Queue or Topic provides us with a reliable communication channel that ensures that the message is not lost. There are times when we expect an acknowledge from the receiver(s), the case when a message based system is the best option. 
In general, we need a message base solution when we are working with a command like an approach, where we have a publisher that publishes content on the wire. The expectation is that at least one system will handle the command and a job is executed. For these situations, we shall rely on messages.
A characteristic of message base systems like Azure Service Bus is the contract that exists between producer and consumer. Both sides know exactly how the message looks like, the data format and expect information in a specific way. In general, even the producer expect a response in a specific format.

Events
From the previous example with datagrams, events are more close to datagrams that messages. The purpose of data that is transported by events is more to inform than to expect a specific action to happen. They capture activities or changes on a system and publish them on different channels.
One of the biggest difference between events and messages is the expectation of the producer. For messages, the producer expects activity to happen, in comparison with an event producer that does not have any expectation.
In comparison with messages, events don’t contain any information related to who triggered the data and context information. They just report a state change or a specific value.
In general, events are used as data points generated by different systems or store information in the change of state of different actions. A sensor that reads the car speed does not expect a reaction from the consumer each time when the speed is reported. Beside this, it generates a stream of data that can be consumed and viewed in different ways – data points. In comparison with this, a device that starts the car remotely is sending a command to the car and expect action to be executed together with some acknowledgments. In this case, the device needs to generate a message that is sent to the car.
Another characteristic of events stream is the ability to consume events from specific time periods multiple times, or even peek data at a specific time interval. For message base system, like Azure Service Bus a message can be consumed only once by all subscribers and cannot be retrieved later on once it was consumed.

You might say that there are cases when a consumer can react to different events. Yes, this is true, but the logic and the way a consumer reacts it is detailed in the consumer, not in the event itself. Losing a few events in an IoT solution is accepted and will not impact the solution stability. In comparison losing a message that contains a SELL, THE SHARES can make a difference between bankruptcy or winning 10M $.
Events are seen as streams of events, where the most valuable information is obtaining by comparing events between each other and tracking the trend of data. The value of a single event is ZERO, but analyzing the events that contained the shares value for the last 12 months is priceless.
The previous example is related to statistical information or data points. Beside this, we can have events that inform us about different activities that are happening inside the system. The purpose of these events is just to inform and losing such an event would not affect our system.
A data points stream of events that are widely used is data logs. One of multiple systems or components can produce data logs events that can be later on processed. Azure Event Hubs allow us to automatically store datalog events directly to storage for later on analyze. In most of the cases you might never analyze the log events or to extract just some report from it.

Another difference between a message VS event base system is the filtering/routing capability. An event-based system like Azure Event Hub is able to retrieve us messages from the specific time period, but it does not have the capability to filter messages for consumer groups. In comparison, an ESB system like Azure Service Bus is fully capable to filter messages on each subscription based on predefined rules.
Azure Service Bus or any other messaging system can guarantee the order delivery. In an event-based system, the order can be different. When we are working with transactions, the data order is very important, making us look at a message based solution. Additional to this an event-based system is not capable to detect duplication. One of the main reasons is because is designed for high throughput, where any data processing would increase latency.

From a security perspective, at transport and storage level both approaches are similar offering encrypted storage and a secure connection. At authentication and authorization level, the message-based system is offering more options that can be seen exotic from an event-based system. This is happening because when you need to support high thoughts, complex authentication and authorization solutions that might rely on an external system could increase latency.

Event / Messages comparison
Feature / Characteristic
Azure Service Bus
Azure Event Hub
Command base
Yes
No
Reliable async delivery
Yes
Partial
Receiver confirmation
Yes
No
Data Stream
No
Yes
Content Routing/Filtering
YES
No
Fault tolerance
YES
No
Transaction handling
Yes
No
High volume of data
No
Yes
Multiple consumers
Yes
Yes
Telemetry use case
No
Yes
Order processing
Yes
No
Low latency
No
Yes
At least once delivery
Yes
Yes
One time delivery
Yes
No
Data ordering
Yes
No
Duplicate detection
Yes
No
Data consistency
Yes
No
FIFO like
Yes
No
Batching
Yes
Yes
Session
Yes
No
Dead-letter
Yes
No
Durability
Yes
Yes
TTL
Yes
Yes
Token base access (SAS)
Yes
Yes
Auto-forwarding
Yes
No
Scheduled delivery
Yes
No
Message deferral
Yes
No
Auto-delete
Yes
Yes
Retrieve old data
No
Yes

Conclusion
In a complex system, you will need both message and event-based solution. They don’t exclude one another and you can develop a solid system that combines messages and events together. A short personal for both of them can be found below.

  • Events – <inform> notifications of a condition of state change 
  • Messages – <commands> raw data produced by a system to be consumed

Comments

Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Entity Framework (EF) TransactionScope vs Database.BeginTransaction

In today blog post we will talk a little about a new feature that is available on EF6+ related to Transactions.
Until now, when we had to use transaction we used ‘TransactionScope’. It works great and I would say that is something that is now in our blood.
using (var scope = new TransactionScope(TransactionScopeOption.Required)) { using (SqlConnection conn = new SqlConnection("...")) { conn.Open(); SqlCommand sqlCommand = new SqlCommand(); sqlCommand.Connection = conn; sqlCommand.CommandText = ... sqlCommand.ExecuteNonQuery(); ... } scope.Complete(); } Starting with EF6.0 we have a new way to work with transactions. The new approach is based on Database.BeginTransaction(), Database.Rollback(), Database.Commit(). Yes, no more TransactionScope.
In the followi…

GET call of REST API that contains '/'-slash character in the value of a parameter

Let’s assume that we have the following scenario: I have a public HTTP endpoint and I need to post some content using GET command. One of the parameters contains special characters like “\” and “/”. If the endpoint is an ApiController than you may have problems if you encode the parameter using the http encoder.
using (var httpClient = new HttpClient()) { httpClient.BaseAddress = baseUrl; Task<HttpResponseMessage> response = httpClient.GetAsync(string.Format("api/foo/{0}", "qwert/qwerqwer"))); response.Wait(); response.Result.EnsureSuccessStatusCode(); } One possible solution would be to encode the query parameter using UrlTokenEncode method of HttpServerUtility class and GetBytes method ofUTF8. In this way you would get the array of bytes of the parameter and encode them as a url token.
The following code show to you how you could write the encode and decode methods.
publ…