Skip to main content

(Part 1) Testing the limits of Windows Azure Service Bus


At the end of the last year I had the opportunity to write an application that had to handle millions of messages every day. To distribute the messages between different components of the application we used Windows Azure Service Bus Service.
In the past I talked a lot about this distribution messaging system. In this post we will see what are the magic numbers of a Topic from Windows Azure Service Bus Service.
Scope:
To see how many messages can be process in as short period of time. What is the maximum number of messages that can be processed using Service Bus using only one Topic. What is the optimal number of consumers for a subscriber or topic.
Environment:
Each message that was added to Service Bus was pretty small. We had around 100 characters in UTF7 and 3 properties added to each BrokeredMessage.
We run the tests with one; two and three subscribers with different filter rules and the result were similar.
Action:
We created 5 to 10 worker roles that started to send hundreds of messages to our Service Bus Topic.  In a very short period of time we could send millions of messages.
Results:
We tried pushing on the topic around 5 million messages. We observe that after a part of the messages are send to the Topic, the server starts to be slower. By slower I mean that the response latency increased drastically. Not only on the send operation, but also on the subscribers – the messages started to be received by consumer slower and slower.
One interesting part was on the portal. Even if the Topic was almost full, the number of messages that were reported on the Topic was only around 200.000-300.000.
After different experiment, we observe that with the current version of Service Bus and with the current configuration, a Topic can handle around 1.000.000 messages every 30 minutes.
Note: This value was obtain based on our experiment with our topic and messages configuration. This value is not a generic value, for any kind of project and configuration.
Cost:
From the cost perspective, we end up with a cost around 1$ per day if we use Windows Azure Service Bus for processing 1.000.000 messages. The size of each message was calculated by us around 24kB. This means that for processing 10.000.000 messages every day for a month would cost us around 300$. Before thinking that this are a lot of money, you should take into account that Service Bus will not lose any messages and in the end we process 300.000.000 (300M) messages per month.

I would conclude that the obtained value is pretty good. It is more that we expect to process a 30 minutes job, using one topic.
In the next post I will present the number of consumers that we used to process the messages in the best way possible.

Comments

Popular posts from this blog

How to check in AngularJS if a service was register or not

There are cases when you need to check in a service or a controller was register in AngularJS.
For example a valid use case is when you have the same implementation running on multiple application. In this case, you may want to intercept the HTTP provider and add a custom step there. This step don’t needs to run on all the application, only in the one where the service exist and register.
A solution for this case would be to have a flag in the configuration that specify this. In the core you would have an IF that would check the value of this flag.
Another solution is to check if a specific service was register in AngularJS or not. If the service was register that you would execute your own logic.
To check if a service was register or not in AngularJS container you need to call the ‘has’ method of ‘inhector’. It will return TRUE if the service was register.
if ($injector.has('httpInterceptorService')) { $httpProvider.interceptors.push('httpInterceptorService&#…

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Run native .NET application in Docker (.NET Framework 4.6.2)

Scope
The main scope of this post is to see how we can run a legacy application written in .NET Framework in Docker.

Context
First of all, let’s define what is a legacy application in our context. By a legacy application we understand an application that runs .NET Framework 3.5 or higher in a production environment where we don’t have any more the people or documentation that would help us to understand what is happening behind the scene.
In this scenarios, you might want to migrate the current solution from a standard environment to Docker. There are many advantages for such a migration, like:

Continuous DeploymentTestingIsolationSecurity at container levelVersioning ControlEnvironment Standardization
Until now, we didn’t had the possibility to run a .NET application in Docker. With .NET Core, there was support for .NET Core in Docker, but migration from a full .NET framework to .NET Core can be costly and even impossible. Not only because of lack of features, but also because once you…