Skip to main content

(Part 1) Testing the limits of Windows Azure Service Bus


At the end of the last year I had the opportunity to write an application that had to handle millions of messages every day. To distribute the messages between different components of the application we used Windows Azure Service Bus Service.
In the past I talked a lot about this distribution messaging system. In this post we will see what are the magic numbers of a Topic from Windows Azure Service Bus Service.
Scope:
To see how many messages can be process in as short period of time. What is the maximum number of messages that can be processed using Service Bus using only one Topic. What is the optimal number of consumers for a subscriber or topic.
Environment:
Each message that was added to Service Bus was pretty small. We had around 100 characters in UTF7 and 3 properties added to each BrokeredMessage.
We run the tests with one; two and three subscribers with different filter rules and the result were similar.
Action:
We created 5 to 10 worker roles that started to send hundreds of messages to our Service Bus Topic.  In a very short period of time we could send millions of messages.
Results:
We tried pushing on the topic around 5 million messages. We observe that after a part of the messages are send to the Topic, the server starts to be slower. By slower I mean that the response latency increased drastically. Not only on the send operation, but also on the subscribers – the messages started to be received by consumer slower and slower.
One interesting part was on the portal. Even if the Topic was almost full, the number of messages that were reported on the Topic was only around 200.000-300.000.
After different experiment, we observe that with the current version of Service Bus and with the current configuration, a Topic can handle around 1.000.000 messages every 30 minutes.
Note: This value was obtain based on our experiment with our topic and messages configuration. This value is not a generic value, for any kind of project and configuration.
Cost:
From the cost perspective, we end up with a cost around 1$ per day if we use Windows Azure Service Bus for processing 1.000.000 messages. The size of each message was calculated by us around 24kB. This means that for processing 10.000.000 messages every day for a month would cost us around 300$. Before thinking that this are a lot of money, you should take into account that Service Bus will not lose any messages and in the end we process 300.000.000 (300M) messages per month.

I would conclude that the obtained value is pretty good. It is more that we expect to process a 30 minutes job, using one topic.
In the next post I will present the number of consumers that we used to process the messages in the best way possible.

Comments

Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Entity Framework (EF) TransactionScope vs Database.BeginTransaction

In today blog post we will talk a little about a new feature that is available on EF6+ related to Transactions.
Until now, when we had to use transaction we used ‘TransactionScope’. It works great and I would say that is something that is now in our blood.
using (var scope = new TransactionScope(TransactionScopeOption.Required)) { using (SqlConnection conn = new SqlConnection("...")) { conn.Open(); SqlCommand sqlCommand = new SqlCommand(); sqlCommand.Connection = conn; sqlCommand.CommandText = ... sqlCommand.ExecuteNonQuery(); ... } scope.Complete(); } Starting with EF6.0 we have a new way to work with transactions. The new approach is based on Database.BeginTransaction(), Database.Rollback(), Database.Commit(). Yes, no more TransactionScope.
In the followi…

GET call of REST API that contains '/'-slash character in the value of a parameter

Let’s assume that we have the following scenario: I have a public HTTP endpoint and I need to post some content using GET command. One of the parameters contains special characters like “\” and “/”. If the endpoint is an ApiController than you may have problems if you encode the parameter using the http encoder.
using (var httpClient = new HttpClient()) { httpClient.BaseAddress = baseUrl; Task<HttpResponseMessage> response = httpClient.GetAsync(string.Format("api/foo/{0}", "qwert/qwerqwer"))); response.Wait(); response.Result.EnsureSuccessStatusCode(); } One possible solution would be to encode the query parameter using UrlTokenEncode method of HttpServerUtility class and GetBytes method ofUTF8. In this way you would get the array of bytes of the parameter and encode them as a url token.
The following code show to you how you could write the encode and decode methods.
publ…