Skip to main content

(Part 3) Testing the limits of Windows Azure Service Bus

In the latest post series about the limits of Windows Azure Service Bus, we saw that is the maximum number of messages that we can process through a single topic (1.000.000 messages every 30 minutes) and what kind of worker role we should use to process this messages in the optimum way (Medium size).
In this post I will try to respond to another question that is normal to appear when we are using a cloud solution.
Can I scale the number of instances that consume messages from the topic?
Environment:
Each message that was added to Service Bus was pretty small. We had around 100 characters in UTF7 and 3 properties added to each BrokeredMessage.
We run the tests with one; two and three subscribers with different filter rules and the result were similar.
Each message that is received from Service Bus required a custom action to be executed. This action is pretty complicated and consumes CPU power. Also the logic requires to access remote services (that are stored in the same data-center).
For all this tests we used a medium size worker role.
Action:
In the first phase we pushed on the topic 100.000 messages and measure how long it takes to process all the available messages. We tried to identify what is the best number of instances that we can have in parallel from the perspective of costs and time.
The next phase was to see detect what is the optimum number of messages that can be processed using the given number of instances that we found in the first phase.
Results:
Phase 1: We tried to process 100.000 of messages with different number of worker roles instances. The size of all instances was Medium. The results of these tests were:
  • 1 instance – 18 minutes and 10 seconds
  • 2 instance – 11 minutes and 40 seconds
  • 3 instance – 7 minutes and 15 seconds
  • 4 instance – 4 minutes and 15 seconds
  • 5 instance – 3 minutes and 20 seconds
  • 6 instance – 2 minutes and 59 seconds
  • 7 instance – 2 minutes and 27 seconds
  • 8 instance – 2 minutes and 5 seconds
We can observe that at while scaling up, the time decrease with around 30% percent until we reach 4-5 instances, when the time start to decrease more slowly. This happens because when a topic is intensive used, the response time increase. Don’t expect to have the same latency when the topic is hit 10 times per seconds or 1000 per seconds.
When calculating the price, we should take into account to different prices. The first price represents the cost of running N instances for the period of time when messages from topic are processed. The second price is the obsolete price, which represents the cost of running N instances for the minimum period of time – in the case of Windows Azure the smallest time unit is hour.
Taking all this things into account, we observed that 4 instance of medium size can process our messages in the shortest period of time with the best costs.
Having this “magic” number of instances and the size of them we made the next step. We measure how long it takes for our application that is deployed on 4 worker roles with medium size to process different number of messages. The results of this test were:
  • 100.000 messages – 4 minutes
  • 200.000 messages – 8 minutes
  • 300.000 messages – 11 minutes
  • 500.000 messages – 19 minutes
  • 1.000.000 messages – 34 minutes
  • 3.000.000 messages – 2 hours and 32 minutes
The results were pretty interesting. The performance is also the same until we reach a critical point. Do you remember the first post when I mentioned that we observe that 1.000.000 messages per 30 minutes seems to be the maximum number of messages that we can process in an optimum way?
This is the point when the performance starts to go down. We can observe that the difference between processing 1.000.000 messages and 3.000.000 messages is pretty big. From the time perspective the last test requested almost 4.5x more time.
At the end of the tests we decided to use only one worker role of medium size to see how long it takes to process different number of messages. The time results were:
  • 100.000 messages – 18 minutes
  • 200.000 messages – 32 minutes
  • 300.000 messages – 51 minutes
  • 500.000 messages – 1 hour and 31 minutes
  • 1.000.000 messages – 2 hour and 58 minutes
What I liked at this result it was how the duration increased. The duration time increases almost like the number of messages.
Conclusion
For our business problems we observed that having 4 worker roles of medium size of the same topic is the best configuration that we can have from time and costs perspective. This result is extremely important because we know where is the point when we need to scale in a different way.
In the next post related to this topic we will talked about costs.
Remarks: Usually all the duration values are rounded to minutes (without seconds and milliseconds). This does not mean that we don’t have these values.

Part 4

Comments

Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Entity Framework (EF) TransactionScope vs Database.BeginTransaction

In today blog post we will talk a little about a new feature that is available on EF6+ related to Transactions.
Until now, when we had to use transaction we used ‘TransactionScope’. It works great and I would say that is something that is now in our blood.
using (var scope = new TransactionScope(TransactionScopeOption.Required)) { using (SqlConnection conn = new SqlConnection("...")) { conn.Open(); SqlCommand sqlCommand = new SqlCommand(); sqlCommand.Connection = conn; sqlCommand.CommandText = ... sqlCommand.ExecuteNonQuery(); ... } scope.Complete(); } Starting with EF6.0 we have a new way to work with transactions. The new approach is based on Database.BeginTransaction(), Database.Rollback(), Database.Commit(). Yes, no more TransactionScope.
In the followi…

GET call of REST API that contains '/'-slash character in the value of a parameter

Let’s assume that we have the following scenario: I have a public HTTP endpoint and I need to post some content using GET command. One of the parameters contains special characters like “\” and “/”. If the endpoint is an ApiController than you may have problems if you encode the parameter using the http encoder.
using (var httpClient = new HttpClient()) { httpClient.BaseAddress = baseUrl; Task<HttpResponseMessage> response = httpClient.GetAsync(string.Format("api/foo/{0}", "qwert/qwerqwer"))); response.Wait(); response.Result.EnsureSuccessStatusCode(); } One possible solution would be to encode the query parameter using UrlTokenEncode method of HttpServerUtility class and GetBytes method ofUTF8. In this way you would get the array of bytes of the parameter and encode them as a url token.
The following code show to you how you could write the encode and decode methods.
publ…