Skip to main content

Design a state machine mechanism using Windows Azure Service Bus (part 1)

In this post we try to see a possible solution to implement a simple state machine (with a simple workflow) over Windows Azure Service Bus.
Let’s imagine a system that needs to process over 1 million requests per day. Each request that enter in the system will pass through different statutes until the process will be finished. If we have simple rules, it will be very easily for us to implement a system that will process each request and change the state of our request.
This can be imagined as a simple workflow, where a request flows from one state to another. At each step, we are making executing a small action. The big question here is how we can design a solution in a way where we will don’t have any kind of scalability problems.
The biggest problem that can appear at this step is related to scalability and distribution the logic over machines. This can become a nightmare when death-locks appears or when the system will reach his limits. The costs of hardware’s that will help us to survey another day can be enormous. 
We will see a simple solution that uses Windows Azure Service Bus to resolve our problem. In the next post we see how easily we can scale this solution.
Let’s assume that each request will go through 4 states before it will process completely. Each action that need to be done implicate an external system or an internal system that was already implemented.
For this case we can use Windows Azure Service Bus with success to implement a mechanism that will send a request from one state to another. In this way we will have reliability on our site. Not only this, we will not have any kind of problem with losing the requests that has a specific state if something happens with the system.
In this first solution, we will use Windows Azure Service Bus Topic. As we already know, Windows Azure Service Bus Topic permit give us the possibility to send the same message to more than one listener. From some perspective it is very similar to message broadcast. Each listener need to listen the topic through a subscription.
What is very useful at this is what we can with a subscription. We have the possibility to have not only more than one client to our subscription, but we can define custom rules. Using these rules we will be able to define what type of messages will be accepted by our subscription.
If a subscription will accept only messages that has a property with a specific value, than we will be able to process only messages with a specific state by our subscription. After the message is processes, we can change create a new message with the new state and send it back to the Windows Azure Service Bus Topic.
The system would look something like this:

As we can see, we use the same topic to add messages that are in a new state. Each subscription will process messages on the given state. In this current implementation, we are using Windows Azure Service Bus Topics because we don’t want to use a different place to store each message that has a different state. In the same time, for each subscriber we can have more than one listener. For the states where we know that computing power is more expensive, than we can allocate more listeners.
In the next post we will see how we can make this system more scalable using Windows Azure Service Bus features.

Comments

Post a Comment

Popular posts from this blog

Windows Docker Containers can make WIN32 API calls, use COM and ASP.NET WebForms

After the last post , I received two interesting questions related to Docker and Windows. People were interested if we do Win32 API calls from a Docker container and if there is support for COM. WIN32 Support To test calls to WIN32 API, let’s try to populate SYSTEM_INFO class. [StructLayout(LayoutKind.Sequential)] public struct SYSTEM_INFO { public uint dwOemId; public uint dwPageSize; public uint lpMinimumApplicationAddress; public uint lpMaximumApplicationAddress; public uint dwActiveProcessorMask; public uint dwNumberOfProcessors; public uint dwProcessorType; public uint dwAllocationGranularity; public uint dwProcessorLevel; public uint dwProcessorRevision; } ... [DllImport("kernel32")] static extern void GetSystemInfo(ref SYSTEM_INFO pSI); ... SYSTEM_INFO pSI = new SYSTEM_INFO(

Azure AD and AWS Cognito side-by-side

In the last few weeks, I was involved in multiple opportunities on Microsoft Azure and Amazon, where we had to analyse AWS Cognito, Azure AD and other solutions that are available on the market. I decided to consolidate in one post all features and differences that I identified for both of them that we should need to take into account. Take into account that Azure AD is an identity and access management services well integrated with Microsoft stack. In comparison, AWS Cognito is just a user sign-up, sign-in and access control and nothing more. The focus is not on the main features, is more on small things that can make a difference when you want to decide where we want to store and manage our users.  This information might be useful in the future when we need to decide where we want to keep and manage our users.  Feature Azure AD (B2C, B2C) AWS Cognito Access token lifetime Default 1h – the value is configurable 1h – cannot be modified

What to do when you hit the throughput limits of Azure Storage (Blobs)

In this post we will talk about how we can detect when we hit a throughput limit of Azure Storage and what we can do in that moment. Context If we take a look on Scalability Targets of Azure Storage ( https://azure.microsoft.com/en-us/documentation/articles/storage-scalability-targets/ ) we will observe that the limits are prety high. But, based on our business logic we can end up at this limits. If you create a system that is hitted by a high number of device, you can hit easily the total number of requests rate that can be done on a Storage Account. This limits on Azure is 20.000 IOPS (entities or messages per second) where (and this is very important) the size of the request is 1KB. Normally, if you make a load tests where 20.000 clients will hit different blobs storages from the same Azure Storage Account, this limits can be reached. How we can detect this problem? From client, we can detect that this limits was reached based on the HTTP error code that is returned by HTTP