Wednesday, October 26, 2016

Where can I backup my content - Azure File Storage

Nowadays taking pictures is so easy, you can end up easily with 20-30 shots per day. After a few years of using external HDD as backups I realize that is a time for a change.
Until now I used OneDrive to backup pictures from my phone and 3 external HDD that were putted in mirror. This setup works great as long you have enough storage and you external HDDs.
But now I ended up with the OneDrive storage at 99.8% full (from 30GB) and with external storage almost full also.

What I need?
I looked up for a solution that is cheap, can be accessed from anywhere I don't need to install custom software on my machine to be able to access it. The ideal solution would be to be able to attach the storage as a partition.
Buying external HDD s is not a solution anymore for me anymore. I would prefer a cloud or storage provider that could offer me a reliable storage for a good price.

Azure Offer
An interesting offer is coming from Microsoft Azure, more exactly from Azure File Storage. The service is part of Azure Storage and allow us to access Azure Storage as a file system. Having the classical folders, files and so on.

Azure File
The concept is simple and powerful. Under your Azure Subscription you can create a Storage Account that can be used to store files under a directory structure. You can upload, access, remove content from any location.
From technical perspective, each 'drive' (if we can call in this way), that you create under Storage Account is called Shared. The maximum size is 5TB, that is more than enough. Under a Share you can have as many directories you want. The number of files is not limited, but the maximum size of a file is limited to 1TB. When you create a new file share you need to specify the quota limit (max size of the file share). This quota can be changed anytime without having to recopy the content.
For my personal needs, this is more and enough. If I would need more than 5TB, than I can create multiple Shares under the same Storage Account.

This is one of the things that attracted me. You have multiple tires that you can use. From 3 replicas in the same Azure Region you can go with more complex ones where content is replicated in another Azure Region where read access is also available (RA-GRS).
Based on how often you access your data there are two types of tires that can be used Cool or Hot tier. Hot tier is used when the content is access and updated often. For backups scenarios, like in my case, Cold storage is a better option. Usually this kind of backups are not accessed often. In my case, I don't read/access all the more often than one time per month.
For this kind of backups the best solution is a Cold Storage that is Geo-Redundant Storage (GRS). This means that storage is replicated in two different Azure Regions. In each Azure Region that are 3 different copies of your content.
When you calculate the cost you need to take into account 3 different things

  • Storage cost (€1.69 per month for 100GB)
  • Transaction cost (€0.16 or €0.0084 for 100.000, based on transaction type)
  • Data Read (€0.0084 per GB) or Data Write (€0.0042 per GB)
For a normal use I calculated that I will never pay more than €1.8 per months for 100GB. This means that the cost per year is under €18, in the context where the content is replicated in two different locations and is accessible from anywhere.

Content access
The content can be accessed directly from Azure Portal, but of course you don't want to do something like this when you have 50.000 files. 
On top of this we have Microsoft Azure Storage Explorer. This nice tool allow us to access content directly from desktop. There is full support for Windows, Linux and Mac. 2The experience is nice and similar with the one that you have from File Explorer.
For access your content from this tools you can use your Azure account credentials or Azure Storage account name and access key. If it is your first time when you use Azure Storage I recommend to check a nice tutorial provided by Microsoft - About Azure storage accounts.

This is not all. You can attach the file storage as a partition on your system. With only one command executed in command line you can do this. This magic can happens because Azure Files Storage has support for SMB 3.0 protocol. This means that you can mount the file share on Windows 7+ or Windows Server 2008+ operation system or Linux machines.
// Windows Machine
net use 
     v: \\\pictures 

// Linux Machine
sudo mount -t 
     cifs // 
     [mount point] -o vers=3.0,

Simple like this, you can attach the Azure File Storage as a partition.

Azure File Storage is a good option if you need to backup you content. The price per month is low and being able to attach it as a partition to my computer makes this solution the best one on the market.

Friday, October 21, 2016

Let’s be inventive on how we can process and collect data

Let's discover together another approach to collect and transform information that is send by devices.

In the world of smart devices, devices are more and more chatty.  Let’s assume that we have a smart device that needs to sends every 30 seconds a Location Heartbeat that contains <device ID, GPS location, time sample>. Worldwide we have 1.000.000 devices that sends this information to our backend.
At global level, backend runs on 4 different Azure Regions, with an equal distribution of devices. This means that on each instance of our backend there will be 250.000 devices that sends heartbeats with their locations.
From a load perspective, it means that on each Azure Region there are around 8.300 heartbeats requests every second. 8K messages per second might be or not an acceptable load, it depends on what actions we need to do for each request.

From the end user there are two clear requirements that needs to be full fit:

  • Full history of device locations is stored for audit
  • Last device location needs to be accessible all the time

The first requirement is pretty relax. From implementation point of view we need to persist all the location heartbeats in a repository. The repository can be a file base repository.
The second requirement is a little more complex. It requires to have available all the time the last location of the device.

Classical solution
The standard solution for this situation is to have an event messaging system that aggregates all the location heartbeats from devices. On Azure, we can use with success Azure Event Hub, that is able to ingest high throughput.
Behind it we need a computation unit that can scale up easily. The main scope of this unit shall be to process each location heartbeat by dumping the information in a storage for audit and update the last location of the device.
For processing we can use with success Worker Roles or Azure Service Fabric. In both situations, you shall expect that the processing action will take around 0.4-0.5s (dumping data and update the last location). This means that you will need at least 16 instances of Worker Roles that will do this job.
The audit of the device location can be dumped easily in blobs. If you don’t want to do this by hand you can use Azure Event Hubs Archive ( This new feature offered by Azure, can do this job for you, dumping all the events directly in blobs. This solution is applicable as long as you don’t need to do a transformation of messages.  

For storing the last location, there are many options. The faster one is to have in memory cache like NCache and update the last location. The downside of this approach is that you will need to access this information and an API shall be available or at a specific time interval a dump to Azure Table or SQL shall be done.
You might ask why I not propose Azure Redis Cache or another cache service. When you do a lot of writes, a cache system doesn’t behave as you expect. In the past we had problems with such solution where the read and write latency has 2-5s because of heavy writes (yes, there were a lot of write operations).

A better solution is with Azure Tables. Each raw stores the last location of a device. Each time when a new location is received we update the location information. You can replace Azure Table with SQL if the reports that you need to do are complex or you need to be able to generate the reports on the fly (even for this situations, Azure Table can be a good friend).

Even if we try to optimize the code, writing the last location of devices is expensive. You need to identify the device, get the table, write the content. This means that even if you reduce the time from 0.4-0.5s to 0.2-0.3 it will still be expensive.
To be able to consume and process 8.300 location heartbeats per second it will still be costly. Let’s assume that you manage to process 500 on a Worker Role. This would translate in at least 17-18 instances of worker roles.

Identify the bottleneck
We need to try to see how we can use Azure Services that are available now on the market in our advantages. We need to try to reduce our custom code. For long term this means that will have lowest cost for maintenance and support – number of bugs will be also lower.
Our assumption is that we cannot use Azure Event Hub Archive because there is some transformation that needs to be done to store in the agreed audit format.

First thing that we shall do is to separate these two requirements in different modules. The first module would create the audit data and the other one will store the last location of the device.

Remarks: Whatever you do, you shall use Event Processor Host to consume messages from Event Hub. This will ensure you that you will have:

  • High throughout
  • Scalable solution
  • Failover between nodes

If we run separately this two modules, we observe that processing the location heartbeats only for audit is extremely fast. On a single worker role, we are able to process 4000-5000 location heartbeats per second. In contrasts with the action of updating the last device location that is an expensive action.
The problem is not coming from Azure Table, where latency is low. The difference is that for audit you only transform the message and dump them, where strategies like buffering can be used to optimize this kind of actions. Whatever we do, we cannot process more than max 600 location heartbeats per second on each computation instance.

Let’s be inventive
We observed that creating the audit data is a low consuming action. Storing the last location of device is expensive. If we are looking again at last device location, we realize that this action could be done directly by device.
Why not making the device to update directly his own Azure Table Row with his last known location? In this way we don’t have to process the data and update the last location using backend resources.

From access perspective, Azure Table Shared Access Signature (SAS) allows us to offer granularity access at partition and row key level. Each device can have access only to his own raw from table.

In the end each device ends up doing a request to Azure Table to update the last known location and another requests to Azure Event Hub to push the location heartbeat for audit. On the backend the load is decreased drastically. We need only the Worker Roles instances that create our audit data. In Azure Tables we will have the last location of each device that is available for reporting or other type of actions.
At device level things change a little bit. Now, for each location heartbeat, a device needs to do two different requests. This means that the device will consume more bandwidth, a little bit more CPU and more bandwidth. This approach is valid for devices that have a good internet connection (cheap) and no CPU strict limitations.
In this way we move the load from one place to another, increasing the noise on the internet (smile).

Another approach is to have a dedicated Azure Table for each device, where devices add new rows all the time. We will not explore this solution, that might be interesting as long as we don't need complex reports.

To solve a business problem there are all the time multiple solutions and approaches. There are times, when is a good approach to make a step back and see how we can use services that are available on the market to solve different problems.
Making direct calls to different services might be a better solution, but never forget about the cost of making that request at device level.

Tuesday, October 18, 2016

Windows Docker Containers can make WIN32 API calls, use COM and ASP.NET WebForms

After the last post, I received two interesting questions related to Docker and Windows. People were interested if we do Win32 API calls from a Docker container and if there is support for COM.

WIN32 Support
To test calls to WIN32 API, let’s try to populate SYSTEM_INFO class.
        public struct SYSTEM_INFO
            public uint dwOemId;
            public uint dwPageSize;
            public uint lpMinimumApplicationAddress;
            public uint lpMaximumApplicationAddress;
            public uint dwActiveProcessorMask;
            public uint dwNumberOfProcessors;
            public uint dwProcessorType;
            public uint dwAllocationGranularity;
            public uint dwProcessorLevel;
            public uint dwProcessorRevision;
        static extern void GetSystemInfo(ref SYSTEM_INFO pSI);
        SYSTEM_INFO pSI = new SYSTEM_INFO();
        GetSystemInfo(ref pSI);

The sample can be found on Github and can be run easily using the following commands:
docker build -f Docker.Win32.Image.txt -t win32 .
docker run win32
As we can see, we were able to make with success a call to a WIN32 API.

Code sample:

How is this possible?
Starting from Windows Server 2016 (and Windows 10 Anniversary Upgrade), Windows can run containers in two different ways.
The first mechanism is using Windows Containers, that runs on top of Host OS kernel, sharing the same kernel between containers. This means that the kernel, binary libraries and many more are shared between containers. On top of this, all the containers are affected and use the same kernel.
In contrast with Windows Containers, Hyper-V Containers runs directly on top of Hyper-V. Each container uses his own kernel image. This allows us to run multiple containers, each of them using a different version of the OS. We have more flexibility, but the footprint might be a little higher.

Docker for Windows is using Hyper-V containers. This means for each container that we run, we will have our own instance of kernel, that is different from the Host version. Having our own kernel instance, give us access to all information, including the Win32 API, as long as the Docker image that we use has that Win32 API available.

COM Support
The second topic is related to COM. The short response is yes, we can make COM calls. Things are more complex at setup level. We need to be sure that that COM is register and available in our image. In the sample provided on GitHub, I register directly from C# code the COM that I use.

        public static void RegistarDlls(string filePath)
                string fileinfo = "/s" + " " + "\"" + filePath + "\"";
                Process reg = new Process();
                reg.StartInfo.FileName = "regsvr32.exe";
                reg.StartInfo.Arguments = fileinfo;
                reg.StartInfo.UseShellExecute = false;
                reg.StartInfo.CreateNoWindow = true;
                reg.StartInfo.RedirectStandardOutput = true;

COM call
            const string guidFWPolicy2 = "{E2B3C97F-6AE1-41AC-817A-F6F92166D7DD}";
            const string guidRWRule = "{2C5BC43E-3369-4C33-AB0C-BE9469677AF4}";
            Type typeFWPolicy2 = Type.GetTypeFromCLSID(new Guid(guidFWPolicy2));
            Type typeFWRule = Type.GetTypeFromCLSID(new Guid(guidRWRule));
            INetFwPolicy2 fwPolicy2 = (INetFwPolicy2)Activator.CreateInstance(typeFWPolicy2);
            INetFwRule newRule = (INetFwRule)Activator.CreateInstance(typeFWRule);
            newRule.Name = "MabuAsTcpLocker_OutBound_Rule";
            newRule.Description = "Block outbound traffic  over TCP port 80";
            newRule.Protocol = (int)NET_FW_IP_PROTOCOL_.NET_FW_IP_PROTOCOL_TCP;
            newRule.RemotePorts = "8888";
            newRule.Direction = NET_FW_RULE_DIRECTION_.NET_FW_RULE_DIR_OUT;
            newRule.Enabled = true;
            newRule.Profiles = 6;
            newRule.Action = NET_FW_ACTION_.NET_FW_ACTION_BLOCK;

As we can see above, we can make WIN32 and COM calls from Docker.

The last question that I received yesterday is related to ASP.NET Web Forms and if we can run them in Docker Containers. Microsoft created a Docker image for ASP.NET applications ( . Inside it, we can run any kind of ASP.NET applications the MVC to Web Forms or other kind of applications. The IIS is already configured, so we don’t need to configure any bindings to it.
The sample code for ASP.NET WebForms can be found on GitHub:

In comparison with the previous samples we used ‘microsoft/aspnet’ Docker image. Don’t forget to specify in the browser the IP of the container, when you want to access the web application. The IP of the Docker container can be obtained using the following command:
docker inspect -f "{{ .NetworkSettings.Networks.nat.IPAddress }}" webforms

In this post we validated that we can do the following things in a Docker Container:

  • Make Win32 calls
  • Use COM Interop
  • Run a ASP.NET WebForms application

Monday, October 17, 2016

Run native .NET application in Docker (.NET Framework 4.6.2)

The main scope of this post is to see how we can run a legacy application written in .NET Framework in Docker.

First of all, let’s define what is a legacy application in our context. By a legacy application we understand an application that runs .NET Framework 3.5 or higher in a production environment where we don’t have any more the people or documentation that would help us to understand what is happening behind the scene.
In this scenarios, you might want to migrate the current solution from a standard environment to Docker. There are many advantages for such a migration, like:

  • Continuous Deployment
  • Testing
  • Isolation
  • Security at container level
  • Versioning Control
  • Environment Standardization

Until now, we didn’t had the possibility to run a .NET application in Docker. With .NET Core, there was support for .NET Core in Docker, but migration from a full .NET framework to .NET Core can be costly and even impossible. Not only because of lack of features, but also because once you modify an existing application phases like V&V (Verification and Validation) can be time consuming.

Docker support on Windows Server 2016
Now, with Windows Server 2016 and native support for containers, things started to change. On Windows Server 2016, we can have a console application that runs native in Docker.
The most important thing at this level is that we don’t need to modify our existing code to run in Docker. Based on what kind of resources we use we might need to map different folders, endpoints or install on the Docker image some applications and services. This are the same steps that we do on a classical system – but in this case we need to configure Docker and not a machine.

Initial Use Case
Let’s take as example a consoler application that is written in .NET 4.6.1. We need to be able to take this application and run in in Docker, without changing a line of code in our console application.

In the above example we have Legacy.Code assembly that runs a custom code in StockCheck class. Legacy.Runner is our console application that calls StockCheck. Nothing special or exotic.

Docker Setup
To be able to do this migration we will need Windows Server 2016. For PoC and playground, I recommend to create an instance of Windows Server 2016 on Azure. You will be able to find an image with Containers.
Once you have a machine with Windows Server 2016, you will need to install and setup Docker. There is a great article that describes all this steps that needs to be done on MSDN -
You should continue only when you manage to run with success the Microsoft/sample-dotnet image, presented in the above link.

Docker Image for .NET Framework
In Docker Hub, we can find numerous images created by Microsoft, even for .NET Framework (4.6.2, 3.5, ASP.NET and so on). The one that we need for this demo is.NET Framework 4.6.2 -
To pull the right image from Docker we need to run the following command:
docker pull microsoft/dotnet-framework
Now, we have the image on our system and we can play with it. If we want we can run we can do this from Power Shell using the following command:
docker run microsoft/dotnet-framework
As we can see in below picture, we run the image. Nothing happen because in this moment we don’t have nothings deployed.

Create a custom image with our code
On GitHub, I shared the base project, that you can use to run this demo - Once you downloaded it, you will need to build the project. To be able to build it with success you will need also Docker Tools for Visual Studio.
At solution folder, you can find Dockerfile.txt. This contains the commands and actions that needs to be executed by Docker to build the image. The first line specifies what image shall be used to create a new image (FROM). The second like specifies to Docker to copy the output of the build to root folder of the image (COPY). The last line specifies the entry point of our image.
Do be able to work you will need to run the below image from the same location where the .sln project is. Otherwise, you need to modify the COPY action to use a valid path from where our application is copied.
docker build -f Dockerfile.txt -t legacy .
Because we added .txt prefix to Dockerfile, we need to specify the input instruction file. ‘-t’ adds a custom tag to our new image. This tag will be used by us later on to run this image. The above commands creates a new image, adding our code to it.

Run our custom image
Now, we have our image created and we are ready to run our new image for the first time.
docker run legacy
Congratulation, you have a legacy application that runs in Docker, using .NET Framework 4.6.2

Useful commands
Inspect image:
docker inspect legacy

Remove docker image legacy:
docker rmi legacy --force

Remove docker container legacy
docker rm legacy

Remove all containers
docker rm $(docker ps -a -q)

For developers it is the perfect moment when we need to look at containers and Dockers. Even if we work with systems that are production and use .NET Framework <=4.6.2, we can still use Docker with success. Based on our needs we might be able to migrate them to containers pretty easily.

Friday, October 14, 2016

Vampires | How to clean you Azure Subscription

Playing and testing different scenarios and use cases in Azure can generate a lot of ‘garbage’ under your subscriptions.

When I say garbage I’m referring to different resources that you create but forget to remove. Don't think at resources like Storage, Web Apps or Worker Roles, but many times when you remove things in odd order, you can end up with some resources that are allocated, but you don’t use anymore and you forget about them. For example the storage of a VM, or Traffic Manager that was used for a Web App.
I named 'vampires' this kind of resources. Why? They consume small amount of money every month. Even the value is not high, you can end up at the end of the month with $10 or $20 consumed on them. They are like electrical vampires that we have in our house – a TV that is in standby, a phone/tablet charger, audio system.

Best Practice
The best practice is very clear and simple. Remove all the time resources that you don’t use anymore. But, sometimes you are in hurry or you don’t have time to double check if you removed everything.
In this situations, you can use a simple rule that will help you to save time and avoid this kind of situations. Add all resources that you create under an Azure Subscription for a demo or for a specific test under the same Azure Resource Group.
A Resource Group is like a container where you can put together resources for the same solution. Don’t forget to give a meaningful name to the Resource Group. Otherwise, you will don’t know what was the purpose of it.
After you finish playing with a demo or with an Azure Solution, the only thing that you need to do is to remove the Resource Group. Removing the Resource Group will remove all the resources that were created under it.

But, what you do if you realize that after a few months playing with Azure, you have under your subscription a lot of resources that you forgot about or a lot of resource groups that are empty.
Well, deleting each of them would take a lot of time. It is a boring task and you don’t want to do this. The other solution is to run a power shell script that remove all the Resource Groups that you have under you Azure Subscription. Let’s call it CleanStartScript.

$selectedSubscription = Get-AzureSubscription -Current
Write-Host Subscription in use: $selectedSubscription.SubscriptionName

Write-Host Start Deleting all Resource Groups

$selectedResourceGroups = Get-AzureRmResourceGroup
foreach($currentResourceGroup in $selectedResourceGroups)
  Write-Host Deleting Resource Group: $currentResourceGroup.ResourceGroupName
    Remove-AzureRmResourceGroup -Name $currentResourceGroup.ResourceGroupName -Force -Verbose
  Write-Host Deleted Resource Group:  $currentResourceGroup.ResourceGroupName

Write-Host Ended Deleting all Resource Groups
The scrips can be found on GitHub:

Don't forget to select the right subscription before running this script. Do do this, you might find the following power shell scrips usefull:

  • Get-AzureSubscription -Current |> Get Current Subscription
  • Login-AzureRmAccount |> Login to Azure account
  • Get-AzureAccount |> Get A list of all Azure Subscription under the current account
  • Get-AzureSubscription -SubscriptionId XXX | Select-AzureRmSubscription |> Set the given subscription as the current one 

Tuesday, October 4, 2016

Team Service & TFS - Release Management and Multiple Environment Support

I was impressed to discover a new feature on Team Service and TFS 2015 - Release Management.
Release Management allow us two things that can improve our release procedures:
  • Define multiple environments and push the binaries (content) from one environment from another
  • Request manual approve before pushing the packages from one environment to another
Environments Definition
Once the CI finish the build, we can specify in what environment we want to push the output. Once the content is pushed to a specific environment, we can run any kind of steps, from simples once like Integration Tests to more complex one, where people interaction is required. 
Team Service support us to define one or more release definitions. Each release definitions represents the tasks that needs to be run on each environment. It can be a simple script that change a version, or adding/removing content from the build. In the end can be any action that needs to be executed when we deploy content to a specific environment.  

From one another we can push the content to another environment once:
  • The build is with success 
  • Time scheduler
  • Success of a previous deployment
The nice thing is that we can add a new environment directly from the UI. For each env. we can specify what are the tasks that we want to execute. Don't forget that once you defined an environment it might be more easily to clone the current environment and modify it based on your needs. Usually, at steps that needs to be executed on each environment there is a delta, but you might have some steps that are common.

On top of this we can deploy in the same time the output of a build or from another environment to one or multiple environments. This means that you can deploy in two env. (QA and Pre1Production). A similar thing that can be done on the other side - push the package to an env only of we are 'green' on multiple env.

For each environment we can have variables. In this way we can have different configuration for each separate env. From the usability perspective, the nice thing is that the list of variables that are defined contains also the value for all environment. In this way it is very easy to check or to compare what values are set.
From the security perspective, we are allowed to specify what users or group of users has access to each env. In this way you can have the dev. env. where only devs can access it n QA env. where only the testing team can access it. And so on other environments. 

Manual Approve
There are times when you cannot configure an automatic mechanism to deploy from one env. to another. Very often, you need a person that decided if a build goes to pre-production for example. Or you need to QA Lead to decide if the current build that is deployed in the QA Environment can go to the next environment or not.
For this kind of scenarios, we are allowed to specify that there is a need of a manual release approval. A list of persons or a group needs to be specified. This persons can approve a deployment before beeing push to it or after is deployed (pre/post-deployment approve).
In this way we can ensure that only if a specific person approves a release, the content is pushed to Pre-production or even to Production environment. 

On top of this you can specify a list of people that needs to approve the action. The options in this case are:
  • Any one user
  • All users in sequential order
  • All users in any order
In this way you can map any process flow and gates over it.

From a DevOps point of view, more and more features are added to TFS each month. With features like this, even complex flows and procedures can be mapped easily.

Monday, October 3, 2016

MVP for another year

Happy to be part of Microsoft MVP for another year. The news I received in Atlanta, during Microsoft Ignite Conference, when I was exited to receive this great new.
Proud and happy to be part of this of a group of people, that is open and ready to offer support for online and local communities with their technical expertise.

As for the last year, I plan to sustain the local community. On top of this I have two new initiatives on my radar:

  • One blog post per week that will review Azure Service (more details next week)
  • More activity on the online communities