Skip to main content

How to replicate Azure Blobs

In today post we will talk about how we can replicate content in multiple storage accounts of Azure.
If you remember the last post that I had, this one is related to it and describes the replication process.
First of all
Why we need to do something like this?
There are cases when we have a system that is distributed on multiple data centers and we want to be able to have the specific content in all locations around the globe. We don’t want a user from China to requests binary content from Europe if the system is deployed in China also. We could use CDNs but CDNs don’t have support for Shared Access Signature (SAS)
Beside this, if we have a lot of clients that will requests the same blob content than we will have a problem. A blob can handle maximum 500 requests and the bandwidth that is available per blob is around 60MB. This mean that if we have 100 clients, that wants to download the content with 1MB each than... will have a small problem.
When we talk about replication don’t imagine that replication needs to be done on different datacenters. The content can be replicated in the same datacenters on different blobs under the same storage account or using different storage accounts.
Below, we can see a diagram that present a replication schema that is:

  • On two storage accounts in the main datacenters
  • On four other datacenters (one storage account per data centers)


The secret around replication action is ‘StartCopyFromBlobAsync’. This method allow us to trigger a copy action from a blob to another blob. In this way we can easily copy all the content of a blob (including properties, metadata and payload) to another blob without having manually to manage each step.
Once the action is triggered, you will need to check the async operation status and make sure that the copy action finished with success.
await destinationBlob.StartCopyFromBlobAsync(new Uri(sourceBlobUri));

  • The status that is returned by can be:
  • CopyStatus.Aborded – the copy action was aborted
  • CopyStatus.Failed – an error occurred
  • CopyStatus.Pending – copy is still in progress
  • CopyStatus.Success – copy was finished

We don’t have any control at how long the copy action will take. Because of this we need to wait the action to finish and check the status code. As input uri, we can even use a secure blob storage that has a SAS token.
public class BlobReplication
    {       
        public async Task ReplicateBlobAsync(string blobSasUri, CancellationToken cancellationToken = default(CancellationToken))
        {
            List<ICloudBlob> destBlobs = new List<ICloudBlob>();

            try
            {
                var sasUri = new Uri(blobSasUri);

                var blobClient = new CloudBlobClient(sasUri);

                ICloudBlob sourceBlob = blobClient.GetBlobReferenceFromServer(sasUri);

                var startCopyTasks = new List<Task<ICloudBlob>>();

                foreach (var destConnectionString in configuration.ReplicationStorageAccounts.Select(c => c.ConnectionString))
                {                    
                    var destStorageAccount = CloudStorageAccount.Parse(destConnectionString);                    
                    if (destStorageAccount.BlobEndpoint.ToString() != storageAccount.BlobEndpoint.ToString())
                    {
                        var task = StartReplicationAsync(sourceBlob, blobSasUri, destStorageAccount, cancellationToken);
                        startCopyTasks.Add(task);
                    }
                }

                await Task.WhenAll(startCopyTasks);
                destBlobs = startCopyTasks.Select(t => t.Result).ToList();
                
        await WaitForBlobCopy(destBlobs, cancellationToken);
            }
            catch (OperationCanceledException)
            {                
                foreach (var dest in destBlobs)
                {
                    try
                    {
                        var destBlob = dest.Container.GetBlobReferenceFromServer(dest.Name);

                        if (destBlob.CopyState.Status == CopyStatus.Pending)
                        {
                            destBlob.AbortCopy(destBlob.CopyState.CopyId);
                        }

                    }
                    catch (Exception)
                    {
            ...
                    }
                }
                throw;
            }
        }

        private async Task<ICloudBlob> StartReplicationAsync(ICloudBlob sourceBlob, string sourceBlobSasUri, CloudStorageAccount destStorageAccount,
            CancellationToken cancellationToken = default (CancellationToken))
        {
            string containerName = sourceBlob.Container.Name;

            CloudBlobContainer destContainer = await destStorageAccount.GetBlobContainerAsync(containerName);

            ICloudBlob destBlob;    
            if (sourceBlob.Properties.BlobType == BlobType.BlockBlob)
            {
                destBlob = destContainer.GetBlockBlobReference(sourceBlob.Name);
            }
            else
            {
                destBlob = destContainer.GetPageBlobReference(sourceBlob.Name);
            }

            await destBlob.StartCopyFromBlobAsync(new Uri(sourceBlobSasUri), cancellationToken);

            return destBlob;
        }

        private async Task WaitForBlobCopy(List<ICloudBlob> blobs, CancellationToken cancellationToken = default(CancellationToken))
        {
            Dictionary<string, int> failedAttempts = blobs.ToDictionary(k => k.Uri.ToString(), v => 0);
            bool pendingCopy = true;

            while (pendingCopy)
            {
                cancellationToken.ThrowIfCancellationRequested();

                pendingCopy = false;

                foreach (var dest in blobs)
                {
                    pendingCopy |= await IsCopyInProgress(dest, failedAttempts, cancellationToken);
                }

                if (pendingCopy)
                {
                    await Task.Delay(configuration.ReplicationStatusPollingInterval, cancellationToken);
                }
            }
        }

        private async Task<bool> IsCopyInProgress(ICloudBlob dest, IDictionary<string, int> failedAttempts, CancellationToken cancellationToken = default(CancellationToken))
        {
            bool pendingCopy = false;
            var destBlob = await dest.Container.GetBlobReferenceFromServerAsync(dest.Name, cancellationToken);

            switch (destBlob.CopyState.Status)
            {
                case CopyStatus.Aborted:
                case CopyStatus.Failed:

                    int numberOfFailes;
                    if (failedAttempts.TryGetValue(destBlob.Uri.ToString(), out numberOfFailes) &&
                        numberOfFailes < configuration.MaximumReplicationRetries)
                    {
                        pendingCopy = true;

                        await destBlob.StartCopyFromBlobAsync(destBlob.CopyState.Source, cancellationToken);
                        failedAttempts[destBlob.Uri.ToString()] = numberOfFailes + 1;
                    }
                    else
                    {
                        throw new Exception("Failed to replicate blob " + destBlob.Name);
                    }
                    break;
                case CopyStatus.Pending:
                    pendingCopy = true;
                    break;
                case CopyStatus.Success:
                    pendingCopy = false;
                    break;
                default:
                    throw new Exception("Undefined copy state");
            }

            return pendingCopy;
        }      
    }
In the above sample code we can see how to replicate the content to multiple location and how to ensure that copy action had finished with success. There is only one case that is not take into account: What happens if the process ends before replication action ends. How we can check that the replication was done with success?
This topic will be discussed tomorrow.
Enjoy!

Comments

Popular posts from this blog

How to check in AngularJS if a service was register or not

There are cases when you need to check in a service or a controller was register in AngularJS.
For example a valid use case is when you have the same implementation running on multiple application. In this case, you may want to intercept the HTTP provider and add a custom step there. This step don’t needs to run on all the application, only in the one where the service exist and register.
A solution for this case would be to have a flag in the configuration that specify this. In the core you would have an IF that would check the value of this flag.
Another solution is to check if a specific service was register in AngularJS or not. If the service was register that you would execute your own logic.
To check if a service was register or not in AngularJS container you need to call the ‘has’ method of ‘inhector’. It will return TRUE if the service was register.
if ($injector.has('httpInterceptorService')) { $httpProvider.interceptors.push('httpInterceptorService&#…

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

[Post-Event] Codecamp Conference Cluj-Napoca - Nov 19, 2016

Last day I was invited to another Codecamp Conference, that took place in Cluj-Napoca. Like other Codecamp Conferences, the event was very big, with more than 1.000 participants and 70 sessions. There were 10 tracks in parallel, so it was pretty hard to decide at  what session you want to join.
It was great to join this conference and I hope that you discovered something new during the conference.
At this event I talked about Azure IoT Hub and how we can use it to connect devices from the field. I had a lot of demos using Raspberry PI 3 and Simplelink SensorTag. Most of the samples were written in C++ and Node.JS and people were impressed that even if we are using Microsoft technologies, we are not limited to C# and .NET. World and Microsoft are changing so fast. Just looking and Azure IoT Hub and new features that were launched and I'm pressed (Jobs, Methods, Device Twin).
On backend my demos covered Stream Analytics, Event Hub, Azure Object Storage and DocumentDB.

Title:
What abo…