Skip to main content

Data moving to/from Azure Storage using Azure Data Factory and Copy Activity

This post covers how we can use Azure Data Factory to copy our data from one location to another.
Even if the copy activity might sound like a basic task that is not complex or challenging when you need to move 10 TB or 50 TB of data from one storage to another, things are becoming interesting. Reliability, resume and continue, task parallelization, data consistency are becoming essential for your project.
There are multiple solutions on the market that helps you to move data to and from Azure or between different Azure locations. Today we will focus on Azure Data Factory and how it can help us to our job better.
Azure Data Factory it is a service created to help us to work with data when we need to store, process and move it, without having to things about anything else.

Inside Azure Data Factory there we have an activity called Copy Activity that can be used to move (copy) data from source to another. When we talk about Azure Data Factory, we call:
  • Source the repository from where we read data
  • Sink the repository where we write data
  • IR is the Integration Runtime that allows us to run/execute custom code

The copying activity inside Azure Data Factory runs inside Azure IR, meaning that we already have a scalable and reliable environment where we can execute our copy action. A flavor of Azure IR that allows us to host yourself this activity is Self-Hosted IR used when we are working with high secure VNETS or on-premises storages.
The power of IR are the activities that can be done besides the copy activity itself, that is reliable and scalable. Inside the pipeline, we can do any other activities related to data serialization, compression or data transformation and mapping. This is super useful when data needs to be consolidated in different ways at the destination repository (sink).
The list of Azure repositories that are available for copy activities can be found on the following link https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview and covers most common data repositories including (NoSQL and SQL data storages).
If you don’t need to do any data transformation, you can get with just a few clicks a solution that can move data super fast. When you need to do data transformation, you shall remember that there is native support for JSON, Text, Parquet, ORC and Avro format in combination with ZipDeflate, BZip2, Deflate and Gzip compression.

When you use Azure Data Factory, you will pay only for the resources that you are consuming at the moment when you do the copy activity. Beside this, we can specify the actions that need to be executed at the moment when an error occurs. For example, if there is an incompatibility between two data format of a specific file, you can decide to skip and log the fault.
The performance of copy activity it is well documented, allowing us to be able to estimate better how long it takes to do different actions and set clear expectation related to performance. For example, the copy activity from Azure Blob to another Azure Blob si 105Mbps when you have 8 DMUs (unit measure for Azure Data Factory).

The copying activity has support for incremental and delta copies. This means that you can copy only new content from one location to another, without having to develop a custom mechanism that detects changes.
When you copy content to/from Azure Blob Storage, you can specify how data will land on the other side. For example, you can specify if you want to persist the original file structure or to copy content in a flat location or to generate from all the files only a huge file.

Conclusion

Azure Data Factory it is a good option when we need to do copy activities from different data sources. It is helping us to focus on our needs and less on the infrastructure. For data moving from/to Azure Storage I highly recommend to take a look on Copy functionality of it. 

Comments

Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Entity Framework (EF) TransactionScope vs Database.BeginTransaction

In today blog post we will talk a little about a new feature that is available on EF6+ related to Transactions.
Until now, when we had to use transaction we used ‘TransactionScope’. It works great and I would say that is something that is now in our blood.
using (var scope = new TransactionScope(TransactionScopeOption.Required)) { using (SqlConnection conn = new SqlConnection("...")) { conn.Open(); SqlCommand sqlCommand = new SqlCommand(); sqlCommand.Connection = conn; sqlCommand.CommandText = ... sqlCommand.ExecuteNonQuery(); ... } scope.Complete(); } Starting with EF6.0 we have a new way to work with transactions. The new approach is based on Database.BeginTransaction(), Database.Rollback(), Database.Commit(). Yes, no more TransactionScope.
In the followi…

GET call of REST API that contains '/'-slash character in the value of a parameter

Let’s assume that we have the following scenario: I have a public HTTP endpoint and I need to post some content using GET command. One of the parameters contains special characters like “\” and “/”. If the endpoint is an ApiController than you may have problems if you encode the parameter using the http encoder.
using (var httpClient = new HttpClient()) { httpClient.BaseAddress = baseUrl; Task<HttpResponseMessage> response = httpClient.GetAsync(string.Format("api/foo/{0}", "qwert/qwerqwer"))); response.Wait(); response.Result.EnsureSuccessStatusCode(); } One possible solution would be to encode the query parameter using UrlTokenEncode method of HttpServerUtility class and GetBytes method ofUTF8. In this way you would get the array of bytes of the parameter and encode them as a url token.
The following code show to you how you could write the encode and decode methods.
publ…