This post covers how we can use Azure Data Factory to copy our data from one location to another.
Even if the copy activity might sound like a basic task that is not complex or challenging when you need to move 10 TB or 50 TB of data from one storage to another, things are becoming interesting. Reliability, resume and continue, task parallelization, data consistency are becoming essential for your project.
There are multiple solutions on the market that helps you to move data to and from Azure or between different Azure locations. Today we will focus on Azure Data Factory and how it can help us to our job better.
Azure Data Factory it is a service created to help us to work with data when we need to store, process and move it, without having to things about anything else.
Inside Azure Data Factory there we have an activity called Copy Activity that can be used to move (copy) data from source to another. When we talk about Azure Data Factory, we call:
- Source the repository from where we read data
- Sink the repository where we write data
- IR is the Integration Runtime that allows us to run/execute custom code
The copying activity inside Azure Data Factory runs inside Azure IR, meaning that we already have a scalable and reliable environment where we can execute our copy action. A flavor of Azure IR that allows us to host yourself this activity is Self-Hosted IR used when we are working with high secure VNETS or on-premises storages.
The power of IR are the activities that can be done besides the copy activity itself, that is reliable and scalable. Inside the pipeline, we can do any other activities related to data serialization, compression or data transformation and mapping. This is super useful when data needs to be consolidated in different ways at the destination repository (sink).
The list of Azure repositories that are available for copy activities can be found on the following link https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview and covers most common data repositories including (NoSQL and SQL data storages).
If you don’t need to do any data transformation, you can get with just a few clicks a solution that can move data super fast. When you need to do data transformation, you shall remember that there is native support for JSON, Text, Parquet, ORC and Avro format in combination with ZipDeflate, BZip2, Deflate and Gzip compression.
When you use Azure Data Factory, you will pay only for the resources that you are consuming at the moment when you do the copy activity. Beside this, we can specify the actions that need to be executed at the moment when an error occurs. For example, if there is an incompatibility between two data format of a specific file, you can decide to skip and log the fault.
The performance of copy activity it is well documented, allowing us to be able to estimate better how long it takes to do different actions and set clear expectation related to performance. For example, the copy activity from Azure Blob to another Azure Blob si 105Mbps when you have 8 DMUs (unit measure for Azure Data Factory).
The copying activity has support for incremental and delta copies. This means that you can copy only new content from one location to another, without having to develop a custom mechanism that detects changes.
When you copy content to/from Azure Blob Storage, you can specify how data will land on the other side. For example, you can specify if you want to persist the original file structure or to copy content in a flat location or to generate from all the files only a huge file.
Conclusion
Azure Data Factory it is a good option when we need to do copy activities from different data sources. It is helping us to focus on our needs and less on the infrastructure. For data moving from/to Azure Storage I highly recommend to take a look on Copy functionality of it.
Comments
Post a Comment