Skip to main content

Azure Data Factory (Day 28 of 31)

List of all posts from this series: http://vunvulearadu.blogspot.ro/2014/11/azure-blog-post-marathon-is-ready-to.html

Short Description 
Azure Data Factory is one of the services that is part of data analytics functionality offered by Microsoft Azure. This service can be used with success when you need to orchestrate and manage data movement and different data transformation.


Main Features 
Move data between different sources
Using Azure Data Factory you can move data between different storages, from the file based one, to SQL and even NoSQL storages. This can be very useful when you need to change the location of data to be able to process or store it for long time.
On-premises and Cloud
The data storage used as input or output can be not only on Microsoft Azure but also in another storages that can be on-premises.
Data Source Aggregation
We have the ability to combine multiple data sources in only one data sources. In this way we can easily merge different data sources.
Data Processing
All the input data sources can be processed and base transformation functions can be applied on them. In this way we can monitor our data.
Complex data transformation
Data Factory supports complex transformations functions, being able to use external components like Pig, Hive, HDInsight or C# libraries.
Configurable
There is full support for custom behaviors and fault-tolerance. We have the ability to manage clusters, change the retry policy, the alert rules and timeouts policy.
Management
The management of pipelines can be made using different tools from PowerShell scripts to .NET libraries.
Workflow
Data processing and pipeline management can be managed easily using a simple workflow UI. Using this workflow the pipeline configuration is very simple.
Linked Services
A linked service is represented by an external data source like a blob storage or a SQL database. A linked resources can be also a computing service. We can link one or more linked services.
Input/Output Tables
It is a rectangular dataset that is used for data that are coming to Data Factory or that are produced by it. This tables are produces from the linked resources data stored and can be persistent in external data sources. The JSON format is used for this tables.
Pipeline
A pipeline is an action that is executed over inputs table and produce some data in the output table(s). Over the this data we can apply different processing and transformation operations.

Limitations 
One of the current limitations is the on-premises storages type that is supported in this moment. Now, only SQL Server is supported for on-premises sources.

Applicable Use Cases 
Below you can find some use cases when I would use Azure Data Factory.
Logs file processing
Logs can be produced by multiple data sources and can be stored in different data sources. We can use Azure Data Factory to fetch data from all data sources and combine all this logs in only one location, having the same format of data.
Transform data to the same format
If we work with different legacy system we will have data in different formats. Data Factory can be used with success to aggregate all this data and transform to a common format.

Code Sample 
A great sample can be found on the fallowing address: http://azure.microsoft.com/en-us/documentation/articles/data-factory-troubleshoot/

Pros and Cons 
Pros
Integration with on-premises data sources
Support multiple data sources
Aggregation and transformation of data can be made easily
Cons
In this moment there is no support for other on-premises data sources except SQL Server.

Pricing 
If you want to calculate the cost of Azure Data Factory you should take into account:

  • Number of activities
  • Outbound traffic (from on-premises data sources)
  • Run on Azure data sources or on-premises 


Conclusion
Azure Data Factory is an interesting services that can be used with success when you need to manage and orchestrate data from different data sources and formats.

Comments

Popular posts from this blog

How to check in AngularJS if a service was register or not

There are cases when you need to check in a service or a controller was register in AngularJS.
For example a valid use case is when you have the same implementation running on multiple application. In this case, you may want to intercept the HTTP provider and add a custom step there. This step don’t needs to run on all the application, only in the one where the service exist and register.
A solution for this case would be to have a flag in the configuration that specify this. In the core you would have an IF that would check the value of this flag.
Another solution is to check if a specific service was register in AngularJS or not. If the service was register that you would execute your own logic.
To check if a service was register or not in AngularJS container you need to call the ‘has’ method of ‘inhector’. It will return TRUE if the service was register.
if ($injector.has('httpInterceptorService')) { $httpProvider.interceptors.push('httpInterceptorService&#…

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

[Post-Event] Codecamp Conference Cluj-Napoca - Nov 19, 2016

Last day I was invited to another Codecamp Conference, that took place in Cluj-Napoca. Like other Codecamp Conferences, the event was very big, with more than 1.000 participants and 70 sessions. There were 10 tracks in parallel, so it was pretty hard to decide at  what session you want to join.
It was great to join this conference and I hope that you discovered something new during the conference.
At this event I talked about Azure IoT Hub and how we can use it to connect devices from the field. I had a lot of demos using Raspberry PI 3 and Simplelink SensorTag. Most of the samples were written in C++ and Node.JS and people were impressed that even if we are using Microsoft technologies, we are not limited to C# and .NET. World and Microsoft are changing so fast. Just looking and Azure IoT Hub and new features that were launched and I'm pressed (Jobs, Methods, Device Twin).
On backend my demos covered Stream Analytics, Event Hub, Azure Object Storage and DocumentDB.

Title:
What abo…