Skip to main content

Azure HDInsight (Day 25 of 31)

List of all posts from this series:

Short Description 
HDInsight enables us to provision Hadoop and HBase cluster over Azure infrastructure. This mean that we can use Hadoop over Azure platform without having to install, provision and make complex configurations of Hadoop. We can use HDInsight as an out of the box Hadoop solution.

Main Features 
Hadoop Version
The Hadoop distribution that is running on Azure is developed and maintained by Hortonworks Data Platform.
New version releasing
It seems that the new versions of Hadoop reach Azure faster and faster. If in the past it use to take 4-6 months for a new version of Hadoop to reach Azure and HDInsight, now it is taking only a few weeks.
Hadoop Distributed File System(HDFS) exist also in HDInsight giving us the possibility to store data in a distributed file system.
The same programming model that we already know from Hadoop can be found into HDInsight. In general you should know that all the features that exist in Hadoop can be found in HDInsight, having the same properties and functionalities.
Can be used to monitor, manage and provision the cluster that is used for processing. The API of Ambari hide all the complexity that is behind Hadoop and let us to focus on the final result and less on the infrastructure.
The NoSQL Solution develop by Hadoop and used to store data in a semi-structured way. We can store billions of rows in this data storage. Behind HBase, Azure implementation is using Azure Storage.
Developed over MapReduce, it allows us to query using SQL like the Hadoop storage. Can be used with success when the stored data that is queried in a more structured.
Similar with Hive, but is recommended to use when data is not structured and we need to execute complex MapReduce operations over large data sets.
It allow us to define a machine learning that run over Hadoop. Based on past evens and data sets we can determine what will be the future behavior of different components.
On the core components of Hadoop that allow us to execute operations over large data sets in a distributed way (on multiple nodes – parallel). The data is organized in a (key, value) for later processing.
The next generation of MapReduce (MRv2) that will allow us to slit the jobs in two different entities:

  • Scheduling and Monitor
  • Resource Management

It is the framework used to coordinate the workflow that is behind this great system. Can be used for different tasks like shell script or scheduling.
Real time computing system for processing streams of data.
Contains small amount of metadata (like location, configuration, status) and is used to coordinate large distributed system using a simple hierarchy of data regions.
It used to import/export data from Hadoop to relational database like SQL.
Out of the box clusters
Hadoop clusters can be provision in minutes, without having to configure and manage them
Integration with Azure Services
HDInsight is integrated with different Azure Services like SQL Database or Websites. In this way we can implement different scenarios more easily.
Power Query
There is full support for Excel using Power Query to access and view data from Hadoop.
Cluster Size
We have the ability to specify the size of the Hadoop cluster that we want to use. Based on our needs we can set a custom cluster size.
Instances type
Users have the ability to specify what kind of machines to use in their HDInsight cluster.
Low storage cost
Because Azure Storage Blobs are used to store Hadoop data, the cost of storage is low. In this way we can store large amount of data without thinking about costs.
Storage scaling
Storing data in blobs remove the problems that can appear when we need to scale out the HDFS.
Migrate Hadoop from on-premises to HDInsight
With integration of HDP (Hortonworks Data Platform) we can migrate data from one storage to another automatically. Also using HDP we are allowed to query HDInsight and on-premises instance in the same time.

Start/Stop Cluster
In this moment we don’t have the ability to stop a cluster. The current state of stop is equivalent with delete. Content from blobs is not lost.
Elastic size of cluster
In this moment we cannot dynamically change the size of a cluster. In this moment you need to tear down and rebuild the clusters if you need to change the size of it.

Applicable Use Cases 
Below you can find some use cases when I would use HDInsight.
Analyze information from Smart Home solution
Data that are produced by a Smart Home solution can be very valuable, but the volume of data is very high. For this scenario HDInsight can be used with success to analyze the data and extract valuable information from it.
eCommerce application logs
 An eCommerce application logs is like a gold mine. Based on this information you can know what the system bottleneck are, what products are more attractive for clients and many more. To analyze of all this data you can use with success HDInsight.
Image analyzing
Image analyzing is a job that consume not only a lot of CPU power, but also a lot of storage. Because of this an image analyzing system can be constructed over HDInsight.

Code Sample 
A sample that a realy enjoyed and is very usefull can be found here:

Pros and Cons 

  • Scalable
  • Easy to integrate
  • Out of the box solution
  • Integration with a lot of external libraries and systems


  • Start/Stop features is not yet supported
  • Elastic-scale is not yet supported

If you are taking into account HDInsight, you should keep an eye on:

  • Cluster Size
  • Instances type
  • Outbound traffic
  • Storage size

Azure HDInsight it is a have to if you store your data and logs in Azure Storage. Don’t be afraid to analyze your data, because you may find very interesting things/facts.


Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

GET call of REST API that contains '/'-slash character in the value of a parameter

Let’s assume that we have the following scenario: I have a public HTTP endpoint and I need to post some content using GET command. One of the parameters contains special characters like “\” and “/”. If the endpoint is an ApiController than you may have problems if you encode the parameter using the http encoder.
using (var httpClient = new HttpClient()) { httpClient.BaseAddress = baseUrl; Task<HttpResponseMessage> response = httpClient.GetAsync(string.Format("api/foo/{0}", "qwert/qwerqwer"))); response.Wait(); response.Result.EnsureSuccessStatusCode(); } One possible solution would be to encode the query parameter using UrlTokenEncode method of HttpServerUtility class and GetBytes method ofUTF8. In this way you would get the array of bytes of the parameter and encode them as a url token.
The following code show to you how you could write the encode and decode methods.

Entity Framework (EF) TransactionScope vs Database.BeginTransaction

In today blog post we will talk a little about a new feature that is available on EF6+ related to Transactions.
Until now, when we had to use transaction we used ‘TransactionScope’. It works great and I would say that is something that is now in our blood.
using (var scope = new TransactionScope(TransactionScopeOption.Required)) { using (SqlConnection conn = new SqlConnection("...")) { conn.Open(); SqlCommand sqlCommand = new SqlCommand(); sqlCommand.Connection = conn; sqlCommand.CommandText = ... sqlCommand.ExecuteNonQuery(); ... } scope.Complete(); } Starting with EF6.0 we have a new way to work with transactions. The new approach is based on Database.BeginTransaction(), Database.Rollback(), Database.Commit(). Yes, no more TransactionScope.
In the followi…