Skip to main content

How to track who is accessing your blob content

In this post we’ll talk about how we can monitor our blobs from Windows Azure.
When hosting content on a storage one of the most commune request that is coming from clients is

  • How I can monitor each request that is coming to my storage?
  • For them it is very important to know 
  • Who downloaded the content?
  • When the content was downloaded?
  • How the request ended (with success)?

I saw different solution for this problem. Usually the solutions involve services that are called by the client after the download ends. It is not nothing wrong to have a service for confirmation, but you add another service that you need to maintain. Also it will be pretty hard to identify why a specific device cannot download your content.
In this moment, Windows Azure has a build-in feature that give us the possibility to track all the requests that are made to the storage. In this way you will be able to provide all the information related to the download process.
Monitor
In Windows Azure portal you will need to go to your storage and navigate to the Monitoring section. For blobs you will need to set the monitoring level to Verbose. Having the monitoring level to Verbose all metrics related to your storage will be persisted. The main different between Minimal and Verbose is the level of monitoring.
This data can be persisted from 1 day to 1 year. Based on your needs and how often you collect the data you can set the best value that suites you. If your storage is used very often I recommend to set maximum 7 days. You can define a simple process that extract monitor information from the last 7 days, store it in a different location and analyze it using your own rules. For example you may way want to raise an alert to your admins if a request coming from the same source failed for more than 10 times.
This table contains all the monitoring information for your storage. In this moment we don’t have support to write monitoring data for a specific blob to a specific table, but we can make query over this table and select only the information that we need.
All the information related to this will be persisted in Azure table from your storage named ‘$MetricsCapacityBlob’.
Logging
The feature that we really need is logging. Using logging functionality we will be able to trace all request history. The activation of this feature can be done from the portal, under the logging section. You can activate logging for the main 3 operations that can be made over a blob: Read/Write/Delete.
All this data is stored under the $logs container:
https://<accountname>.blob.core.windows.net/$logs/blob/YYYY/MM/DD/hhm/counter.log
Everything that we can imagine can be found in this table:

  • Successful and failed requests with or without Shared Access Signature
  • Server errors
  • Timeouts errors
  • Authorization, network, throttling errors
  • ...

Each entity from the table contains helpful information like:

  • LogType (write, read, delete)
  • StartTime 
  • EndTime
  • LogVersion (for future, in this moment we have only one version – 1.0)
  • Request URL
  • Client IP

The most useful information for is usually found under the ‘Client IP’ and ‘Request URL’. Maybe you ask yourself why we have a start time and an end time. This can be very useful for a read request for example. In this way we will be able to know how long the download process took.

I invite you to explore this feature when you need to track the clients that access your blob resources.

Comments

Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Entity Framework (EF) TransactionScope vs Database.BeginTransaction

In today blog post we will talk a little about a new feature that is available on EF6+ related to Transactions.
Until now, when we had to use transaction we used ‘TransactionScope’. It works great and I would say that is something that is now in our blood.
using (var scope = new TransactionScope(TransactionScopeOption.Required)) { using (SqlConnection conn = new SqlConnection("...")) { conn.Open(); SqlCommand sqlCommand = new SqlCommand(); sqlCommand.Connection = conn; sqlCommand.CommandText = ... sqlCommand.ExecuteNonQuery(); ... } scope.Complete(); } Starting with EF6.0 we have a new way to work with transactions. The new approach is based on Database.BeginTransaction(), Database.Rollback(), Database.Commit(). Yes, no more TransactionScope.
In the followi…

GET call of REST API that contains '/'-slash character in the value of a parameter

Let’s assume that we have the following scenario: I have a public HTTP endpoint and I need to post some content using GET command. One of the parameters contains special characters like “\” and “/”. If the endpoint is an ApiController than you may have problems if you encode the parameter using the http encoder.
using (var httpClient = new HttpClient()) { httpClient.BaseAddress = baseUrl; Task<HttpResponseMessage> response = httpClient.GetAsync(string.Format("api/foo/{0}", "qwert/qwerqwer"))); response.Wait(); response.Result.EnsureSuccessStatusCode(); } One possible solution would be to encode the query parameter using UrlTokenEncode method of HttpServerUtility class and GetBytes method ofUTF8. In this way you would get the array of bytes of the parameter and encode them as a url token.
The following code show to you how you could write the encode and decode methods.
publ…