Cloud as a Story - Vunvulea Radu

There are moments in time when you focus on small things and you might miss bigger and more important things. Working on a solution that is distributed in multiple Azure Regions across the world we had to replicated binary content (files) in all data centers where our solution runs. The system that push this content is an on-premises system. In this context we decided to use a temporary location where the content is copied first and from where we replicate the content in other regions. As we can see, the Temp location is used to replicate the content all around the glob. The system is smart enough to balance the load and if multiple replicas are needed in the same Azure Regions, only one copy will be done from the Temp location. Once the replication is done, the binary content from the Temp location is removed automatically. Nothing special until this, until in the moment when you ask yourself what is happening if one of the Replicas is corrupted or something happens and you ...

Storing data in a cloud provider like Microsoft Azure or AWS is trivial. If you have an application that is running on a cloud provider you will start to generate content that is stored there. Many times this content is private and you need a secure solution to transfer it to different locations around the glob. One solution for this problem could be Azure CDNs, but in this moment only HTTP protocol is supported. This means that you will need to encrypt the content before sending it on the wire. This might be possible, but if need to transfer 1 TB of data, the encryption and decryption will take some time and will consume resources (especially CPU). A better solution might be Aspera. Using Aspera services, you will get a transport platform that offer a secure channel (encrypted) to transfer data from one location to another. All the things that you normally need to take into account like bandwidth, security layer over HTTP and so on are handled by Aspera. The communication channel o...

In this post we will talk about how we can detect when we hit a throughput limit of Azure Storage and what we can do in that moment. Context If we take a look on Scalability Targets of Azure Storage ( https://azure.microsoft.com/en-us/documentation/articles/storage-scalability-targets/ ) we will observe that the limits are prety high. But, based on our business logic we can end up at this limits. If you create a system that is hitted by a high number of device, you can hit easily the total number of requests rate that can be done on a Storage Account. This limits on Azure is 20.000 IOPS (entities or messages per second) where (and this is very important) the size of the request is 1KB. Normally, if you make a load tests where 20.000 clients will hit different blobs storages from the same Azure Storage Account, this limits can be reached. How we can detect this problem? From client, we can detect that this limits was reached based on the HTTP error code that is returned by HTTP ...

In this pot we will talk about 'ListBlobsSegmented' command, that allow us to get blobs from a container. When this command is useful? 'ListBlobsSegmented' is used when we need to fetch the list of blobs that are under a container of Azure Storage. This command will not fetch the content of blobs, only the metadata of blob will be fetched. Based on this information, if needed we can trigger the download. An important thing is related to the number of blobs that will be fetched when we make a call. The number of blobs that will be retrieved from a call is maximum 5.000 blobs metadata. If the container has more than 5.000 items, the response will contain also a BlobContinuationToken. This token can be used to fetch the next 5.000 blobs from the container. The size of the result cannot be changed. We cannot modify this value. Example: BlobResultSegment blobResultSegment = blobContainer.ListBlobsSegmented(new BlobContinuationToken()); while (blobResultSegment.Contin...

A few days ago, client-side encryption for Azure Storage was announced. In this post we will take a look over this feature. First of all, you should know that the encryption/decryption takes place on client side. This means, that the content will be already encryption when it will arrive on Azure. This encryption technique is called Envelop Technique. It is very useful when you want to add another security layer over your data. Out of the box, there are client library for .NET (including Windows Phone). For other languages, like Java is not yet supported, but because the encryption algorithm is a well know one, you may be able to implement it on other platforms also. The encryption algorithm that is used by client library is AES (Advanced Encryption Standard). It is important to know that the encryption keys are generated by the client library. And is NEVER stored in Azure Storage. The encryption key should be stored in a different location. This library is full integration with K...

In this post we will take a look about a new type of blob - Append Blob Until now, we had two types of blobs in Azure Storage: Page Blob Block Blob Block Blob is useful when we needs to work with large files. Each block can have different sized (max 4 MB) and we can work with each block independently. Features similar with transactions are supported on blocks. The maximum size of a block blob is 195 GB. Page Blob are optimized for random access. A Page Blob is a collection of pages of 512 bytes and are useful when we need to support random access to the content (read and write). We can refer to a specific location of a Page Blob very similar with a cursor. The maximum size of a Page Blob is 1 TB. When you create a Page Blob you need to specify the maximum size of the Page Blob (and you will pay for it from the beginning, even if you don't use all the space. Append Blob Concept This new type of blob is formed by multiple blocks. Each time when you create a block, it wi...

Using blobs from Windows Azure, we can store any type of content (from text file to images or videos). We can look at blobs like a hard disk partition where we can store any type of data. By default a blob is accessed based on a storage connection string. A consumer will be able to access a specific blob from the storage account based on his credentials. But there are cased when you don’t want to let a user for an unlimited type to have access to your storage account. For this cased the Shared Access Signature. This feature from Windows Azure permits us to give access to someone for a limited type. Based on the link that is generated by Shared Access Signature the consumer will be able to use the given blob. Basically, the Shared Access Signature is a URL that gives the user access to a specific resource. In general these resources can be a container, a blob, table or queues. In this blog post we will talk about blobs and containers. The administrator that create the Shared Access ...

In mod default, cand pornim storage-ul local, acesta accesează 127.0.0.1. Dar ce ne facem daca avem o echipa de dezvoltare care doreste sa lucreze pe un storage comun. Prima posibilitatea este sa ne configuram un storage sus pe cloud, dar asta înseamna bani in plus si un lag, in cazul in care nu avem o conexiune foarte rapida. O alta varianta este s schimbam configurarea pe mașinile de development, astfel incat acestea sa bata spre aceiasi adresa. Pentru a putea face acest lucru trebuie sa deschidem fisierul DSService.exe.config din directorul ~Program Files\Windows Azure SDK\v1.4\bin\devstore\DSService.exe.config . Sub nodul services o sa gasim lista de adrese pentru blobs, queue si tables. Fiecare adresa a serviciului poate sa fie modificata, ca in exemplul de mai jos: <services> <service name="Blob" url="http://10.123.1.9:10000/"/> <service name="Queue" url="http://10.123.1.9:10001/"/> <service name="Table...

La fiecare blob din Windows Azure se poate seta content type. De cele mai multe ori acest parametru trece neobservat. Mai jos este un exemplu care arata cum se poate schimba aceasta valoare: Paste your text here.var blob = Container.GetBlockBlobReference("numeBlob"); blob.Properties.ContentType = "application/octet-stream"; Cand aceasta propietate devine utila? In momentul in care vrem sa stocam pe blob resurse statice. De exemplu vrem sa adaugam un CSS pe blob, iar href pentru CSS sa fie spre adresa din blob. Pentru ca aceasta solutie sa functioneze pe orice brower( Firefox are nevoie de un content type valid) este nevoie sa setam valoarea la content type egala cu "text/css". var blob = Container.GetBlockBlobReference("numeBlob"); blob.Properties.ContentType = "text/css"; Daca aceasta valoare nu este setata pe Firefox nu o sa se incarce deloc CSS-ul, desi adresa setata este valida.

Pe blob-urile din Windows Azure putem sa stocam orice fel de conținut, de la poze si filme la carti sau arhive. Cand vrem sa partajam aceasta informație cu alte persoane apare mai multe intrebari: Ce poate sa faca un utilizator pe blob? Ce poate sa faca un utilizator pe container? Un utilizator are dreptul sa acceseze container-ul si sa il parcurgă? La nivel de container putem sa setam trei nivele de acces. Varianta pe care o alegeți trebuie aleasa cu grija, deoarece aceasta va afecta in mod direct modul in care un utilizator anonim poate sa acceseze un blob. Prima varianta este " Full public read access ". Un utilizator anonim poate sa citească din orice blob sau container. Acesta poate sa itereze in interiorul container. Singurul lucru pe care nu il poate sa faca este sa intereze prin containere. A doua varianta este " Public read access for blobs only ". Aceasta se aseamana cu prima varianta, doar ca un utilizator nu poate sa itereze prin conținutul unui containe...