Skip to main content

Address data processing outside the data sovereignty region (GDPR)

 The high level of cloud adoption brings closer and closer to us the day to day problems that were easily solved using on-premises solutions. The data regulations related to data sovereignty and specific region and country regulations like GDPR add an extra complexity layer to the application we are building inside the cloud.

We have an imaginary organization called Osotnoc that is active in Australia, the US, UK and the EU. Each country has strict data regulations that require Osotnoc to store customers data inside each region. 

Microsoft Azure has a strong presence in each region, so the team is building a solution on Azure SQL and App Service. The solution is deployed in all 4 regions, fully compliant with all local requirements but pretty expensive. Except for the database and computation layer, 4 different instances of Azure Application Gateways are deployed with WAF on top of them. 

Besides this, more than 30% of active users requires access data from other regions. The current solution requires them to specify the tenant/region they want to access during the login step. To have the ability to compare data across regions, they need to log into two separate browsers. It's not a nice UX.

But what if we could make the solution more simple? The local data requirements are mainly referring to the data storage layer and not to the computation. Meaning that nobody stops us from getting content from repositories that are from different regions.

One way to do this is to use the sharding capability of Azure SQL Database and the Shard Map Manager. This would enable the application layer to have the ability to manage from which shard (DB) to connect. It allows users to run the same query across multiple regions (shards) and display them on the UI.

The solution works pretty well, but some logic needs to be handled at the application layer. It is not complicated, but an extra effort is required to support this. The good thing is that we have support for .NET and Java, the ShartMapManager, part of the Elastic Database package that can be used with success for this purpose. 

There is an impact on latency when you need to load content from databases that are in a different region, but you can provide a better experience for users in this way. It's a good tradeof that you can do.

Depending on the regulations and the user patterns, you could even add a cache layer in each region for data that is often accessed. A cache is not seen as long term storage, and you are allowed to use it in specific situations, depending on the content type and the level of encryption that you use. The downside with a cache solution is that they are one of the weakest parts of a system. The security layer exists but is not their main straight. In the end in most of the cases a CDN or a remote cache it is used for non-sensitive content.

In both cases (CDN or cache) you need to have content full encrypted, the encryption key stored in EU and clear procedures related to how you manage data and what kind of actions you take.

What you need to take into account from data compliance and GDPR point of view:

  • Processing EU citizen data outside the EU region is allowed as long as the encryption key is stored in EU. Special conditions are required to be allowed to do something like this
  • A encrypted cache outside EU can be used, but content needs to be encrypted and additional technical and organizational measures needs to be taken. 


Popular posts from this blog

Azure AD and AWS Cognito side-by-side

In the last few weeks, I was involved in multiple opportunities on Microsoft Azure and Amazon, where we had to analyse AWS Cognito, Azure AD and other solutions that are available on the market. I decided to consolidate in one post all features and differences that I identified for both of them that we should need to take into account. Take into account that Azure AD is an identity and access management services well integrated with Microsoft stack. In comparison, AWS Cognito is just a user sign-up, sign-in and access control and nothing more. The focus is not on the main features, is more on small things that can make a difference when you want to decide where we want to store and manage our users.  This information might be useful in the future when we need to decide where we want to keep and manage our users.  Feature Azure AD (B2C, B2C) AWS Cognito Access token lifetime Default 1h – the value is configurable 1h – cannot be modified

How to audit an Azure Cosmos DB

In this post, we will talk about how we can audit an Azure Cosmos DB database. Before jumping into the problem let us define the business requirement: As an Administrator I want to be able to audit all changes that were done to specific collection inside my Azure Cosmos DB. The requirement is simple, but can be a little tricky to implement fully. First of all when you are using Azure Cosmos DB or any other storage solution there are 99% odds that you’ll have more than one system that writes data to it. This means that you have or not have control on the systems that are doing any create/update/delete operations. Solution 1: Diagnostic Logs Cosmos DB allows us activate diagnostics logs and stream the output a storage account for achieving to other systems like Event Hub or Log Analytics. This would allow us to have information related to who, when, what, response code and how the access operation to our Cosmos DB was done. Beside this there is a field that specifies what was th

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine: threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration: TeamCity .NET 4.51 EF 6.0.2 VS2013 It see