Skip to main content

Why our Integration Tests were down

From some time, one of our projects has the Integration Tests on RED. It seems that randomly, some
tests are failing. This usually happens only on the CI machine (Visual Studio Team Services).
The strangest thing is the behavior. The behavior cannot be reproduced on the development machine. The reported errors are caused by asserts or strange exceptions - for example that an external resources doesn't exist.
From time to time, the error can be reproduced on the local machine, but only once. The fail cannot be reproduced twice on the same machine. After a few sprints and a lot of time invested in this issue, we still had the same problem.

After a review of <the problem,the code, how the tests were written>, the root causes were identified and isolated. Let's see what were the steps and causes of this problem.

Step: Isolate the integration tests in a dedicated build.
Why: You want to reduce the build time us much as possible and run only the tests that you want without affecting the rest of the team. You need an 'isolated' environment where you want to play and test different configuration without sending hundreds of email notification to the team.

Step: Don't try to resolve all the issues in the same.
Why: Take each problem separated, analyze it and fix it. Only after you fixed it go to the second one.

Step: Understand what the test is testing.
Why: If you want to resolve a problem you need to understand what you want to achieve, in our case what is the purpose of the test.

Cause: Avoid refactoring Unit Tests to reduce duplication.
Why: Yes, is nice to have fewer lines of code, but in tests this can be a killer. Once you reduced the number of lines of code from unit tests by extracting in common methods you will need to check that any change that you do to the code will not affect or alter the behavior of all the tests that are using that code.

Cause: Avoid sharing resources between tests from the same test class.
Why: You will end up in one tests to remove or alternate a resource that is needed by another test. A good example here is for tests that cover a special flow that will end up with an error. To be able to throw an error you might change a configuration that is needed by another test.
To avoid this kind of scenarios and share resources between tests you might want to run a preparation or cleanup step before and after each test - the main purpose of this step is to reset the environment to the 'standard' setup.

Cause: Try to fix the tests before understanding what the test is testing.
Why:  Very often we are jumping to the solution, to change the code, the classical try&error mode before making the investigation. This can costs us a lot time and on top of this it can add more bugs and problems to the system, making it more unstable.

Cause: Testing an async method in the same way like a syncmethod. 
Why: In this case this was the root of the strange behavior on the system. Based on the load of the machine and the test order, the async method that was called had enough time to execute before the test ended or not. A few years ago I wrote posts about this topic, where you can find more information about how you can test an async method - topic

public async Task SomeTest() 
     await .....

In our case the main causes were:

  • Testing async methods in the wrong way
  • Refactoring Unit Tests to much
In conclusion I would say that if you have a strange behavior, don't try to resolve it before understanding the root cause of the problem. 


Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

GET call of REST API that contains '/'-slash character in the value of a parameter

Let’s assume that we have the following scenario: I have a public HTTP endpoint and I need to post some content using GET command. One of the parameters contains special characters like “\” and “/”. If the endpoint is an ApiController than you may have problems if you encode the parameter using the http encoder.
using (var httpClient = new HttpClient()) { httpClient.BaseAddress = baseUrl; Task<HttpResponseMessage> response = httpClient.GetAsync(string.Format("api/foo/{0}", "qwert/qwerqwer"))); response.Wait(); response.Result.EnsureSuccessStatusCode(); } One possible solution would be to encode the query parameter using UrlTokenEncode method of HttpServerUtility class and GetBytes method ofUTF8. In this way you would get the array of bytes of the parameter and encode them as a url token.
The following code show to you how you could write the encode and decode methods.

Entity Framework (EF) TransactionScope vs Database.BeginTransaction

In today blog post we will talk a little about a new feature that is available on EF6+ related to Transactions.
Until now, when we had to use transaction we used ‘TransactionScope’. It works great and I would say that is something that is now in our blood.
using (var scope = new TransactionScope(TransactionScopeOption.Required)) { using (SqlConnection conn = new SqlConnection("...")) { conn.Open(); SqlCommand sqlCommand = new SqlCommand(); sqlCommand.Connection = conn; sqlCommand.CommandText = ... sqlCommand.ExecuteNonQuery(); ... } scope.Complete(); } Starting with EF6.0 we have a new way to work with transactions. The new approach is based on Database.BeginTransaction(), Database.Rollback(), Database.Commit(). Yes, no more TransactionScope.
In the followi…