Skip to main content

Why our Integration Tests were down

From some time, one of our projects has the Integration Tests on RED. It seems that randomly, some
tests are failing. This usually happens only on the CI machine (Visual Studio Team Services).
The strangest thing is the behavior. The behavior cannot be reproduced on the development machine. The reported errors are caused by asserts or strange exceptions - for example that an external resources doesn't exist.
From time to time, the error can be reproduced on the local machine, but only once. The fail cannot be reproduced twice on the same machine. After a few sprints and a lot of time invested in this issue, we still had the same problem.

After a review of <the problem,the code, how the tests were written>, the root causes were identified and isolated. Let's see what were the steps and causes of this problem.

Step: Isolate the integration tests in a dedicated build.
Why: You want to reduce the build time us much as possible and run only the tests that you want without affecting the rest of the team. You need an 'isolated' environment where you want to play and test different configuration without sending hundreds of email notification to the team.

Step: Don't try to resolve all the issues in the same.
Why: Take each problem separated, analyze it and fix it. Only after you fixed it go to the second one.

Step: Understand what the test is testing.
Why: If you want to resolve a problem you need to understand what you want to achieve, in our case what is the purpose of the test.

Cause: Avoid refactoring Unit Tests to reduce duplication.
Why: Yes, is nice to have fewer lines of code, but in tests this can be a killer. Once you reduced the number of lines of code from unit tests by extracting in common methods you will need to check that any change that you do to the code will not affect or alter the behavior of all the tests that are using that code.

Cause: Avoid sharing resources between tests from the same test class.
Why: You will end up in one tests to remove or alternate a resource that is needed by another test. A good example here is for tests that cover a special flow that will end up with an error. To be able to throw an error you might change a configuration that is needed by another test.
To avoid this kind of scenarios and share resources between tests you might want to run a preparation or cleanup step before and after each test - the main purpose of this step is to reset the environment to the 'standard' setup.

Cause: Try to fix the tests before understanding what the test is testing.
Why:  Very often we are jumping to the solution, to change the code, the classical try&error mode before making the investigation. This can costs us a lot time and on top of this it can add more bugs and problems to the system, making it more unstable.

Cause: Testing an async method in the same way like a syncmethod. 
Why: In this case this was the root of the strange behavior on the system. Based on the load of the machine and the test order, the async method that was called had enough time to execute before the test ended or not. A few years ago I wrote posts about this topic, where you can find more information about how you can test an async method - topic http://vunvulearadu.blogspot.ro/2013/04/how-to-write-unit-tests-for-async.html

[TestMethod]
public async Task SomeTest() 
{ 
     await .....
}

In our case the main causes were:

  • Testing async methods in the wrong way
  • Refactoring Unit Tests to much
In conclusion I would say that if you have a strange behavior, don't try to resolve it before understanding the root cause of the problem. 

Comments

Popular posts from this blog

How to check in AngularJS if a service was register or not

There are cases when you need to check in a service or a controller was register in AngularJS.
For example a valid use case is when you have the same implementation running on multiple application. In this case, you may want to intercept the HTTP provider and add a custom step there. This step don’t needs to run on all the application, only in the one where the service exist and register.
A solution for this case would be to have a flag in the configuration that specify this. In the core you would have an IF that would check the value of this flag.
Another solution is to check if a specific service was register in AngularJS or not. If the service was register that you would execute your own logic.
To check if a service was register or not in AngularJS container you need to call the ‘has’ method of ‘inhector’. It will return TRUE if the service was register.
if ($injector.has('httpInterceptorService')) { $httpProvider.interceptors.push('httpInterceptorService&#…

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Run native .NET application in Docker (.NET Framework 4.6.2)

Scope
The main scope of this post is to see how we can run a legacy application written in .NET Framework in Docker.

Context
First of all, let’s define what is a legacy application in our context. By a legacy application we understand an application that runs .NET Framework 3.5 or higher in a production environment where we don’t have any more the people or documentation that would help us to understand what is happening behind the scene.
In this scenarios, you might want to migrate the current solution from a standard environment to Docker. There are many advantages for such a migration, like:

Continuous DeploymentTestingIsolationSecurity at container levelVersioning ControlEnvironment Standardization
Until now, we didn’t had the possibility to run a .NET application in Docker. With .NET Core, there was support for .NET Core in Docker, but migration from a full .NET framework to .NET Core can be costly and even impossible. Not only because of lack of features, but also because once you…