Skip to main content

Steam.CopyTo - Performance problems

This week I heard about an interesting performance problem, that I want to share with you.
There is a desktop application that is used to import data from files to database. These files were in different formats, from XML to binary. Everything was fine, the application work great in the development and first phase of testing. All the code was double check, each stream was dispose when it was no longer used.
During the testing phase, the application started to have an odd behavior, the RAM memory started to go crazy. Normally, the application use around 200MB during an import, but there were times when more than 1500MB of ram were consumed. Not only this, but “Out of memory” exception appeared a lot of time during different imports. All this started too appeared at the testing phase.
With the same setup, developers manage to reproduce the problem and using the memory profiles that comes with Visual Studio the problem was spotted.
Stream stream1 = …
Stream stream2 = …
stream2.CopyTo(stream1);
The problem was with CopyTo method. Because this method was called a lot of time in a short period of time, the GC didn’t had time to dispose all the resources used during the copy process. This can become a big problem when the size of stream is pretty big. This method was introduce in .NET 4.0 and the scope of it is to copy the content of one stream to another. This is done by reading to a buffer of bytes (default buffer size is 4096) chunks from the stream and write them to the second stream. 
It is very similar with what we usually do in .NET 3.5 when we had to copy one stream to another:
public void CopyStream(Stream input, Stream output)
{
    byte[] buffer = new byte[size];
    int read;
    while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
    {
        output.Write (buffer, 0, read);
    }
}
The problem is with the buffer that is represented as an array of bytes. If we call the CopyTo method from time to time with small streams will not have any kind of problem. But when we use big streams intensive, calling this method very often can create huge problems. Each time when we make new calls to this method, a new part of our memory will be allocated for the buffer. GC will not have time to clean the memory and we can end up with a lot of memory that is not disposed.
The solution for this problem was pretty simple. The CopyTo method was replaced with a BinaryReader that was used to copy the content of stream to an array of bytes. This array of bytes was later used to create the MemoryStream. In this way, the stream that was used for read data is disposed immediately (BinaryReader) and the array of bytes that is used as a buffer is used directly in the memory stream.
byte[] bytes;
using( BinaryReader reader = new BinaryReader(inputStream))
{
    bytes = reader.ReadBytes((int)inputStream.Length); 
}
outputStream = new MemoryStream(bytes, 0, bytes.Count());
Using this solution, the problem with the memory leak disappeared. It seems that not all time, the new API is the best solution when we need performance.

Comments

  1. What if size of input stream will be too big?
    it will try to create huge bytearray, and possible OutOfMemoryException - no?

    ReplyDelete
  2. Wow, thanks, this helps alot !

    ReplyDelete

Post a Comment

Popular posts from this blog

How to check in AngularJS if a service was register or not

There are cases when you need to check in a service or a controller was register in AngularJS.
For example a valid use case is when you have the same implementation running on multiple application. In this case, you may want to intercept the HTTP provider and add a custom step there. This step don’t needs to run on all the application, only in the one where the service exist and register.
A solution for this case would be to have a flag in the configuration that specify this. In the core you would have an IF that would check the value of this flag.
Another solution is to check if a specific service was register in AngularJS or not. If the service was register that you would execute your own logic.
To check if a service was register or not in AngularJS container you need to call the ‘has’ method of ‘inhector’. It will return TRUE if the service was register.
if ($injector.has('httpInterceptorService')) { $httpProvider.interceptors.push('httpInterceptorService&#…

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Run native .NET application in Docker (.NET Framework 4.6.2)

Scope
The main scope of this post is to see how we can run a legacy application written in .NET Framework in Docker.

Context
First of all, let’s define what is a legacy application in our context. By a legacy application we understand an application that runs .NET Framework 3.5 or higher in a production environment where we don’t have any more the people or documentation that would help us to understand what is happening behind the scene.
In this scenarios, you might want to migrate the current solution from a standard environment to Docker. There are many advantages for such a migration, like:

Continuous DeploymentTestingIsolationSecurity at container levelVersioning ControlEnvironment Standardization
Until now, we didn’t had the possibility to run a .NET application in Docker. With .NET Core, there was support for .NET Core in Docker, but migration from a full .NET framework to .NET Core can be costly and even impossible. Not only because of lack of features, but also because once you…