Skip to main content

Steam.CopyTo - Performance problems

This week I heard about an interesting performance problem, that I want to share with you.
There is a desktop application that is used to import data from files to database. These files were in different formats, from XML to binary. Everything was fine, the application work great in the development and first phase of testing. All the code was double check, each stream was dispose when it was no longer used.
During the testing phase, the application started to have an odd behavior, the RAM memory started to go crazy. Normally, the application use around 200MB during an import, but there were times when more than 1500MB of ram were consumed. Not only this, but “Out of memory” exception appeared a lot of time during different imports. All this started too appeared at the testing phase.
With the same setup, developers manage to reproduce the problem and using the memory profiles that comes with Visual Studio the problem was spotted.
Stream stream1 = …
Stream stream2 = …
stream2.CopyTo(stream1);
The problem was with CopyTo method. Because this method was called a lot of time in a short period of time, the GC didn’t had time to dispose all the resources used during the copy process. This can become a big problem when the size of stream is pretty big. This method was introduce in .NET 4.0 and the scope of it is to copy the content of one stream to another. This is done by reading to a buffer of bytes (default buffer size is 4096) chunks from the stream and write them to the second stream. 
It is very similar with what we usually do in .NET 3.5 when we had to copy one stream to another:
public void CopyStream(Stream input, Stream output)
{
    byte[] buffer = new byte[size];
    int read;
    while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
    {
        output.Write (buffer, 0, read);
    }
}
The problem is with the buffer that is represented as an array of bytes. If we call the CopyTo method from time to time with small streams will not have any kind of problem. But when we use big streams intensive, calling this method very often can create huge problems. Each time when we make new calls to this method, a new part of our memory will be allocated for the buffer. GC will not have time to clean the memory and we can end up with a lot of memory that is not disposed.
The solution for this problem was pretty simple. The CopyTo method was replaced with a BinaryReader that was used to copy the content of stream to an array of bytes. This array of bytes was later used to create the MemoryStream. In this way, the stream that was used for read data is disposed immediately (BinaryReader) and the array of bytes that is used as a buffer is used directly in the memory stream.
byte[] bytes;
using( BinaryReader reader = new BinaryReader(inputStream))
{
    bytes = reader.ReadBytes((int)inputStream.Length); 
}
outputStream = new MemoryStream(bytes, 0, bytes.Count());
Using this solution, the problem with the memory leak disappeared. It seems that not all time, the new API is the best solution when we need performance.

Comments

  1. What if size of input stream will be too big?
    it will try to create huge bytearray, and possible OutOfMemoryException - no?

    ReplyDelete
  2. Wow, thanks, this helps alot !

    ReplyDelete

Post a Comment

Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

Entity Framework (EF) TransactionScope vs Database.BeginTransaction

In today blog post we will talk a little about a new feature that is available on EF6+ related to Transactions.
Until now, when we had to use transaction we used ‘TransactionScope’. It works great and I would say that is something that is now in our blood.
using (var scope = new TransactionScope(TransactionScopeOption.Required)) { using (SqlConnection conn = new SqlConnection("...")) { conn.Open(); SqlCommand sqlCommand = new SqlCommand(); sqlCommand.Connection = conn; sqlCommand.CommandText = ... sqlCommand.ExecuteNonQuery(); ... } scope.Complete(); } Starting with EF6.0 we have a new way to work with transactions. The new approach is based on Database.BeginTransaction(), Database.Rollback(), Database.Commit(). Yes, no more TransactionScope.
In the followi…

GET call of REST API that contains '/'-slash character in the value of a parameter

Let’s assume that we have the following scenario: I have a public HTTP endpoint and I need to post some content using GET command. One of the parameters contains special characters like “\” and “/”. If the endpoint is an ApiController than you may have problems if you encode the parameter using the http encoder.
using (var httpClient = new HttpClient()) { httpClient.BaseAddress = baseUrl; Task<HttpResponseMessage> response = httpClient.GetAsync(string.Format("api/foo/{0}", "qwert/qwerqwer"))); response.Wait(); response.Result.EnsureSuccessStatusCode(); } One possible solution would be to encode the query parameter using UrlTokenEncode method of HttpServerUtility class and GetBytes method ofUTF8. In this way you would get the array of bytes of the parameter and encode them as a url token.
The following code show to you how you could write the encode and decode methods.
publ…