Skip to main content

Steam.CopyTo - Performance problems

This week I heard about an interesting performance problem, that I want to share with you.
There is a desktop application that is used to import data from files to database. These files were in different formats, from XML to binary. Everything was fine, the application work great in the development and first phase of testing. All the code was double check, each stream was dispose when it was no longer used.
During the testing phase, the application started to have an odd behavior, the RAM memory started to go crazy. Normally, the application use around 200MB during an import, but there were times when more than 1500MB of ram were consumed. Not only this, but “Out of memory” exception appeared a lot of time during different imports. All this started too appeared at the testing phase.
With the same setup, developers manage to reproduce the problem and using the memory profiles that comes with Visual Studio the problem was spotted.
Stream stream1 = …
Stream stream2 = …
stream2.CopyTo(stream1);
The problem was with CopyTo method. Because this method was called a lot of time in a short period of time, the GC didn’t had time to dispose all the resources used during the copy process. This can become a big problem when the size of stream is pretty big. This method was introduce in .NET 4.0 and the scope of it is to copy the content of one stream to another. This is done by reading to a buffer of bytes (default buffer size is 4096) chunks from the stream and write them to the second stream. 
It is very similar with what we usually do in .NET 3.5 when we had to copy one stream to another:
public void CopyStream(Stream input, Stream output)
{
    byte[] buffer = new byte[size];
    int read;
    while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
    {
        output.Write (buffer, 0, read);
    }
}
The problem is with the buffer that is represented as an array of bytes. If we call the CopyTo method from time to time with small streams will not have any kind of problem. But when we use big streams intensive, calling this method very often can create huge problems. Each time when we make new calls to this method, a new part of our memory will be allocated for the buffer. GC will not have time to clean the memory and we can end up with a lot of memory that is not disposed.
The solution for this problem was pretty simple. The CopyTo method was replaced with a BinaryReader that was used to copy the content of stream to an array of bytes. This array of bytes was later used to create the MemoryStream. In this way, the stream that was used for read data is disposed immediately (BinaryReader) and the array of bytes that is used as a buffer is used directly in the memory stream.
byte[] bytes;
using( BinaryReader reader = new BinaryReader(inputStream))
{
    bytes = reader.ReadBytes((int)inputStream.Length); 
}
outputStream = new MemoryStream(bytes, 0, bytes.Count());
Using this solution, the problem with the memory leak disappeared. It seems that not all time, the new API is the best solution when we need performance.

Comments

  1. What if size of input stream will be too big?
    it will try to create huge bytearray, and possible OutOfMemoryException - no?

    ReplyDelete
  2. Wow, thanks, this helps alot !

    ReplyDelete

Post a Comment

Popular posts from this blog

Windows Docker Containers can make WIN32 API calls, use COM and ASP.NET WebForms

After the last post , I received two interesting questions related to Docker and Windows. People were interested if we do Win32 API calls from a Docker container and if there is support for COM. WIN32 Support To test calls to WIN32 API, let’s try to populate SYSTEM_INFO class. [StructLayout(LayoutKind.Sequential)] public struct SYSTEM_INFO { public uint dwOemId; public uint dwPageSize; public uint lpMinimumApplicationAddress; public uint lpMaximumApplicationAddress; public uint dwActiveProcessorMask; public uint dwNumberOfProcessors; public uint dwProcessorType; public uint dwAllocationGranularity; public uint dwProcessorLevel; public uint dwProcessorRevision; } ... [DllImport("kernel32")] static extern void GetSystemInfo(ref SYSTEM_INFO pSI); ... SYSTEM_INFO pSI = new SYSTEM_INFO(

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine: threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration: TeamCity .NET 4.51 EF 6.0.2 VS2013 It see

Navigating Cloud Strategy after Azure Central US Region Outage

 Looking back, July 19, 2024, was challenging for customers using Microsoft Azure or Windows machines. Two major outages affected customers using CrowdStrike Falcon or Microsoft Azure computation resources in the Central US. These two outages affected many people and put many businesses on pause for a few hours or even days. The overlap of these two issues was a nightmare for travellers. In addition to blue screens in the airport terminals, they could not get additional information from the airport website, airline personnel, or the support line because they were affected by the outage in the Central US region or the CrowdStrike outage.   But what happened in reality? A faulty CrowdStrike update affected Windows computers globally, from airports and healthcare to small businesses, affecting over 8.5m computers. Even if the Falson Sensor software defect was identified and a fix deployed shortly after, the recovery took longer. In parallel with CrowdStrike, Microsoft provided a too