Skip to main content

Steam.CopyTo - Performance problems

This week I heard about an interesting performance problem, that I want to share with you.
There is a desktop application that is used to import data from files to database. These files were in different formats, from XML to binary. Everything was fine, the application work great in the development and first phase of testing. All the code was double check, each stream was dispose when it was no longer used.
During the testing phase, the application started to have an odd behavior, the RAM memory started to go crazy. Normally, the application use around 200MB during an import, but there were times when more than 1500MB of ram were consumed. Not only this, but “Out of memory” exception appeared a lot of time during different imports. All this started too appeared at the testing phase.
With the same setup, developers manage to reproduce the problem and using the memory profiles that comes with Visual Studio the problem was spotted.
Stream stream1 = …
Stream stream2 = …
The problem was with CopyTo method. Because this method was called a lot of time in a short period of time, the GC didn’t had time to dispose all the resources used during the copy process. This can become a big problem when the size of stream is pretty big. This method was introduce in .NET 4.0 and the scope of it is to copy the content of one stream to another. This is done by reading to a buffer of bytes (default buffer size is 4096) chunks from the stream and write them to the second stream. 
It is very similar with what we usually do in .NET 3.5 when we had to copy one stream to another:
public void CopyStream(Stream input, Stream output)
    byte[] buffer = new byte[size];
    int read;
    while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
        output.Write (buffer, 0, read);
The problem is with the buffer that is represented as an array of bytes. If we call the CopyTo method from time to time with small streams will not have any kind of problem. But when we use big streams intensive, calling this method very often can create huge problems. Each time when we make new calls to this method, a new part of our memory will be allocated for the buffer. GC will not have time to clean the memory and we can end up with a lot of memory that is not disposed.
The solution for this problem was pretty simple. The CopyTo method was replaced with a BinaryReader that was used to copy the content of stream to an array of bytes. This array of bytes was later used to create the MemoryStream. In this way, the stream that was used for read data is disposed immediately (BinaryReader) and the array of bytes that is used as a buffer is used directly in the memory stream.
byte[] bytes;
using( BinaryReader reader = new BinaryReader(inputStream))
    bytes = reader.ReadBytes((int)inputStream.Length); 
outputStream = new MemoryStream(bytes, 0, bytes.Count());
Using this solution, the problem with the memory leak disappeared. It seems that not all time, the new API is the best solution when we need performance.


  1. What if size of input stream will be too big?
    it will try to create huge bytearray, and possible OutOfMemoryException - no?

  2. Wow, thanks, this helps alot !


Post a Comment

Popular posts from this blog

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine:
threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration:

TeamCity.NET 4.51EF 6.0.2VS2013
It seems that there …

How to check in AngularJS if a service was register or not

There are cases when you need to check in a service or a controller was register in AngularJS.
For example a valid use case is when you have the same implementation running on multiple application. In this case, you may want to intercept the HTTP provider and add a custom step there. This step don’t needs to run on all the application, only in the one where the service exist and register.
A solution for this case would be to have a flag in the configuration that specify this. In the core you would have an IF that would check the value of this flag.
Another solution is to check if a specific service was register in AngularJS or not. If the service was register that you would execute your own logic.
To check if a service was register or not in AngularJS container you need to call the ‘has’ method of ‘inhector’. It will return TRUE if the service was register.
if ($injector.has('httpInterceptorService')) { $httpProvider.interceptors.push('httpInterceptorService&#…

Fundamental Books of a Software Engineer (version 2018)

More then six years ago I wrote a blog post about fundamental books that any software engineer (developer) should read. Now it is an excellent time to update this list with new entries.

There are 5 different categories of books, that represent the recommended path. For example, you start with Coding books, after that, you read books about Programming, Design and so on.
There are some books about C++ that I recommend not because you shall know C++, only because the concepts that you can learn from it.


Writing solid codeCode completeProgramming Pearls, more programming pearls(recommended)[NEW] Introduction to Algorithms


Refactoring (M. Fowler)Pragmatic ProgrammerClean code[NEW] Software Engineering: A Practitioner's Approach[NEW] The Mythical Man-Month[NEW] The Art of Computer Programming


Applying UML and Patterns (GRASP patterns)C++ coding standards (Sutter, Alexandrescu)The C++ programming language (Stroustrup, Part IV)Object-oriented programming (Peter Coad)P…