Garbage Collection and Memory Usage

mase posted on Thursday, June 28, 2007

I've built an extensive application upon the CSLA framework with about 80+ business objects.  The site has been rebuilt from classic ASP.  In classic, our web server handled the load pretty well, keeping the CPU between 20% - 80%.  However, we deployed the new ASP.NET app w/ CSLA and had to revert back because our server could not handle it. 100% CPU usage and lots of memory.

Since then, I've gone through and made sure that I'm properly disposing all IDisposable objects correctly. None of my BO's are implementing IDisposable, and only a few of them contain expensive resources that may need it. I also started running some memory tracing/profiling applications (ANTS) on the site.  I've noticed several things here.  Many (in fact, most) of my BO's are kept in memory up to age 17 or 18 sometimes. I expected that they would have been taken care of by GC long before age 18 or even age 10 for that matter.  I do have some cached objects, but am taking these into consideration and filtering them out.

So a few questions...

1. Do I need to implement IDisposable on the select few objects, which are using System.Drawing objects?  They are implented with the "using ()" statement so they should be properly releasing their memory. 
   
2. Since it's undeterminable when GC will run, do I need to worry about BO's that make it to such old ages?  I'm thinking with hundreds or thousands of concurrent users, having so many BO's in memory for such a long time may be causing my problem?  Then again, if there are that many users and therefore objects, GC will run more often - right?
   
3. Can anyone help provide some advice on how to track down the culprit of our application? I'm using ANTS profiler for memory profiling. As well as the IIS Diagnostics Tool with various stress testing tools.
   

So it seems that since the architecture and framework behind my new web application are solid (right?), there must be some places where I can tweak minor things to accomplish bigger performance gains and less resource intensive.  I feel very confident in 90% of my code and 100% of my architecture, so tracking this down has been an interesting challenge.

Thanks for your help.

RockoWPB replied on Thursday, June 28, 2007

I would also like to know if we should implement IDisposable on all our BO's or should we just let GC handle it.

ajj3085 replied on Friday, June 29, 2007

Well, you will need to implement IDisposable if your objects have instance fields which hold on to other IDisposable resources.  If you just have a method that creates such an object and have a using block, you should be ok.  If you do implement IDisposable, make sure you implement a finalizer as well.

It doesn't seem that your object should be getting that old.  Are you storing them into a session?  That could keep them around much longer than needed.  Remember, if you have a reference to even a single object, and that object has references and that other object has references... you can end up with many objects not being collected. 

I would review asp.net best practices and such.  The toughest part of moving from asp to asp.net is forgetting all the bad habits and design tricks for asp.  Its really a totally different beast..

RockfordLhotka replied on Friday, June 29, 2007

Keep in mind that ASP.NET pages created dozens or hundreds of object instances. Every control is at least one object instance, plus the page, and various support objects. On each page request these objects are created.

So when considering the number of objects in memory in a web app, odds are that your business objects are a tiny minority compared to the ASP.NET objects themselves.

In general terms you should avoid implementing IDisposable unless you must. But when you must, you must. And Andy is right, if your business object has an instance field referencing another IDisposable object then you must implement IDisposable, and a finalizer. There's a very specific pattern you must follow to get this right. When you implement IDisposable in VB, the IDE will automatically insert the correct pattern for you. In C# I believe there's a code snippet to help.

But normal business objects should not implement IDisposable.

Remember that IDisposable does not release the object from memory. All it does is give you a chance to release non-.NET resources from memory (like file handles, graphic object handles, etc). So implementing IDisposable gains you nothing in terms of reclaiming memory consumed by .NET objects.

And incorrectly implementing IDisposable can cost you memory and performance because you can make the GC go through extra work to clean memory. That's why it is so critical that you implement the pattern correctly.

mase replied on Friday, June 29, 2007

Thanks for the advice.  It seems what I have done is all correct, from what your telling me.  I'll need to examine the objects in memory more closely. As I thought, I won't need to implement IDisposable on any of my BO's.  What should be the average age of a CSLA BO if it were not stored in the session and has properly released it's references as needed?  Theoretically, my guess is about age 5 or less, right?

The other consideration when looking at the big picture and overall problem is that my server box was probably not adequate to handle such an extensive .NET web application. It is a single-CPU 2GHz machine w/ 2 GB of RAM.  Now with our classic ASP site, it worked like a charm on this machine. So my initial thought was that it should handle the new .NET app just fine.  What do you guys think?  Someone made a good analagy for me in comparing it that shed some light for me.  Even though the .NET framework is faster and more efficient, there is still much more overhead than in Classic, correct?  This could be compared to putting Vista on a machine that had Win 2k on it.  Win 2k ran just great, but when you put Vista on there it's a whole different ball game.

I've plugged up a lot of our memory holes where we were using resource intensive .NET objects, such as Readers, Writers, Images, Files, etc.  Without profiling the actual objects in memory, I can see that the memory usage on our production server seems to be in control much more. A page loads, it goes up, and as soon as the page finishes, you can see the memory decrease slightly as well.  What is considered normal/acceptable memory usage for a large website with CSLA (w/ 80+ different BO's)?  Sure, the answer is "it depends" but in reality, are we saying that 1.5 GB of memory usage is acceptable?  Seems like it would be.

This has been a great help, I'll keep this thread updated with my findings.

RockfordLhotka replied on Friday, June 29, 2007

You need to understand how the GC works. Especially on the server, it doesn't trigger after every page request, or even on a regular schedule. It triggers based on environmental constraints, like overall memory consumption.

In other words, ASP.NET will tend to consume a lot of memory, and will only reclaim that memory through the GC when it needs to do so. The more memory you have, the more it will tend to consume, at least that's my understanding.

Also remember that there are two GCs in .NET - the client version and the server version. They act and work differently. The client version is more aggressive in some ways, and runs more often. The server version runs less often and is more threshold-driven, and when it runs it has a bigger impact in that it blocks other activity.

The idea is that, on the server, you can pause everything for a short while to clean up memory and no one will really notice. But on a client, you can't freeze everything for half a second or something, because that would result in a poor user experience.

Google around - there are a lot of good, deep articles on how the GC works and what to expect from it.

mase replied on Tuesday, July 03, 2007

I have read more into GC and understand better now. However, I still have issues with my web app and I'm not quite sure how I can track down what's going on. I hoped by tracking down my memory leaking issues that it would solve things. Could the memory issues cause the CPU spike, and still be the culprit?

I think my real issue here is the 100% CPU utilization and thought that my memory leaks were causing that.  The site cannot handle much of a load before peaking out at 100%. I've setup performance counters and tracked various items while load tesitng but I'm not sure how to nail down the cause of the CPU spike. The site runs pretty good when there is a small number of Requests/sec. Once it gets to about 20-30 RPS then the CPU is maxed out. Eventually the app stops responding, gives OutOfMemory exceptions and restarts the application pool.

The process still does take up quite a bit of memory, between 800MB-1.5GB ... it will even stay this high when there is little or zero traffic, seems ok considering GC will run when it needs to and free memory if it were needed. One question I had in regard to GC, was having practices about not allocating too much. What should be the proper way to temporarily cast the BO in a repeater loop?  When calling a method to fetch a property from the BO, I cast the object to get it's property. The site is full of repeaters, many on each page in fact - so this could make a difference. Is one of these methods better than the other? Does the first allocate memory the second not?
public string ObjProperty(object obj)
{
Item i = (Item)obj;
return i.Property;
}

public string ObjProperty(object obj)
{
return ((Item)obj).Property;
}

All the help is greatly appreacitaed.  Looking for some next steps here, not sure how to track down what is exactly happening. Soon I'll have a new server that I can properly test in an isolated environment. Until then I'll keep staring at the performance counters and load testing until I can see whats happening.
Thanks

ajj3085 replied on Tuesday, July 03, 2007

Hey,

I would think that the CPU spike may be the problem.  I sounds like there's an infinite loop somewhere that is causing more and more objects to be allocated (and not released, because the loop never returns).  Its possible that you've coding things in such a way that you run out of memory before stack space, although that seems kinda odd.

Are you using Shared / static variables alot?  Using them would cause references to never be released, and may be part of your problem. You should avoid them whenever possible in a web application, because they do NOT get released until the AppDomain is torn down.

The first property get method does allocate a memory address to point to an Item, but it should be released as soon as the method exits, so it should not be an issue.  Such a method looks very odd to me though... I'm not sure why you'd want to give any arbitrary object to a method and attempt to cast it.  It doesn't seem like a good design.

Another thing to look for would be any race conditions.  Again Shared / statics will likely be the cause of problems, although objects in the Application store may also cause problems too if they are used in a non-thread safe way.  Remember, Asp.net will always be a multithreaded application.

mase replied on Tuesday, July 03, 2007

Thanks for your reply. I have a hard time believing that it would be an infinite loop. We have tested the app extremely thoroughly and it runs ok when there are less than 10 users on the site, so the infinite loop would have been caught before the simulated load was tested.  Don't you think?

Hmm, your point about shared/static variables seems interesting. I have been trying to figure out how a reference pointer would not be "released" and still referenced something. I did a search for static on my solution and came up with only a handful of variables - ones that I'm caching and want to keep around anyway. I do have lots of static methods though. With 80+ business objects, each one has a few static methods.  These should be ok, right? Methods where I don't want to require instantiation for access. It doesn't seem that I have even a moderate amoutn of static variables, only static methods on non-static classes.

The reason that the method is accpeting a generic object is because the object is being passed from the page during databinding. This is the only way I've ever seen it, is it possible to pass the actual Item object type from the ASP repeater? I didn't think this was possible. I'll experiment with this.

Any other insights? Other message boards are still suggesting I have a memory leak, because I am getting OutOfMemory exceptions, but I have gone through and updated my code for any possible leaks. Also considering the fact that a few Classic ASP pages are living inside my ASP.NET web app - could this do it?