Fix the 3 silent performance killers for IIS / ASP.NET apps

33 Comments

High CPU in ASP.NET application

If you could double your IIS/ASP.NET application performance by making just a few small tweaks, would you do it?

Of course you would!

Struggling with slow IIS website performance? You are not alone.  

Head over to LeanSentry to get a comprehensive performance checkup. We’ll find your actual IIS and ASP.NET performance issues, and even show you how to fix them in code.

 

It can’t possibly be that easy, can it? If you’ve ever done a performance investigation in production, you know the drill. Try to catch high CPU in production, then login to the server to capture a CPU profile (if you are lucky to have VS or ANTS profiler installed). Analyze the CPU profile in hopes of catching something interesting. Make code changes, deploy, wait, rinse, repeat.

Thankfully, that’s not what I am talking about this time.

In the last 5 years, we’ve helped hundreds of companies address performance problems in their IIS & ASP.NET applications. Many of them had the same common problems when it came to high CPU usage.

The good news is, several of these problems are very easy to fix. Perhaps even more importantly, they are easy to find.

1. Handled exceptions & Response.Redirect

You’ve heard this sage advice before: “don’t use exceptions as flow control in your application”. As it turns out, most applications end up breaking this advice here and there. This often happens when dealing with potentially error prone code, which is sometimes wrapped in a try/catch instead of figuring out why its throwing an exception in the first place. This can be a bad idea for many reasons, one of them being excessive CPU usage due to exception handling.

Detect it
To determine whether your application is suffering from too many exceptions being thrown, monitor the “.NET CLR Exceptions# of Exceps Thrown / sec” performance counter. If the value is high (50+) on a consistent basis, you may have a problem.

Fix it
Here are some common areas that can cause excessive exceptions:

  1. ASP.NET MVC rethrows exceptions from controller’s actions multiple times before handling them with the [HandleError] attribute or returning an error. It also does this during view compilation and view resolution when using partial view names like:  
    return View("MyView");

    To fix it:

    Dont use HandleError for control purposes, only for legitimate error reporting.

    Use fully qualified view paths in View():

    return View("~/Views/MyView.cshtml");

    If you have frequently hit URLs that report errors, restructure your app to write HTTP errors via HttpResponse.StatusCode instead of via exceptions.

  2. HttpWebRequest throws a WebException for 404s, 401s, and other non-200 responses that are correctly handled by your code. The Windows Azure StorageClient library is guilty of this as well. This one is a bit harder to fix because you can’t turn off exceptions for failed status codes. The best thing to do is be aware of it, and restructure your call pattern to avoid error responses if possible.
  3. HttpResponse.Redirect(). This is the worst offender, because its incredibly common and also because it throws a ThreadAbortException. Besides the usual exception overhead, this also causes the thread on which its thrown to exit, requiring the CLR threadpool to allocate another thread later. This alone can have a huge CPU impact on a busy application. To fix it, simply use the Response.Redirect(newUrl, FALSE) overload. Make sure that your code exits your code gracefully afterwards since calling this method will no longer abort it.NOTE: ASP.NET MVC already handles Response.Redirect correctly when using the built in RedirectResult in an MVC action, like: 
    return Redirect(newUrl);

2. LINQ to SQL & non-compiled queries

For web apps that use LINQ to SQL or Entity Framework as their data backend, compiling per-request LINQ queries often has the highest CPU overhead. Every time a LINQ query is executed, it is compiled into a SQL query by the LINQ to SQL provider. This is very CPU intensive, and gets progressively worse with more complicated LINQ expressions.

Detect it
While there is no easy way to monitor for this externally, you can assume you have this problem if your application uses LINQ to SQL or EF, and does not explicitly implement query compilation for per-request queries (ask your developer).

Fix it
To fix it, compile your LINQ to SQL queries:

// create compiled query
public static Func<northwnd, string,="" iqueryable<customer="">> CustomersByCity =
    CompiledQuery.Compile(
    (Northwnd db, string city) =>
        from c in db.Customers where c.City == city select c
 );

// invoke compiled query
var myDb = GetNorthwind();
return Queries.CustomersByCity(myDb, city);
</northwnd,>

You can also compile your Entity Framework queries. Starting with Entity Framework 4.5, queries can be compiled automatically by the framework.

There has been a lot of discussion of negative performance benefits of compiled queries. The bottom line is if a query is going to be called in the context of a request, compiling it once and reusing it should always be a net win. Especially when you consider the overhead of per-request query compilation on application CPU usage, and not just the execution speed of the query itself.

3. Memory allocation & “% Time In GC”

We finally come to the last, and possibly the most important, cause of wasted CPU cycles/bad performance for ASP.NET applications: .NET Garbage Collection.

The CLR Garbage Collector (GC) automatically cleans up unused objects allocated by your application in the background. This process can be very efficient for well-tuned applications, but for many applications it can take up anywhere from 10% to 50% or more percent of the CPU time. This problem is very easy to detect, and although it can be more difficult to fix, its almost always worth it.

Detect it
To determine whether your application is suffering from high CPU overhead due to garbage collection, monitor the “.NET Memory% Time in GC” performance counter. If this counter exceeds 5% – 10% , you have a garbage collection problem you should investigate.

This counter is particularly important to monitor after each major deployment of your application, because it can help spot major performance regressions due to changes in memory allocation. In my experience, its always easier to fix memory problems when they are detected right away, rather than hunt for them many code changes later.

Fix it
Fixing memory allocation problems that cause high “% Time in GC” can be tricky. It almost always boils down to either a) reducing allocations, or b) making sure to release references to objects to make them short lived. The GC is a lot more efficient at collecting short lived objects (known as Gen0) than it is at collecting objects that live for a few seconds or longer (Gen1, and Gen2). Your developer will need to profile the memory allocations in your application, and determine the right memory strategy for your application.

I’ll blog about effective production memory profiling techniques (hint: without using a memory profiler) in a future post.

Wrapping up

Watching for and fixing these 3 low-hanging issues could make a big difference in the performance of your ASP.NET application, with a minimal amount of work. There are many other common wins for increasing IIS/ASP.NET performance, I’ll be sure to blog about these later.

Having said that, the most direct way to improve performance is to proactively monitor it and periodically load test/profile it to address the actual bottlenecks in your code.

If you want to learn how to quickly troubleshoot high CPU & memory leaks in production ASP.NET apps, sign up for our new LeanSentry Production Troubleshooting Course. Its a free email course I put together based on our own production troubleshooting techniques.

Best,
Mike

P.S. We are working on automatically spotting these and other problems in ASP.NET apps with LeanSentry. Memory usage diagnostic in particular is coming soon. In the meantime, check the demo to see what problems we already catch.

33 Comments

  2. Anonymous

    LINQ to SQL queries aways a big CPU hog … often we end up writing our own queries with SqlCommand instead anyway.

    Reply

  3. @Anonymous,

    Agreed, LINQ to SQL can definitely sometimes be heavy, we’ve often fell back to writing SQL queries by hand to simplify the query and take better advantage of indexes. Usually we do this only for the slower queries. For the rest, we found the default LINQ to SQL queries good enough (and of course hugely faster to write).

    By contrast, compiling the queries was almost always worth it for CPU savings.

    Reply

  4. Nice write up man. I always fill that compiled query performed much better then the other one.

    I have one question what is the best practice to manage that Garbage collection time?

    Reply

  6. Raymond

    Nice article! Sometimes it takes more time to find these than to really build the apps. Agree 100% with the comments about LINQ and EF, we too use micro ORM frameworks as the alternative.

    Reply

  7. Great content! I’ve personally seen all of these things happen in production environments and fixed them so this is dead on accurate.

    I’m consulting in Irvine doing MVC at the top pay grade and am seeing this type of stuff littered throughout the code in the organization. I appreciate you writing this so I can simply link to it as a resource and share with the team.

    Another frequent problem I’ve seen is heavy use of server-side network requests, database queries, long running processes and uploads tying up threads which hog memory and kill the server. Using caching, prefetching via service and offloading to SAAS can help reduce these problems.

    Reply

  8. @jalpesh, thanks.

    Re: managing GC time, I’ll write more about this in an upcoming post. The short answer is:

    1. Reduce allocation rate.
    2. Release objects before they hit Gen2 (to prevent many full collections)
    3. Reduce number of large objects (>85000 bytes)
    4. Make sure the finalizer thread does not become blocked

    Of course, the tricky thing is figuring out what problem you have when GC is high, and how to fix it. This usually requires extensive debugging (if you want to dive deep, Tess Fernandez blog is a good source of info for how to do memory debugging). We also cover some fast production memory debugging techniques in the LeanSentry Production Troubleshooting course.

    Reply

  11. I find the best way to combat the LINQ to SQL issue and also to increase security is to create and use Stored Procedures (within the DB) and use LINQ to call the SPs.
    No compilation necessary that way… as far as I know.

    Reply

  12. This article talk about Fix the 3 silent performance killers for ASP.NET apps Handled exceptions & Response.Redirect, HttpWebRequest throws a Web Exception for 404s, 401s, and other non-200 responses, ttpResponse.Redirect(). In a most easy and very efficient manner.

    Reply

  15. Ray Causey

    Mike, I like your suggestions to help our sluggish site and will look into them.
    However, item 1.3…we use a large number of Redirect(newURL,False) calls in page loads to react to incorrect conditions (aka not logged in ), but follow it with a Response.End to kill the load. Otherwise, unless I am missing something, a simple Exit will not stop the rest of the page’s code (HTML and vb.net), including controls from being evaluated, processed, expanded, instantiated, etc. Unless the page load thread is stopped, subsequent code starts failing because the page load was exited prematurely and the objects are not setup properly, How else can I stop the page load thread after the redirect?
    Thanks

    Reply

  17. Dave

    We bought a commercial source code that was LINQ heavy and horribly inefficient. Ditching LINQ and going direct (old school) with stored procedures (if you run MSSQL) got us a whopping 4-fold increase in performance. LINQ is good for small sites however and makes things easier with other DBs, but is no match for real coding on SQL Server. Alternatively, I suppose you could use LINQ to call SPs (but why add the overhead?).

    Reply

  18. Nice post.

    My interest is on the Entity Framework perspective.

    EF has proven to me that it makes coding much less maintainable, and adds so much more unnecessary coding to just get some basic simple things.

    For example, to do a single query, and allow non-tech’s to see the raw SQL should any error occur in PROD, and have the query be efficient, really requires 3 extra steps,

    (1) Make a model.
    (2) Make a static compiled EF-Query so as to make it efficient.
    (3) Add extra logic to trap the raw SQL for trouble shooting.

    And in the latter case (3), EF can’t actually do this, until after it connects to the DB, even in the compiled version for that first “cold query call”, to build the raw SQL.

    Also, in practice, with many dev’s using EF, I see them build their raw SQL in SSMS, to ensure they have it correct. Then throw the SQL away, and go to EF, build models and then build lync statement to do the same as the query. This is a waste of time and not maintainable, not to mention, the EF’s slowness when running without a compiled qry, which requires extra steps to make it efficient.

    Reply

    • Beefydog

      Totally correct on EF. What I’ve found is that, while some believe it to be more manageable for small scale applications, it does not scale (not to mention will ALWAYS be slower). EF troubleshooting can gobble up time like crazy. I converted an EF site as an example to get a contract to old school (let the database do the work, stupid), my preferred way and it ran twice as fast w/o cacheing and 5X faster w/cacheing, used significantly less CPU time, AND there’s no “Black Box” to have to contend with. The problem, honestly, is with devs that refuse or are afraid to learn SQL Server. It’s not that hard to write nor debug! EF has it’s place, but is only good for simple stuff. Advanced stuff requires a real knowledge of DBMSes. There’s a LOT more than just CRUD.

      Reply

  19. Thanks for this information. It’s been about 14 years since I needed (or wanted) to know any of this backend stuff, but right now I’m building a website for a cookbook I just finished writing and I am trying to test it locally. So I needed to setup IIS8x and your info was very clear and concise (although I’m still going ‘huh?!’, :), but not with quite the panic/stress I had two hours ago.)

    Reply

  22. I don’t know what I’m talking about. To speechless, that PERFECT TOOL, this of you LeanSentry.

    I checked in this post, since I’m experiencing problems on my server, with the help of the tool, I found even the method that is giving problems.

    PS. sorry for the bad English

    Reply

  25. Dennis Jakobsen

    I should have probably read the date on this article before “fixing” my application. Now that i have reduced my application’s ThreadAbortException’s from an average of 700/sec under load to 0, i find out it’s actually slightly slower now. But hey, i’m doing the right thing 🙂 Maybe MS fixed a recent version of .NET, so these ThreadAbortException’s arent as expensive?

    Reply

    • Hey Dennis,

      The overhead of exceptions has to be viewed relative to the overhead of the rest of your codepath. It’s highly likely that your application has way worse problems than TAE, or perhaps that your fix introduced additional overhead that was worse.

      However, for many high-traffic applications that are already fairly efficient, doing this will have a noticeable improvement.

      As always, profile first to identify your biggest issues and then fix them first before tacking other stuff.

      Best,
      Mike

      Reply

  26. Michael

    Hi,

    I’m using appdynamics to monitor the CLR garbage collector Gc time spent and I have a lot of notification about this because the the GC time spent is more than 50%. But it’s for a short time (5 minutes). I compare the metrics between the CLR and the CPU used, nothing correlate. Is it critical ?

    Thank you

    Reply

    • Hey Michael,

      Thanks for checking in. Time in GC of 50% is pretty bad, and will have a noticeable performance impact to your entire application. You should shoot for time in GC of 10% or less, I prefer under 2-5% ideally.

      Have you analyzed your memory usage and identified the negative patterns that are causing such a high GC overhead in production?

      Just FYI, you can use LeanSentry to do this fairly quickly.

      Best,
      Mike

      Reply

  29. Thank you for providing such information about.net and IIS coding language, I am working as a dot net developer in a reputed organization but because of not knowing about new updates my promotion was stuck but now I can proof myself after reading your post and get good salary hike in my office.

    Reply

  30. If time in GC of 50% is not too good, what about 35% time in GC? I’m asking because mine is always between 35 to 40%. Thanks for sharing this awesome article.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *