Mike Volodarsky's blog

Formerly the core server PM for IIS 7.0 and ASP.NET, now I run LeanSentry.
UPDATES: New troubleshooting guide released! Fix IIS website hangs →

Things you still can’t do with ASP.NET modules on IIS

IIS7 was revolutionary in opening the IIS web server platform for public extensibility. Prior to that, few software vendors wrote extensions for IIS, using the native ISAPI Filter and Extension APIs. IIS7 completely changed this, creating a public extensibility model on top of which the web server itself was implemented, and opening it for managed development via the familiar ASP.NET API.

Here are a couple things you can do only with native modules, and why you will probably never have to do them.

In the dark ages, there was ISAPI

Before IIS7, the only way to extend IIS was the C++ ISAPI Filter or Extension API. This was as complicated and error prone as any native server development, and also had limited ability to extend IIS. Most of the IIS features themselves were hardcoded into the monolithic IIS core, and ISAPI made it possible to implement some very specific scenarios to "work around" the built-in request processing. Only a few software vendors dared to build ISAPI components, and the rest of the world was forced to buy them to be able to customize IIS.

IIS 7 brought extensibility into the public domain

IIS7 changed this completely, by creating a public extensibility API and literally building itself on top of it. This way, we made sure that anyone could extend IIS to do everything we could do, because we would both build on the same API. As a result, the webserver became a small request processing pipeline with 40+ optional modules that implemented all the features, such as authentication, compression, caching, static file serving, URL rewriting, and so on.

More importantly, IIS7 delivered the Integrated Pipeline, the engine that enabled ASP.NET developers to extend IIS with the ASP.NET IHttpModule and IHttpHandler APIs. This was huge, because it effectively eliminated the need for native C++ development in order to extend IIS.

To make this happen, we had to fit the existing ASP.NET API over the new IIS7 model, without breaking backward compatibility with ASP.NET, and allowing the ASP.NET APIs to have as much power to extend IIS as possible. Also while meeting the performance requirements of the NT PERF team, that were historically based on much faster native-only IIS benchmarks. It was an insanely complicated technological feat, which I'll talk about in another post.

So, is there anything I cant do with managed modules?

There are still a couple things that you have to write a native module for.

1. Run in PRE_BEGIN_REQUEST.

This is a special event that occurrs at the very beginning of the request, before IIS has mapped the URL to a physical location, and before it has loaded configuration. A few select native modules use this event for very rapid processing, e.g. the IIS Request Filtering module uses it to rapidly reject danerous requests.

2. Use module priorities.

IIS allows modules to register for request notifications using a priority level, which can range between LAST, LOW, MEDIUM, HIGH, and FIRST. ASP.NET modules are always registered as MEDIUM, meaning that any native module can always elect to run before regardless of their configuration order.

3. Send response asynchronously.

In my opinion, this is one of the most significant limitations. ASP.NET by default buffers the response in memory, and only provides a synchronous HttpResponse.Flush() method for flushing it to the client. If you wanted to efficiently spool large responses to hundreds of concurrent users, you'd have to write a native module that uses asynchronous response writes.

Correction: you can flush ASP.NET response asynchronously with ASP.NET 4.5! Very happy this finally happened, just like the new ability to asynchronously preload request entity. Interestingly enough, both of these features were assigned to me back in 2007, but we just ran out of time to build them (and I regretted that ever since).

4. Intercept/filter request POST data.

While ASP.NET modules can filter the incoming request entity body via the HttpRequest.Filter stream, its not a true filtering mechanism because it requires preloading the entire request into memory/disk first. Secondly, that data will be loaded by ASP.NET, and cannot be made available to a non-ASP.NET application (like PHP or classic ASP). You would have to write a native module to efficiently filter the request data on the fly, preload the request entity asynchronously, and make it available to non-ASP.NET clients.

5. Handle global notifications: MapPath, ConfigurationChange, HealthCheck, RSCAQuery, etc

IIS has global notifications that allow native modules to override functionality for mapping URLs to files, handling async configuration changes, responding to external status requests via the Runtime Status and Control API, and more. These are not available to ASP.NET modules.

6. Get precise control over performance and resource use.

When people ask me whether they should write a native module, my answer will always be - never! It just takes too long to build, stabilize, and support native code - and its next to impossible to find developers these days who can maintain it well.

Now, dont get me wrong. There are few things I enjoy more than writing C++ code, and carefully moving bits with precise control over memory, IO, and the win32 API. I've personally written quite a few of these very complex, very performance centric native modules, including the IIS7 Bit Rate Throttling module, and LeanServer ScaleUP, the absolutely fastest way to do HTTP uploads to an IIS server. But, as much as I enjoy this, I would never consider writing a native module if its possible to write a managed one, for the same reason that its been years since I've written anything in C++.

Nowadays, you can do almost anything with .NET, using the comprehensive support of the .NET framework and plugging any holes with p/invoke to win32 APIs or COM interop. You can even run .NET on Server Core (which you couldnt do for all of 5 minutes when Windows Server 2008 first came out).

Consider this short but sweet list of reasons why you should only build ASP.NET modules:

  1. They take 10 times less time build, and cost 100 times less to support.
  2. You can find sample code on how to do anything you ever needed.
  3. Any .NET developer can write ASP.NET modules, but I wouldnt let most developers near C++ server code.
  4. ASP.NET modules automatically deploy with your application.
  5. ASP.NET modules can run in shared hosting environments, including Microsoft Azure Websites, where native modules don't (they require Administrative priviledges to install).

This is exactly why the ASP.NET Integrated Pipeline was such a huge win for the Microsoft web platform. It removed native development as a requirement for extending the web server, and made it an option for those few that really need it.

So, next time you find yourself tearing your hair out because of an old unsupported ISAPI filter in your application, consider sitting down for a couple hours and rewriting it to an ASP.NET module. It will be faster, cheaper, and you wont have to pay anyone or make international phone calls next time it breaks.

Troubleshoot hanging requests on IIS in 3 steps

Your users are complaining that the site is loading slowly. Requests to your application are hanging. With so many possible causes of request hangs, its difficult to know where to even start.

I have been helping people with hanging requests for a long time, first at Microsoft, and now at LeanSentry. My first step is always to reproduce and observe the problem. Once you do that, you know what you are dealing with and can formulate an effective plan to fix it.

I see lots of people using the "stab in the dark" method, blindly guessing at things or worse yet making code changes to fix what "they think" it is. This usually just ends up wasting their day/week. And often with little to show for it at the end.

Here is my preferred method for diagnosing hanging requests on IIS servers:

1. Dump hanging requests

Run:

%windir%system32inetsrvappcmd list requests /elapsed:30000 

List hanging requests on IIS

If the requests are currently hanging, this will instantly show you 3 critical things:

1. Which URLs are involved

2. Whether all requests to the app are hanging or just specific URLs

3. The module/request stage they are hanging in.

2. Next, get a detailed request trace

The request trace for the hanging request will give you more information about where the request is hanging.

Also, Failed Request Tracing can be used to capture the hanging requests if you are unable to catch it in real time. You can use it to set rules that capture traces when requests to specific URLs exceed a time limit, or fail with specific errors, etc.

When we were working on IIS 7.0, Jaro (the developer on Failed Request Tracing) and I often talked about the primary weakness of FRT for troubleshooting: that you had to modify the application configuration in order to configure the tracing rules. If you had a problem and configured trace rules to capture a trace, the resulting configuration change would restart the application and would likely reset whatever problem condition you were trying to trace. I was always a big advocate of allowing server variables to be used to turn on tracing at runtime, so that I could then at least write a module to dynamically enable traces without touching configuration. Unfortunately, this was never done for the FRT feature. This was one of the reasons that we built a custom trace provider for LeanSentry, that is hosted completely outside of the IIS application and cannot affect it.

To enable FRT rules, run:

%windir%system32inetsrvappcmd configure trace "Default Web Site" /enablesite 
%windir%system32inetsrvappcmd configure trace "Default Web Site" /enable /path:test.aspx /timeTaken:00:00:30

Turn on IIS Failed Request Tracing for hanging requests

Then, after a while, if you were lucky enough to capture the traces, you can find them like this:

appcmd list traces | findstr "test.aspx"

When I wrote AppCmd, I added the ability to specify search filters by doing things like /url:$=*test.aspx* for any parameter. This was not a planned feature at the time (many AppCmd features I added werent), so I got quite a bit of pushback from the test team on it. The unfortunate result for this is that the list traces command did not receive this functionality, which makes it harder to quickly search FRT traces using appcmd. Hence the hacky usage of findstr instead.

Once you have the trace, you can quickly bring up the trace with this trick:

appcmd list traces "Default Web Site/fr000003.xml" /text:path > temp.bat && temp.bat

Detailed request trace of the hanging request

Head over to the "Complete request trace" tab to view the events leading up to the hang. Keep in mind that the trace cuts out on the last event before the timeout hit (another unfortunate but easily understandable design decision). This always confuses people expecting to see a big fat “30 seconds spent here!”. The last event should give you an idea of where the hang started.

3. Diagnose it

At this point, you already have a lot of information about what’s happening. You just don’t know why yet. Here are the questions I usually ask:

1. Are all requests to the application hanging or just requests to specific URLs?
If all, I look for typical signs of an overloaded or deadlocked application (resource overload, threadpool starvation, application deadlock). If some, I look for what is causing the request to hang in the application code.

2. Where is the request hanging?
For application-level overload, this usually happens at the beginning of the request processing pipeline (where the request is queued) vs. the ExecuteRequestHandler stage for application code hangs. Differentiating this could sometimes get tricky but it is usually straightforward.

3. Is the request is hung in application code?
You’ll typically see these hanging in the "ExecuteRequestHandler" stage. Then, you’ll need to either be very good at guessing, or break out a debugger to see exactly where the request is stuck.

Guess which approach I usually use :) Of course, debugging in production is an art in itself and is outside of the scope of this post. For the debugger to work, you also need to be able to repro the problem or be lucky to catch it live. We have several techniques we use for rapid debugging, which I’ll blog about in a later post.

Bottom line, hanging or slow requests are a reality for web applications. Its going to happen sooner or later.

The right way to deal with this is to have a continuous monitoring tool in place that catches slow requests whenever they happen, and helps you determine the cause of the hang (obligatory plug: that’s exactly what we built LeanSentry for).

Otherwise you better be ready to spend some time trying to catch the problem, and then roll up your sleeves and bust out some good old troubleshooting techniques.