Troubleshoot IIS7 errors like a pro
So you just moved your application to an IIS7 server, and now you are getting an error when you make a request to it. This seems to be a popular theme on IIS.NET forums these days, and after answering a hundred or so of these posts, I figured I should do something to help people track down and fix their issues with a little less frustration.
Server software, and web servers specifically, are very complex and highly configurable systems that support multi-tier applications using a variety of technologies and subsystems, and endure further complexity due to security, performance, and reliability demands, so it’s a wonder it all works as well as it does in the end. IIS7 strives to improve the experience of diagnosing and solving problems when they do occur, so knowing how to use the new IIS7 diagnostics features can come a long way in helping you become a pro at troubleshooting server problems.
First things first - what’s the problem?
I often find myself asking this question on the forums when someone posts something like “Help, I moved to IIS7 and now my application doesn’t work!”. Huh, what doesnt work? When your site stops working, the first thing you need to do is determine what the error actually is before we can move any further. IIS7 has much better error messages that will often tell you exactly what the problem is, and help you fix it.
Unfortunately, sometimes getting to this error is the main challenge. Let’s look at a few things you may need to do to get there:
1) Disable IE “Friendly HTTP error messages”
IE will by default replace the actual error messages coming from the server with a “friendly” error message, which hides the error contents we need to see. For example, for a 404 Not Found error, you may instead see the following:
To disable this and see the real error coming from the server, you need to go to “Tools > Internet Options”, choose the Advanced tab, and clear the “Show friendly HTTP error messages” checkbox. Then, close the browser, open it again, and re-request the page.
Now, suppose you are still seeing the generic “cannot display the webpage” or “page not found” error. This is typically caused by IE failing to connect to your website, so there is no server error message to look at here. I have a detailed guide to troubleshooting this condition here: Where did my IIS7 server go? Troubleshooting "server not found" errors.
An aside: If you don’t want to mess with IE for troubleshooting (especially if you suspect you are getting cached pages, due to IE’s super- aggressive caching), the best thing to do is break out a trusty simple http tool of choice and make the request with it. I recommend WFETCH, which is included in the IIS6 Resource Kit tools (you can install on XP/Windows 2003 only, and can either run it from there or copy wfetch.exe to your Vista/LHS box).
2) Enable IIS7 detailed errors
IIS7 introduces a new custom errors feature, which by default hides the error responses issued by the server to remote clients, replacing them with a basic error message. This is critical for security of your site, as errors frequently contain sensitive information that you don’t want others to see, but makes getting to the bottom of the problem harder since you cannot see those very error details. So, if you are requesting your site from another machine, you may still get a basic error that looks like this:
You have two options here:
1) Make the request locally from the server machine.
By default, you will get the detailed error.
2) Enable detailed errors for remote clients.
First, if your error is an ASP.NET exception (you can tell if it says “Runtime Error” and has the framework version), please be aware that ASP.NET overrides the IIS custom error mechanism with its own implementation of custom errors, so you should turn the ASP.NET custom errors off to see detailed ASP.NET exceptions. You DO NOT have to configure IIS7 custom errors for ASP.NET exceptions (it would be silly to have to do it in two places). To turn off ASP.NET custom errors, place the following in your web.config:
<system.web>
<customErrors mode=”Off” />
</system.web>
If the error is not an ASP.NET error, turning off IIS7 custom errors will allow error responses from your application to be sent to remote clients without being censored by the IIS7’s custom errors module.
You can do this from the IIS7 Admin tool by running “Start>Run>inetmgr.exe”, selecting your website/application/virtual directory in the left-hand tree view, clicking on the “Error Pages” icon, clicking “Edit Feature Settings” action, and then selecting “Detailed Errors”.
You can alternatively do this from an elevated command line prompt (Run as Administrator):
> %windir%\system32\inetsrv\appcmd.exe set config "Default Web Site/yourapp" /section:httpErrors /errorMode:Detailed
Where “Default Web Site” should be replaced with your site’s name if different, and “/yourapp” with the virtual path you’d like to enable detailed errors for.
NOTE: When you are done debugging, please return the settings back to custom errors for remote requests, or the security of your website may be compromised. Here is how to do it with AppCmd:
> %windir%\system32\inetsrv\appcmd.exe set config "Default Web Site/yourapp" /section:httpErrors /errorMode:DetailedLocalOnly
Now, you should be getting detailed errors back – for example, the error I was getting before now looks like this (this is the Vista error page – Longhorn Server and Vista SP1 error pages will look much nicer, see this for an example):
Notice that this error contains quite a bit of useful information:
1) The heading contains the substatus code, 404.0, which is an IIS specific code that further describes the problem. The majority of common errors has a unique <status_code>.<substatus_code> combination that you can easily google for additional information.
2) The page indicates what module (IIS Web Core), and in what notification (MapRequestHandler) an error occurred. This information is critical whenever you are debugging server problems, especially on the IIS7+ world when most of them occur in one of the modules that take part in the processing of the request.
3) The page shows you some useful request information, such as the requested url and the physical path to which it resolved. Most 404 errors will be resolved right here, by seeing that the request is being made to a wrong url or resolving to a wrong physical path due to incorrect virtual directory mapping.
4) The “most likely causes” and “what you can try” sections offer the most likely explanation for the error and what you can do about it. They are dead on for the majority of common IIS7 errors.
NOTE: When you are coming to the forums with an error, we expect you to have gotten this far and that you will provide the detailed error to help us with your problem. For more information about the custom errors support in IIS7, check out http://www.iis.net/articles/view.aspx/IIS7/Managing-IIS7/Diagnostics-in-IIS7/Deciphering-Error-Messages/How-to-Use-HTTP-Detailed-Errors-in-IIS7.
3) Detour – the 503 “Service Unavailable” error
Sometimes, you will get an error that looks like this:
This error most likely either indicates a critical misconfiguration of the server that is causing IIS to fail to start a worker process/process your request, or a system overload condition that is causing requests to be rejected early, so no further error information will be available in the response. If you are getting this error, please follow my detailed guide: Where did my IIS7 server go? Troubleshooting "service unavailable" errors.
Deeper diagnostics with Failed Request Tracing (formerly known as FREB)
If the error alone is not sufficient to diagnose the condition, or more information is needed to determine what lead up to the error occurring, or, there is no error (for example, request times out), you can take advantage of the wonderful IIS7 feature called “Failed Request Tracing”.
This feature can be used to generate a detailed trace of events leading up to the failure, much like ETW, but now with more/more useful events and significantly easier to turn on and use for troubleshooting.
More importantly, you can configure Failed Request Tracing to only trace requests that encounter a specific failure condition, such as a specific response status/substatus codes, an event of specific verbosity, or a timeout.
To do this, you are going to need to install it first (if you haven’t already):
On Vista: Control Panel, Programs, Turn Windows Features on or off, check Internet Information Services\World Wide Web Services\Health and Diagnostics\Tracing.
On Windows Server 2008: Start>Run>Server Manager, Roles, Web Server(IIS), Add Role Services, check Web Server\Health and Diagnostics\Tracing
Then, to enable IIS to generate these logs, you need to do two things:
1) Enable Failed Request Tracing for the site you want to trace
In InetMgr, select your site in the left-hand tree view, then under Actions on the right click “Failed Request Tracing …”. There, select the Enable box.
2) Create a Failed Request Tracing rule with the events and failure definition of choice
In InetMgr, select the site/application/virtual directory that you’d like to trace in the tree view, click the “Failed Request Tracing Rules” icon, click the “Add…” action, and follow the wizard to create the rule.
You will be asked what content you’d like to trace (based on an extension), when to consider the request as failed (a list of response status codes, and in Longhorn Server, also event verbosity) or a timeout, and finally the events you’d like to trace. I recommend to leave all events selected by default, unless you are sure you know what you are looking for.
Now, when you make a request, if it meets the failure definition, the detailed trace will be logged as an XML file that you can inspect (or send to us for inspection).
You can by default find this file in %systemdrive%\inetpub\logs\FailedReqLogFiles\W3SVC<ID> folder.
If you double-click this file, it will open with the provided XSL stylesheet that shows the trace events. In Longhorn Server, this stylesheet has gotten a major overhaul to highlight the important information better – for more information, see this: http://blogs.iis.net/bills/archive/2006/10/19/Making-Failed-Request-Tracing-More-Approachable.aspx. You can download the new stylesheet and use it today for your Vista machine.
The Failed Request Tracing log provides verbose execution information for the request, that can be used in a number of ways to further diagnose more complex problems that may result from the operation of more than one module. For example, if a url rewriter module changes a url in the request, which later leads to the static file handler failing to find the file, the SetUrl event can provide the needed information. The log also provides such key info as the times taken by each module to execute, the outcome of each of the built-in module’s operation (often including the reason why something didn’t work), and any times when a module fails a request by setting an error response status code.
Finally, one of the main strengths of Failed Request Tracing is that you do not have to be the client to receive the error information. As an administrator, you can configure it to monitor for errors and log the forensic trace when they occur, allowing you to just leave the feature on in production. Think of it as request logs on steroids. To learn more about Failed Request Tracing, see http://www.iis.net/articles/view.aspx/IIS7/Managing-IIS7/Diagnostics-in-IIS7/Using-Failed-Request-Tracing/Troubleshooting-Failed-Requests-using-Tracing-in-I.
I will post more specific strategies on how to troubleshoot with Failed Request Tracing logs in a future post. In the meantime, you should now be a lot more prepared to deal with errors when they occur, and resolve them before you type that first forum post. And if you do make a post (which you are welcome to do), you’ll be able to provide us with the key information to help you solve your problem.
Thanks,
Mike
For the past 5 years, I was the core Program Manager for Microsoft ASP.NET 2.0 and IIS 7.0 products. I drove the design and development of the IIS 7.0 web server core, the IIS FastCGI support, the AppCmd command line tool, the ASP.NET Integrated pipeline, and other special projects around server security, performance, and scalability. Now, I am working on my own on cutting edge web server tech on top of the Microsoft IIS platform, and continue blogging about it here.