Mike Volodarsky's blog

Formerly the core server PM for IIS 7.0 and ASP.NET, now I run LeanSentry.
UPDATES: New troubleshooting guide released! Fix IIS website hangs →

Trace IIS 7.0 Errors Like a Pro with Failed Request Tracing

When you are working with the web server, dealing with errors is a fact of life.  They creep in when applications change, when you deploy to a new server, make configuration changes, or add new functionality.

The IIS 7.0 philosophy with respect to errors is that an error is better than something seemingly working in a way that you don’t expect.  This helps you maintain control over the behavior of your applications, and helps to discover and address any unexpected issues. This is why I don’t mind errors – they help me eliminate incorrect assumptions and systematically insure that things are working in the correct way.

IIS 7.0 makes easier to diagnose web server errors, primarily through an enhanced detailed error pages, and excellent tracing support for detailed diagnostics. You can find the basic IIS 7.0 troubleshooting techniques in my previous post, Troubleshoot IIS7 errors like a pro.

Most of the time, detailed error pages are enough to diagnose most error conditions.  However, sometimes they are not.  This happens if the error is indirectly caused by module interactions in the pipeline, and the error page does not reveal the exact cause. Another reason is when the error is occurring in an AJAX callback or an image/CSS request, and you don’t see the detailed error – or, if you are just not sure which url is encountering the error.

In these circumstances, I break out the trusty Failed Request Tracing. My favorite way of doing this is through the command line, using AppCmd:

Quickly enable and find failed request tracing logs with AppCmd

Failed Request Tracing: from setting it up to diagnosing in 3 easy steps

1)      Open a command line window with the “Run As Administrator” option.  To fly through this with the use of quick cut/paste, make sure that “QuickEdit mode” is enabled for the command line window.

2)      %windir%system32inetsrvappcmd configure trace "<site/url>" /enablesite /enable

This enables failed request tracing for the site, and also generates default tracing rules for the url (more on this later). You can also enable from InetMgr but I can do it faster here.

3)      Make the request(s) that causes an error

4)      %windir%system32inetsrvappcmd list traces

This shows the list of trace files, including the urls, and status codes.  Highlight the trace id of the one you want and right click to copy it.  You’ll paste it below to view it.

5)      %windir%system32inetsrvappcmd list trace "traceid" /text:path

This will display the path to the trace XML file. Highlight the path, and right click to copy it.

6)      Right click to paste the path to the XML file, and press enter.  This will typically pop up the XML file in your browser, causing it to be displayed using the provided Failed Request Tracing stylesheet for quick analysis.

There are two important parts to this.

First, the appcmd configure trace command is used to enable Failed Request Tracing for the site, AND generate a default trace configuration for the url.  Here is that default trace configuration:

        <tracing>

            <traceFailedRequests>

                <add path="*">

                    <traceAreas>

                        <add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module" verbosity="Verbose" />

                        <add provider="ASP" areas="" verbosity="Verbose" />

                        <add provider="ISAPI Extension" areas="" verbosity="Verbose" />

                        <add provider="ASPNET" areas="Infrastructure,Module,Page,AppServices" verbosity="Verbose" />

                    </traceAreas>

                    <failureDefinitions timeTaken="00:01:00" statusCodes="500,400,401,403" verbosity="Warning" />

                </add>

            </traceFailedRequests>

        </tracing>

Translation:  Trace requests to all extensions for this url or any url below, that result in responses with status codes of 400,400,401, and 403, OR take more than a minute to execute, OR generate events with verbosity of Warning or above. That last clause pretty much captures any requests that set an error response code. The trace automatically includes events from all configured providers/areas at the highest verbosity level for maximum diagnostic fun.

A common task is to include non-error response codes just to see what is going on with the requests. To do that, include the /statuscodes parameter to appcmd configure trace and set it to a list of status codes you want traces, such as 200.

Second, the appcmd list trace command is used to find the trace logs you want.  If you have a bunch of these, finding it through Explorer is a pain, because there is no way to tell which XML file corresponds to which url/error, so you have to open each one. With the appcmd list trace command, you can quickly examine the urls / status codes of the logs, and pick the one you want.

Finding Failed Request Tracing logs with Explorer

Failed Request Tracing log files in Explorer: Which one do I want?

You can actually use the appcmd list trace command to search for the exact log you want, filtering by the url prefix, status code, time taken, site name, application pool name, worker process id, and more. Describing how to do all of these will make for a very long post, so here  is an example:

%windir%system32inetsrvappcmd list traces /url:http://localhost/myapp /verb:get /timetaken:$>200

This will show all trace logs for all GET requests to urls under http://localhost/myapp that took more than 200 ms to execute.

For more info on configuring Failed Request Tracing with appcmd, look in the tool help for the appcmd configure trace and appcmd list trace commands.

Happy troubleshooting!

 

Mike

 

 

Connecting to IIS 7.0 configuration remotely with Microsoft.Web.Administration

As you know, IIS 7.0 provides a number of ways to manage configuration other then MWA, including IIS Configuration COM objects for script, IIS Configuration COM objects from .NET or native clients, WMI, the IIS Manager tool (inetmgr.exe), and AppCmd. While all of these can be used to manage an IIS 7.0 box remotely (AppCmd with extra help of a remote shell environment or WMI), MWA remains one of the most common options because its so easily accessible from .NET.

Microsoft.Web.Administration uses the IIS 7.0 configuration objects under the covers, just like all of the other management options (except for IIS Manager, which uses the IIS Management service, and WMI, which uses the WMI DCOM remoting infrastructure).

Incidentally, this also means that Microsoft.Web.Administration can be used to remotely manage a Server Core installation.

This is despite the fact that currently Server Core does not support the .NET Framework - and is possible because MWA uses the IIS configuration COM objects as the underlying provider, which is native. So, while you cannot use MWA to manage a Server Core server locally, you can do it by running the .NET program remotely on a normal Windows Server 2008 or Windows Vista SP1 machine. However, you cannot use IIS Manager to manage a Server Core server remotely, because IIS Manager uses the IIS Management Service for remote management over HTTPS, which currently is not available on Server Core.

This brings us to the core of the issue: MWA uses the IIS configuration COM objects to manage the server, and therefore in the remote case requires DCOM access to the remote machine.

By default, DCOM access is blocked by Windows Firewall. Therefore, when you try to connect to a Windows Server 2008 or Windows Vista SP1 computer using Microsoft.Web.Administration, or another mechanism that also uses the IIS Configuration COM objects (everything except for WMI and IIS Manager), you will receive the RPC_S_SERVER_UNAVAILABLE error.

To enable DCOM access, you'll normally need to open TCP/135 for the DCOM port mapper, and the entire dynamic port range between 1024 and 65535 used by DCOM. Obviously, this is not something you want to do, especially not on a production server.

Luckily, DCOM provides a way to specify a fixed port for each COM+ package, which allows to enable remote configuration access by opening two ports:

  1. TCP 135 - DCOM port mapper
  2. A single TCP port for the IIS configuration COM objects

You can read more about this approach in Using Distributed COM with Firewalls.

------------------------------------------------------------------------------------

Go here to skip the reading, and grab the batch file to configure remote IIS config, and the RemoteConfigTest.exe tool to test IIS config connectivity.

------------------------------------------------------------------------------------

Here are the 3 steps to do it:

 

1) Configure the IIS configuration objects to use a fixed DCOM port

- Run dcomcnfg.exe
- Expand Component ServicesComputersMy ComputerDCOM Config, select the “ahadmin” COM+ package
- Select PropertiesEndpoints tab, click Add:
- Select “Connection-oriented TCP”
- Set fixed port

Dcomcnfg.exe:

DCOM configuration

Setting the endpoint configuration:

Setting a fixed DCOM endpoint

For command line usage, or if you are on server core, here is the equivalent:

> REG ADD HKEY_LOCAL_MACHINESOFTWAREClassesAppID{9fa5c497-f46d-447f-8011-05d03d7d7ddc} /v EndPoints /d "ncacn_ip_tcp,0,<PORT>" /t REG_MULTI_SZ /f

(where <PORT> is the TCP port between 1024 and 65535 you'd like to dedicate to IIS config)

 

2) Open the required ports in the firewall

- Open TCP port 135 for the DCOM port mapper
- Open the selected TCP port

I additionally secure the opened ports as follows:

  1. Private profile only
  2. Set scope to “Local subnet” only, or specific computer addresses on the network
  3. Restrict ports to the process/services that provide DCOM access to the IIS config COM objects 

For command line usage, or if you are on server core, here are the commands:

> netsh advfirewall firewall add rule name="RPC Mapper" dir=in action=allow profile=private remoteip=localsubnet protocol=tcp localport=135 service=RpcSs

> netsh advfirewall firewall add rule name="AHADMIN Fixed Endpoint" dir=in action=allow profile=private remoteip=localsubnet protocol=tcp localport=<PORT> program=%windir%system32dllhost.exe

(where <PORT> is the fixed TCP port you chose in step 1)

You can also use ipsec to restrict access further.

 

3) Insure adequate access

You will need Administrative privileges on the remote machine in order to use IIS Configuration objects. This means that you either need to be logged in as/impersonating the local Administrator account with a syncronized password, or a domain account that has Administrative privileges on the remote server.

Due to UAC, you will not be able to use a local user other then BUILTINAdministrator, even if that user is a member of the Administrators group on the remote machine (that is, unless you disable UAC).

There are additional things you can do by modifying activation ACLs on the COM+ package or DCOM security settings, but these are out of scope of this post.

4) Reboot 

I've seen cases where a reboot is required to pick up the DCOM changes.  Possibly because RpcSs caches the DCOM activation settings and therefore requires a reboot (its not possible to  restart RpcSs).

If you are unable to connect remotely after going through the steps above, be sure to reboot and try again.

Tools you can use: 

1) Batch file that will perform steps #1 and #2 for you:

> iisremoteconfig.bat 5000 - to setup IIS config access on port 5000

> iisremoteconfig.bat 0 - to remove the IIS config access

(NOTE: A reboot may be required to activate these settings)

2) The RemoteConfigTest.exe tool that will attempt a connection and diagnose the connectivity issues if any:

> RemoteConfigTest.exe 192.168.0.150

This tool will alert you if you have a firewall connectivity problem, forgot to register the IIS configuration COM objects, or do not have sufficient permissions to use them. 

Thanks,

Mike