In a previous post titled Where did my IIS7 server go? Troubleshooting "service unavailable" errors, I covered the basic steps for dealing with cases where the IIS7 web server fails to process the request. This error mostly occurs whenever HTTP.SYS, the kernel HTTP driver that manages http connections for IIS, fails to create an IIS worker process to process the request.
This failure is typically caused by a critical error during worker process initialization, or more likely an unhandled exception / access violation occurring during worker process startup. The most common instance of this is a module AVing inside its RegisterModule function during worker process initialization. It can also be caused by the process going away due to an unhandled exception / AV during the processing of the request that causes the process to be torn down.
After a certain number of failures, the application pool will trigger Rapid Fail Protection, a WAS feature designed to stop application pools with a persistent failure condition to avoid an endless loop of failing to start worker processes. At this point, all requests to applications within the stopped application pool will result in the 503 error, and the application pool will need to be re-started manually.
My previous post covers the steps necessary to diagnose most of the error conditions that may cause 503 errors. In most of these cases, there should be an event log event in the "Application" event log that indicates why the worker process has terminated.
However, in the case of an unhandled exception or AV, you may need to run the worker process under a debugger to get more information about the error. This can give you critical insight into the problem, including the DLL causing the exception, and the exception stack trace, which you can then use to remove the offending module, or provide details to the developer to fix the problem.
1) First, you need to download and install the latest debugging tools.
2) Restart the application pool if it entered RFP (rapid fail protection) and is currently stopped:
> %windir%system32inetsrvappcmd start apppool <APPPOOLNAME>
Where <APPPOOLNAME> is the name of your application pool.
3) If the problem is happening during request processing of a particular request, you can try to make a different request to the site that does not cause the exception, and then attach the debugger by running the following from the command-line:
> windbg -g -p <PID>
Where <PID> is the PID of the IIS worker process you are debugging. You can get it from Task Manager or by running tasklist /fi "imagename eq w3wp.exe".
If you have the debugger attached at this point, skip steps 4 and 5
4) If the problem is happening on every request, then you won't have a worker process running that you can attach to since it will go away as soon as you make a request. In this case, you will need to attach a debugger immediately after the worker process is started, so that you can catch a startup exception before it tears down the process.
This is where startup debugging comes into play. It is done by configuring the debugger to attach to the process as soon as its started, by setting the Image File Execution Options registry key from the command line:
> REG ADD "HKLMSoftwareMicrosoftWindows NTCurrentVersionImage File Execution Optionsw3wp.exe" /v Debugger /d "c:debuggersntsd.exe -server npipe:pipe=w3wp -g" /t REG_SZ /f
This creates a sub-key named "W3WP.EXE" in HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionImage File Execution Options, and in that sub-key creates a String value named "Debugger" and set its value to "c:debuggersntsd.exe -server npipe:pipe=w3wp -g".
For this to work, you have to make sure that the path to the debugger is correct. If the debuggers are already in the path, you can omit the full path.
5) Now, lets attach.
– Make the request that causes the crash
This will trigger the IIS worker process to be started, and will automatically attach the debuger, breaking into the process before any IIS initialization takes place.
– Now, connect to the debugger from the command-line:
> windbg -remote npipe:server=localhost,pipe=w3wp
5) Inspect the exception stack trace
The debugger window should now be broken in at the location of the exception. You can inspect it further by typing "k" and pressing Enter.
This should give you the full stack trace of the exception.
If you are missing symbols, you can get them from the Microsoft's public symbol server. To do that, type the following in the debugger:
> !sympath+ SRV*c:symbols*http://msdl.microsoft.com/download/symbols
> .reload
> k
When you are done, be sure to disable the automatic debugging on start-up:
> REG DELETE "HKLMSoftwareMicrosoftWindows NTCurrentVersionImage File Execution Optionsw3wp.exe" /v Debugger /f
Hope this helps you get to the bottom of particularly nasty start-up crashes. If the exception stack is within IIS or a third party module, please post it on forums.iis.net so we can take a look.
Thanks,
Mike
In a previous post titled Where did my IIS7 server go? Troubleshooting “service unavailable” errors ,
Working like to “Where did my IIS7 server go” http://mvolo.com/where-did-my-iis7-server-go-troubleshooting-503-quotservice-unavailablequot-errors/
In a previous post titled Where did my IIS7 server go? Troubleshooting “service unavailable” errors
Hi! Mike,
I am using vista business and VS2005 TS w sp1. I have installed IIS (click everything). I also install latest debugger from the link you provided.
I got this problem 503. I hv tried both way easy 1 and this but unsuccessful. I cant any w3wp.exe and the debugger from my vista. Any idea?
dratengon,
I am not sure what the problem is from your post – are you saying you are unable to see the w3wp.exe worker process on your machine?
If so, then most likely there is a problem that is preventing the worker process from ever being started. Please see my other post about troubleshooting these problems: http://mvolo.com/blogs/serverside/archive/2006/10/19/Where-did-my-IIS7-server-go_3F00_-Troubleshooting-_2200_service-unavailable_2200_-errors.aspx.
Thanks,
Mike
I had that error and i wasn’t able to solve it. I was getting crazy.
Finally in Basic Configuration of the website i’ve changed Aplications Group to ASP .NET 1.1 and it works 🙂 if it helps anyone 🙂
Mike,
I’m having trouble woth 503 Errors. I’m running IIS7 on vista premium. I get this error going to any page on localhost ex http://localhost/test1/hello.htm
When I run system32inetsrvappcmd start apppool DefaultAppPool, I get an error message that says WAS Service is not available – try starting the service first.
I downloaded the debugging tools. The path is C:Program FilesDebugging Tools for Windowswindbg.exe
The REG ADD command gives an access denied error but I created a registry key using regedit in HKLMSoftwareMicrosoftWindows NTCurrentVersionImage File Execution Optionsw3wp.exe with value
“C:Program FilesDebugging Tools for Windowswindbg.exe” -server npipe:pipe=w3wp -g
The debugger isn’t being started when I try to go to a web site or when I try to start the appPool
Don’t know what to try. I’ve been fighting with this for a long time now. Any help would be great
Hi Ken,
In your case, the worker process isnt even being started because your application pool cannot be started, so this technique is not applicable.
Try starting WAS first:
> net start was
If this fails, look in the “System” eventlog for errors indicating why WAS could not start.
Thanks,
Mike
Mike, how are you ?
We are experiencing the 1011 error (A process serving application pool ‘AppPool’ suffered a fatal communication
error with the World Wide Web Publishing Service. ) and having our IIS crashed several times a week.
We are running on W2003 and IIS 6.0.
I’ve seen that yu have a procedure for IIS7.0, do you have one for IIS 6.0 ?
Can you help me, please ?
I started running the debug diagnostics but I really don’t understand it.
Thanks a lot.
Hi Francisco,
You can apply this technique to IIS6 as well to attach a debugger at the start of the worker process. However, in your case it sounds like your crash does not happen at startup, so you can attach a debugger normally at any time and wait for a crash to occur.
Tools like DebugDiag can do this for you (monitor for a crash, and then attach a debugger).
Thanks,
Mike
Also, don't forget to disable the health monitoring ping (eg. "Ping Enabled" in the advanced settings for the app pool). Otherwise, by default, it'll kill the w3wp process after 90 seconds of unresponsiveness.
Funny none of the blogs on this topic mention having to do this…
Rick,
Disabling the ping is only necessary if you are attaching a debugger in managed-only attach type (e.g. set in VS Attach dialog).
WAS is able to detect when the worker process has a native debugger attached, and won’t kill it.
To avoid this problem without having to disable ping, when using Visual Studio debugger, be sure to attach with both Managed and Native attach modes.
Thanks,
Mike
Thanks Mikes.
Strange, it sounds like that detection wasn’t working for me then. I was using cdb just as you describe here (not VS or another managed-only debugger). After about 90 sec I’d get an error from WinDbg saying that the connection to the server had been terminated. Disabling the ping stopped that. I saw this on two different Win2k8 machines.
IIS7, right, the savior? All I ever got, on my fresh new install, was “HTTP Error 503. The service…
any idea about this error:
HTTP Error 500.21 – Internal Server Error
Handler “PageHandlerFactory-Integrated” has a bad module “ManagedPipelineHandler” in its module list
Hi Mike, have a situation for you 🙂 Our IIS7 is failing with weeks/months in between occurrences. I tracked down errors in the event logs and found the “Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool.” and a few “”A process serving application pool ‘DefaultAppPool’ terminated unexpectedly” before that. I tested and the DefaultAppPool was indeed causing the 503 error that wouldn’t go away. Now I know that. I came upon your page here and also your “Where did my IIS7 server go? Troubleshooting 503 “service unavailable” errors” post. Thanks, very helpful! So how to fix the problem that keeps causing it mysteriously with long periods in between failures? I see other issues in the event logs around the same time and see several “Faulting application name: w3wp.exe” and “Fault bucket , type 0, Event Name: APPCRASH, Response: Not available, Cab Id: 0, Problem signature:P1: w3wp.exe” in the Application Log. So I think it’s the Application crash scenario you mentioned. I look in the actual web server logs and just see a ton of requests for mysql admin tools which we don’t have around that time. So how do I prevent this in the future? Any suggestions? Thanks for any help!
Hi, Did you resolve this issue? We have exactly the same problem!
AS THE SERVICE UNAVAILABLE IS GONNA FALL DOWN AND 503 ERROR MESSAGES WILL BE BURNED FOREVER AS THE WORLD IS GOING TO END ON DECEMBER 21, 2012 AS WE KNOW IT DUE TO THE VERY VERY VERY LONG LONG MAYAN CALENDAR AND THE END OF OUR PLANET EARTH AND THERE WILL BE HUGE TSUNAMIS, EARTHQUAKES, HUGE SNOWSTORMS OR EITHER TRIPLE SUNS TO RISES ON OUR PLANET EARTH AND NO FUEL, NO WEATHER NO WATER AND NO OTHERS AND WE WILL BE UNABLE TO BE SURVIVE AS WE KNOW IT AS THE RESULT OF THE BOT PROJECTS TO BE BURNING ALL ERROR MESSAGES PRIOR TO THE END OF THE WORLD AS IT IS NOW NEAR TILL DECEMBER 21, 2012 AS WE KNOW IT.
Replied to wrong post!
Hi
I try to make an app pool to accept Enable 32-bit=True …bu all the website that I try to browse thru that apppool return 503.
Windows Server 2008 R2 SP1 x64,
I tried to browse by that apppool simple app with no db drivers or else.
I tried the trick with rpc … witch is unusefull …
I do not know what to do more …
IIS HTTP Error 503. The service is unavailable. | Decoupled Logic
[…] […]
I have solved this in different way – in IIS 7.5 in Application Pool, attribute Identity changed from ApplicationPoolIdentity to LocalSystem