Where did my IIS7 server go? Troubleshooting 503 "service unavailable" errors
In the
previous post, I covered the “server not found” error that is a common and frustrating problem that may occur for a number of reasons after a configuration change, or when deploying a new server machine. Another common “what the hell just happened” error is the plain 503 “service available” error that looks like this:
What is it?
This error is generated by the WAS (formerly W3SVC) service, which is responsible for creating IIS worker processes to handle incoming http requests. When WAS fails to create a worker process, it will generate this error.
Why does it happen?
There are a number of reasons why WAS may fail to start an IIS worker process. These include invalid application pool configuration, failure to create the process due to incorrect application pool identity settings, bad IIS configuration that causes the worker process to fail to initialize, or a process crash due to application logic. I will list the most probable reasons and how to investigate / fix them below.
How do I get rid of it?
Let’s get rid of it in 4 easy steps:
STEP 1: Check whether your application pool is stopped
Once WAS fails to start your worker process a set number of times, it goes into Rapid Fail Protection Mode and stops the application pool. This prevents application crashes from taking down the machine by going into a start process / fail loop when a persistent error exists. By default, Rapid Fail Protection is set at 5 times in 5 minutes.
Once the application pool that contains your application has been stopped, all requests to applications in it will result in that plain 503 error. So, that’s the first thing we check:
> %systemroot%\windows\system32\inetsrv\AppCmd.exe list apppools 
If your application pool is stopped, then we are on the right track. The next step is to check the event log for information about why your worker process could not be started.
STEP 2: Check event log
If IIS inside the worker process fails to initialize, or WAS is unable to create the worker process, it will log an event to the “Application” or “System” windows event log respectively. Let’s check this next:
> eventvwr
Navigate to the “Application” event log first. Look for error events from IIS-W3SVC-WP source.
In this case, the IIS worker process could not initialize because it failed to load a module DLL (the path of which I “accidentally” misspelled in my configuration).
If the error occurred before the worker process could be started, such as with WAS failing to create the process, the error will be in the “System” event log. In my case, the IIS worker process had the initial error, and WAS eventually disabled the application pool after triggering rapid fail protection, leaving the following in the “System” event log:
----------------------------------------------------------------
UPDATE 12/10/2007:
The IIS 7.0 Health Model has been published, containing details about all Event Log error codes that are logged for worker process and service (WAS) level conditions. It also includes the suggested diaagnostics and workaround steps for each error condition: http://technet2.microsoft.com/windowsserver2008/en/library/b19873a2-9f72-40c8-b73d-89f39cda62781033.mspx?mfr=true.
When confronted with event log messages from IIS, use this document to find and resolve the error using the health model as reference.
-----------------------------------------------------------------
STEP 3: Fix the error condition
This step of course depends on the specific error condition that you are experiencing, as indicated by the event log errors from the previous step.
Be sure to use the health model link above as reference for most of IIS error conditions that result in EventLog errors.
The following are common:
WAS failed to start the worker process:
- The configuration is invalid.
- The application pool identity has wrong account name or password.
- The maximum number of worker processes is reached or out of resources.
IIS initialization failed:
- The configuration section is invalid
- A module DLL listed in has invalid path, or failed to load
- A module failed to initialize, returning an error from its RegisterModule entrypoint
Application crash:
- A module, or application component has generated a debug break, or memory access violation, causing the process to terminate abruptly.
Assuming you were able to correct the underlying error condition, you should be able to re-start the application pool and try again.
STEP 4: Re-start application pool
Now that the error is fixed, we can start the application pool and try the request again:
> %systemroot%\windows\system32\inetsrv\AppCmd.exe start apppool DefaultAppPool
(Replace
DefaultAppPool with the name of your application pool).
You should see that the application pool successfully started.
Now, you should be able to re-try your request and verify that the error has gone away. If not, rise and repeat.
In the case of an application crash, you may need to debug the worker process to get to the bottom of the error. You can read about this here:
Troubleshooting IIS7 503 "Service unavailable" errors with startup debugging.
You can find the first post about the timeouts and “server not found” errors here:
Where did my IIS7 server go? Troubleshooting "server not found" errors.
Also, you can find a post about general IIS7 error diagnostics here:
Troubleshoot IIS7 errors like a pro.
I hope you find that this takes some of the mystery out of these problems, and helps you get going quickly when you hit this on your server.
I will cover some other useful troubleshooting techniques in future posts - as always, feel free to suggest topics you care about.
For the past 5 years, I was the core Program Manager for Microsoft ASP.NET 2.0 and IIS 7.0 products. I drove the design and development of the IIS 7.0 web server core, the IIS FastCGI support, the AppCmd command line tool, the ASP.NET Integrated pipeline, and other special projects around server security, performance, and scalability. Now, I am working on my own on cutting edge web server tech on top of the Microsoft IIS platform, and continue blogging about it here.