Low Pagefile can cause 503 Service Unavailable on Azure web roles

Is your Azure web role throwing 503 Service Unavailable? It could be because of a poor default Pagefile size.

LeanSentry itself runs entirely on Windows Azure, including our www.leansentry.com web front-ends that are hosted on Large (A3) instances. A couple weeks ago, we had to scramble to address this problem when our site started returning intermittent 503 Service Unavailable errors.

Troubleshooting low virtual memory on Azure

The first sign of trouble was LeanSentry alerting us to 503s. This was also being reported by Pingdom and seen by our CDN provider.

On the server, the application pool was failing to commit memory needed to dequeue and process requests. As a result, requests queued in the HTTP.sys application queue were being abandoned, causing queueing:

A typical signature for this is QueueFull errors as the worker process begins to fall behind in dequeueing requests and the queue exceeds its limit, as well as Connection_Abandoned_By_ReqQUeue which is triggered when the worker process does not process an already-dequeued request in time.

The nasty thing about virtual memory being exhausted (or in this case, really the Commit limit) is that applications don’t just take longer to execute because of paging. They fail, in many generally unpredictable ways. In our case, WAS was failing to PING the worker process and terminating it. LeanSentry now detects worker process failures which was helping, but without LS you would see things like this:

Windows also detected the low virtual memory condition, which triggers the Resource-Exhaustion-Detector event 2004: “Windows successfully diagnosed a low virtual memory condition”:

The default Pagefile size is too low

The A4 (Large) instance has 7 Gb of memory. A typical IT recommendation is to allow the Pagefile to be at least 2x the physical RAM.

However, the Pagefile size is set to 4Gb, almost ½ of the RAM.

In reality, the Pagefile must be able to grow to be large enough to accommodate the Committed memory on the machine – the virtual memory that has been requested for use by applications and needs to be backed by either the physical RAM or the Pagefile.

As long as the sum of Physical RAM + Pagefile is greater than Commited memory, applications will continue to function (although they will get much slower the more the pagefile is used).

Mark Russinovich has a great article explaining Windows memory usage here. In a nutshell, his recommendation for the Pagefile size is:

• MIN: Peak commit – Physical RAM
• MAX: 2x (Peak commit – Physical RAM)

In other words, if your applications end up asking for 10Gb of RAM, and you have 7Gb on the VM, you should have the Pagefile set to 3Gb MIN and 6Gb MAX.

That said, 4Gb on a 7Gb machine allows for 11Gb maximum commit. If your applications use cache extensively, they’ll begin to fail as soon as the total commit grows beyond 11Gb.

Why is the Pagefile set so low in Azure?

It’s hard to say. My theory is that it’s for one of these reasons:

1. Azure wants you to hit the memory limit sooner, so you upgrade to a larger instance that costs more money.
2. Azure wants to limit the IO your VM causes against their shared disks.

These may be somewhat reasonable, because if your applications require a lot more memory than the VM provides, you’ll experience significantly better performance by moving to a larger VM with more physical RAM.

That said, if you experience peak memory rarely and do not want your application to completely fail, increasing the Pagefile will allow it more breathing room when it most needs it.

Increasing Pagefile size on Azure Web roles

To protect your apps against failing if the memory Commit demand grows beyond the physical RAM, you can increase the default Page file size when you deploy your web role.

We do it in our startup script, which is bootstrapped as a startup task:

ServiceDefinition.csdef:

<WebRole name="LeanServer.Sentinel.Dashboard.Web" vmsize="Large">
  <Startup>
    ...
    <Task commandLine="Startup\Startup.cmd &gt;&gt; c:\Startup.cmd.log 2&gt;&gt;&amp;1" executionContext="elevated" taskType="simple" />
  </Startup>
</WebRole>

startup.cmd:

REM Configure pagefile, to increase it from the default custom limit set by Azure (too small, e.g. 4Gb on a 7Gb machine)
call %~dp0pagefile.cmd

pagefile.cmd:

Echo setting PageFile settings ...

wmic pagefileset where name="%systemdrive%\\pagefile.sys" set InitialSize=4096,MaximumSize=10240
if ERRORLEVEL 1 goto FAIL
GOTO END

:FAIL
echo Failed!
:END
echo Done!
exit /B 0

Notes:

• Because Azure sets a specific Pagefile size, you can increase it without restarting.
• If you want to move the Pagefile to another drive (we do this for some of our services), you’ll need to reboot. I’ll blog about this in a future post.

Should you change the Pagefile or just upgrade the VM size?

The full answer to this question is a bit more complex, but the simple answer is: YES, you should as long as you are able to keep you regular memory usage close to the physical memory limits.

Because upgrading to larger VM in order to meet peak memory demand locks you into doubling, or quadrupling your cloud costs long-term … while changing the Pagefile helps you keep your costs lower while protecting you against rare times of high memory demand.

I’ll blog more about optimizing memory usage to keep Cloud costs manageable in an upcoming post. That’s it for now.