What's using all my memory?

SOLUTION: There's a lot of stuff to read in this thread, so i'll just summarize here to help.

The problem is the VMWare ESX balloon driver which "inflates" the amount of memory "in use" that the OS sees in order to force the OS to dump caches and reduce working set size. This creates a large amount of "in use" physical ram that you can't account for by looking at the processes. You can look at the VMWare console performance graph for the VM's and see the balloon driver in use.

The solution to the performance was to raise the amount of reserved memory for the server, which limits the amount of memory the balloon driver can steal from you.

I have a 32-bit Windows 2003 Terminal Server that is running a local copy of SQL Server (full version, not express) and hosts a Visual Studio 2008 development environment for several developers. The machines ia Virtual Machine running on vSphere 4, with 4GB of memory allocated to it.

When I have 2 or 3 users logged in, it's using more than 4GB of committed memory, but I can't seem to find where most of this memory is being used. When I add up all the memory in Task Manager for all users, it only accounts for about 2GB at most. What's using the rest?

I've tried cycling the SQL server, that only takes the memory that I see allocated to SQL servr out. The same is true of visual studio, if I shut it down, it only takes the amount I see allocated in Task Manager out of the committed memory pool. I have also set the memory options for SQL Server to only use 512MB max.

I don't believe this is a system or file cache issue, since Windows doesn't include that memory in the committed statistics AFAIK.

It's driving me nuts. There should be more than enough memory for all these things, based on the amounts they seem to be using. How do I locate where this extra memory is being used?

Update

Regarding which memory statistics i'm looking at, none of them add up.

For example, right now with 1 user logged on and only running Visual Studio and the WebDev server, and SQL Server and IIS in the background.. I add up Private bytes and I get about 1049740K of memory, but task manager says commit charge is 1146M, leaving about 150Mb unaccounted for.

When 3 users are logged in, it's only about 2GB of private bytes (a lot of memory is shared because the users are just running Visual Studio).

VM Size doesn't add up either, as that is typically not that much different than private bytes. Although, oddly enough, the number is very different in Process Explorer for virtual memory than it is in Task Manager. PE shows virtual memory of SQL Server at 1,791,444K for SQL Server and 1,251,432K for Visual Studio, yet commit charge is only 1146M total for the machine.

There is no column for WS Size in Task Manager, but in PE it lists WS as both Private and Sharable, and those numbers are significantly less than the Commit charge, even added up.

UPDATE:

I know this question has a lot of detail, so maybe the question is not clear.

How do I find which processes are using all the committed memory in my system? The memory given by TaskManager and Process Explorer do not add up properly, and only show about 2GB of committed memory even though 4GB is committed.

From serverfault Mystere Man

• Are you serious?

  I have a 32-bit Windows 2003 Terminal Server that is running a local copy of SQL Server (full version, not express) and hosts a Visual Studio 2008 development environment for several developer

  Just as info:

  MY wvirtual work environment is a 4g memory Windows 7 system with Visual Studio and SQL Server. 64 bit. I use up around 3.5 gb of that REGULARLY. My Visual Studio instance alone is often around 500mb, plus debugging takes a lot of stuff, though possibly not for long. Add to that the usual culprits (help system, browsers)....

  ...you are criminally low to start with. The machine you note is good for maybe one developer in active use. Two if not a lot is used.

  What's using the rest?

  The system uses a lot. Do not forget, all the screens must be rendered, and the graphics card is of no use here - so a copy of all the memory buffer has to go in (which can be a lot - what resolutions, how many screens do we talk here?, given 2003 I assume one per developer)

  Hint:

  Upgrade to:

  • 2008 R2. MUCh bettter RDP, including MULTI MONITOR SETUPS. I work with 2 screens remotely at the moment, 2008 R2 copies my monitor setup from my windows 7 client. VERY nice to work with. This update for me really made remote work SOMEHOW usable - I am used to work with 2+ screens for years now, Remote was always paifull.
  • 64 bit OS. 32 bit just is not it. Even with all the software being 32 bit (VIsual Studio etc.) the 64 bit hsot can have a lot more memory and actually hand out 3gb to every process starting. And ther are quite some runnning.

  And you will need a lot more RAM. I would say 2gb PER DEVELOPER is a pretty good lineup (given 2+ developers). You MAY get away with less, but it is not too likely. Depends on what they actually do - but large applications in Visual Studio take a LOT of memory. Especialy during compiles,

  Mystere Man : Yes, I'm serious. We do not need that much memory per developer because A) the user is not using Internet explorer or doing anything other than running visual studio. That's it. Everything else they will do locally on their machine, not in the remote desktop. I cannot upgrade to a newer OS for several reasons. It's only using Visual Studio, and SQL Server. I've limited the SQL to use only 512MB, Visual Studio uses about 350MB per user on average, easily enough RAM for 3 developers.

  Mystere Man : And memory buffer is not as much as you might suggest. Yes, one screen per developer, at 16bpp and around 1920 x 1200 = about 36MB. However, I would assume this memory would show up in the CSRSS or some other process in TaskManager or PE. The problem here is that there is memory committed but no obvious process using it.

  Mystere Man : Furthermore, you're using Windows 7 as your virtual client. That eats up more than 1GB just logging on. Windows 2003 uses significantly less. Including SQL server, startup memory usage is about 350MB. If we were using Windows 2008, we'd need a LOT more memory (there's only 8GB in the host server, which also runs a number of other services). So no, thanks, but your suggestions are not viable, nor do they address my question which is What is using the memory?

  TomTom : Ok, so no IE - that means no integraed help (hint: help is always rendered using IE HTML engine)

  Mystere Man : I'm aware of that. In most cases, integrated help doesn't seem to work very well anymore anyways. Too much stuff. It's a crap shoot if pressing F1 will bring up relavent data. It's easier to google search for what you need, and the developers do that on their local computers not in the remote one. This is a minimal environment.
  From TomTom

• sql server and vmware tools fight for memory space in a vm. This is probably compounded by running terminal services and vs.

  If you haven't already, go into the vm's settings and create a full 4Gb memory reservation under the resources tab. This'll prevent the baloon driver attempting to reclaim memory within the vm.

  You may also want to configure the options within sql server to ensure it only allocates part of the system's ram. Maybe 2gb max.

  Mystere Man : Yes, I have already limited SQL Server to 512MB. However, i'm not sure if you're addressing my question. My question is, how do I find what processes are using all the committed memory. Adding up the memory that Task Manager says is in use does not equal the amount of committed memory.

  Chris Thorpe : Missing memory is potentially accounted for by the baloon driver, as described above. Ensure you have the reservation set up, then re-check the figures and see if they've lined up. Also, when you installed all of your apps, you made sure to set your session to Install mode before installing?

  Mystere Man : The terminology you are using is confusing. It's not "missing" memory. It's there. The system says i have 4GB. It's just that when 2 or 3 users are logged in, it shows about 4GB of committed memory, but the amount of memory allocated to processes according to tools like Task Manager and Process Explorer does not add up to the amount of memory that is allocated. Does this "balloon driver" allocate memory, or take it? It's not a performance issue per se, it's a "what is allocating my memory" issue.

  Mystere Man : I'm curious, what does setting the session to install mode do, and how does it relate to memory usage?

  Mystere Man : According to information i've read, Windows 2003 will automatically switch to Install mode when installing programs, and even if it did not all it would do is affect how the registry is created. I don't see how this would affect memory usage.

  Mystere Man : I'm marking you the answer to this question because it lead me to investigate the VMWare baloon driver, which gave me much more information on what was going on. Thanks.
  From Chris Thorpe

• It's probably SQL server. By default (if I'm not mistaken) SQL server will allocate as much memory as it can unless specifically configured to consume a specific amount of memory (accessed in the SQL server properties). This allocated memory doesn't show up in Task Manager. What you need to do is to launch perfmon and add the "SQL Server:Memory Manager:Target Server Memory" and the "SQL Server:Memory Manager:Total Server Memory" counters and see how much memory SQL server is willing to consume and how much memory it actually is consuming.

  Mystere Man : Well, as I said in several places, I configured SQL Server to only use 512MB. However, I will do as you say and see what it says.

  joeqwerty : Sorry, I missed that in your post. I suppose it doesn't hurt to look though.

  Mystere Man : Well, those counters claim it's using 262144KB, while TaskManager says the SQL Server process is using 315,128K, and 323,380K VM Size. PE says it's using 323,380K Private and 1,791,956K Virtual Size, and 292,616K Working Set Private, and 22,512K WS Shareable. None of it adds up.
  From joeqwerty