Think you don't have this problem?  Run taskmgr on your server and see how much RAM winlogon.exe is using.  On one of my web servers I noticed winlogon.exe using 2.4 GB of ram.  I just assumed it was somehow related to IIS on the machine.  Turns out, it's just a memory leak that accumulates every time you remote desktop to the server.  I realize I'm just rehashing something already reported, but man, this seems too important not to do so.

How to fix it?  launch remote desktop with mstsc /console.

The original source of this info: 

http://stuff.doingwhatworks.com/index.cfm?mode=entry&entry=6B2CD03E-9C49-AA70-D1909DA73B852F9F

Awesome Awesome Awesome.  The server has 4 gigs of ram, but I'm sure that couldn't have been helping anything.  Maybe it's just psychological, but my websites seem faster now.