Here's a graph from New Relic showing that the w3wp.exe process (IIS) is climbing from 0GB to 2GB in about 11 hours:
This is on a staging server with something like 5 users (probably more like 2-3 active users at a time). They are probably spending most of their time in Umbraco publishing things. Any idea of how exactly I could find the cause of this?
Note that I'm not using any caching, aside from some minimal caching in static variables. I'm not using macro caching and I'm not using donut caching.
Note that this staging server has only recently gone live. Here's a graph of memory usage over multiple days:
That first low spot is before the site was being used. The short dip in the middle is probably overnight when nobody was using it. And the large low spot to the right is over a weekend. The memory goes back down whenever the application pool restarts (which is set to happen periodically... the default for IIS). The high memory usage in the most recent time period is probably higher than the other two because there are more active users of the CMS (2-3 instead of 1-2).
My workaround for the time being is to have IIS automatically restart the app pool whenever the memory approaches 1.5GB, though that is not ideal.
Just recycled the app pool a couple hours ago, and now the CPU is pegged. Can't publish anything and everything around the site is super slow.
Any ideas? Maybe a locking issue that has been fixed in a newer version of Umbraco? Or something of that sort. Here are some recent items in the IIS log (scrubbed of sensitive info):
Notice that it's taking up to 10 seconds to do very simple operations. I could fix this with an application pool restart, but I'm sure the issue would just resurface again.
I wonder if it has to do with the fact that I changed a password recently (there is a bug in my version of Umbraco that requires the app pool to restart before the new password will take effect). I did a New Relic thread profile and here are a couple hot spots:
Seems like there is some thread locking happening and some weirdness with the handling of users.
I'm thinking the memory may be a non-issue (perhaps just IIS avoiding garbage collection when it's not necessary). However, the CPU issue is still a problem.
I'm trying to profile the CPU with dotTrace, but keep running into issues (the latest being an apparent incompatibility between dotTrace and New Relic). Will update if I figure anything out, but if anybody has dealt with this before, I'd like to hear about your experience.
For memory, if you use Ants memory profiler it will tell you the type of memory used. Depending on how much memory your server has will depend on how IIS feels like hanging on to memory. For example, I have 16GB of RAM on my machine and for some larger sites when I debug, ANTS might show me something like this:
Notice that the vast majority is just empty/unused/reserved memory. The managed memory is actually only consuming 120mb
ANTs was the only mem profiler that seem to be able to give me this info when I was debugging
Yeah, IIRC that's basically what dotMemory said. dotMemory also has a garbage collection feature, which when I tried it reduced the amount of memory used.
I was only worried about the memory because I was thinking it may be causing the CPU issue, but now I'm pretty sure that's not the case.
For anybody curious, I believe the CPU issue has likely been resolved. Haven't tested it thoroughly, but I did upgrade to Umbraco 7.2.4 and haven't seen the issue since. Refer to this issue for the cause/resolution (basically, a bug with the change tracking when multiple people were logged in with the same user): http://issues.umbraco.org/issue/U4-6292
Again, the memory issue is not really a problem. Just IIS avoiding garbage collection as much as possible (for performance reasons, from what I gather).
Memory Climbs 2GB in 11 Hours in Umbraco 7.1.8
Here's a graph from New Relic showing that the w3wp.exe process (IIS) is climbing from 0GB to 2GB in about 11 hours:
This is on a staging server with something like 5 users (probably more like 2-3 active users at a time). They are probably spending most of their time in Umbraco publishing things. Any idea of how exactly I could find the cause of this?
I recall that the PetaPoco memory fix was supposed to fix some of this issue in a previous Umbraco version: http://umbraco.com/follow-us/blog-archive/2014/10/2/umbraco-717-and-623-saving-you-loads-of-memory
Note that I'm not using any caching, aside from some minimal caching in static variables. I'm not using macro caching and I'm not using donut caching.
Note that this staging server has only recently gone live. Here's a graph of memory usage over multiple days:
That first low spot is before the site was being used. The short dip in the middle is probably overnight when nobody was using it. And the large low spot to the right is over a weekend. The memory goes back down whenever the application pool restarts (which is set to happen periodically... the default for IIS). The high memory usage in the most recent time period is probably higher than the other two because there are more active users of the CMS (2-3 instead of 1-2).
My workaround for the time being is to have IIS automatically restart the app pool whenever the memory approaches 1.5GB, though that is not ideal.
Just recycled the app pool a couple hours ago, and now the CPU is pegged. Can't publish anything and everything around the site is super slow.
Any ideas? Maybe a locking issue that has been fixed in a newer version of Umbraco? Or something of that sort. Here are some recent items in the IIS log (scrubbed of sensitive info):
Notice that it's taking up to 10 seconds to do very simple operations. I could fix this with an application pool restart, but I'm sure the issue would just resurface again.
I wonder if it has to do with the fact that I changed a password recently (there is a bug in my version of Umbraco that requires the app pool to restart before the new password will take effect). I did a New Relic thread profile and here are a couple hot spots:
Seems like there is some thread locking happening and some weirdness with the handling of users.
I'm thinking the memory may be a non-issue (perhaps just IIS avoiding garbage collection when it's not necessary). However, the CPU issue is still a problem.
I'm trying to profile the CPU with dotTrace, but keep running into issues (the latest being an apparent incompatibility between dotTrace and New Relic). Will update if I figure anything out, but if anybody has dealt with this before, I'd like to hear about your experience.
For memory, if you use Ants memory profiler it will tell you the type of memory used. Depending on how much memory your server has will depend on how IIS feels like hanging on to memory. For example, I have 16GB of RAM on my machine and for some larger sites when I debug, ANTS might show me something like this:
Notice that the vast majority is just empty/unused/reserved memory. The managed memory is actually only consuming 120mb
ANTs was the only mem profiler that seem to be able to give me this info when I was debugging
Yeah, IIRC that's basically what dotMemory said. dotMemory also has a garbage collection feature, which when I tried it reduced the amount of memory used.
I was only worried about the memory because I was thinking it may be causing the CPU issue, but now I'm pretty sure that's not the case.
For anybody interested, here's the YouTrack issue for the CPU issue: http://issues.umbraco.org/issue/U4-6292
yup debugging that now
Cool, let me know if I can help in some way. I'm off to be now, but I'll be deploying that site affected by the issue when I wake up.
For anybody curious, I believe the CPU issue has likely been resolved. Haven't tested it thoroughly, but I did upgrade to Umbraco 7.2.4 and haven't seen the issue since. Refer to this issue for the cause/resolution (basically, a bug with the change tracking when multiple people were logged in with the same user): http://issues.umbraco.org/issue/U4-6292
Again, the memory issue is not really a problem. Just IIS avoiding garbage collection as much as possible (for performance reasons, from what I gather).
is working on a reply...