Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Matthew Kirschner 323 posts 611 karma points
    Feb 13, 2015 @ 17:34
    Matthew Kirschner
    0

    Umbraco crashing on one server every 3ms

    I have load balanced setup of Umbraco 7.2.1 with a central file system and have followed everything on this page.

    I've experienced an issue yesterday and today, around the same time, where our site on node 1 (we have 2 server nodes) will become incredibly slow on the back and front end. When I opened the log, I can see it going crazy with multiple entries every millisecond. I also noticed a trend. The log entries below are getting repeated every 3ms for the better part of an hour. And just like yesterday, they eventually stop and everything works fine. I have recently installed and uninstalled the sassy plugin, so that may have something to do with it.

    015-02-13 10:35:17,044 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Determining hash of code files on disk

    2015-02-13 10:35:17,107 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Hash determined (took 52ms)

    2015-02-13 10:35:17,107 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of umbraco.interfaces.IApplicationStartupHandler

    2015-02-13 10:35:17,169 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of umbraco.interfaces.IApplicationStartupHandler, found 46 (took 49ms)

    2015-02-13 10:35:17,341 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter

    2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter, found 0 (took 2ms)

    2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyValueConverter

    2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyValueConverter, found 20 (took 3ms)

    2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Web.Mvc.SurfaceController

    2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Web.Mvc.SurfaceController, found 5 (took 1ms)

    2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Web.WebApi.UmbracoApiController

    2015-02-13 10:35:17,388 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Web.WebApi.UmbracoApiController, found 66 (took 14ms)

    2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.Media.IThumbnailProvider

    2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.Media.IThumbnailProvider, found 3 (took 0ms)

    2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.Media.IImageUrlProvider

    2015-02-13 10:35:17,451 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.Media.IImageUrlProvider, found 1 (took 0ms)

    2015-02-13 10:35:18,669 [54] INFO Umbraco.Web.Search.ExamineEvents - [Thread 29] Initializing Examine and binding to business logic events

    2015-02-13 10:35:18,919 [54] INFO Umbraco.Web.Search.ExamineEvents - [Thread 29] Adding examine event handlers for index providers: 3

    2015-02-13 10:35:19,060 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of umbraco.interfaces.IApplication

    2015-02-13 10:35:19,060 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of umbraco.interfaces.IApplication, found 7 (took 0ms)

    2015-02-13 10:35:19,716 [54] INFO Umbraco.Web.UmbracoModule - [Thread 57] Setting OriginalRequestUrl: www.research.fsu.edu:80/umbraco

    2015-02-13 10:35:19,998 [54] INFO umbraco.content - [Thread 57] Loading content from disk cache...

    2015-02-13 10:35:20,373 [54] INFO Umbraco.Core.UmbracoApplicationBase - [Thread 59] Application shutdown. Reason: HostingEnvironment

    2015-02-13 10:35:20,670 [54] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner`1[[Umbraco.Web.Scheduling.LogScrubber, umbraco, Version=1.0.5462.37518, Culture=neutral, PublicKeyToken=null]] - [Thread 2] Application is shutting down immediately

    2015-02-13 10:35:20,670 [54] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner`1[[Umbraco.Web.Scheduling.ScheduledTasks, umbraco, Version=1.0.5462.37518, Culture=neutral, PublicKeyToken=null]] - [Thread 2] Application is shutting down immediately

    2015-02-13 10:35:20,670 [54] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner`1[[Umbraco.Web.Scheduling.ScheduledPublishing, umbraco, Version=1.0.5462.37518, Culture=neutral, PublicKeyToken=null]] - [Thread 2] Application is shutting down immediately

  • Paul Sterling 718 posts 1534 karma points MVP 8x admin c-trib
    Feb 13, 2015 @ 19:02
    Paul Sterling
    0

    @Matthew 

    The behavior would be consistent with having shared Examine Indexes or shared umbraco cache.  Make sure the /app_data/temp/umbraco.config is unique per server node - same for /app_data/temp/examineindexes.

    Other than that, you can see that the app starts, then shuts down due to the hosting environment less than 0.5 seconds later:

    2015-02-13 10:35:19,998 [54] INFO umbraco.content - [Thread 57] Loading content from disk cache...
    2015-02-13 10:35:20,373 [54] INFO Umbraco.Core.UmbracoApplicationBase - [Thread 59] Application shutdown. Reason: HostingEnvironment

    So check to make sure you don't have something that could cause a shutdown - i.e. change to config or /bin/ folder.  It also may be helpful for you to dive into the Windows Event log as something is causing your app to shut down, so anti-virus, etc...should log events there.

  • Matthew Kirschner 323 posts 611 karma points
    Feb 13, 2015 @ 19:58
    Matthew Kirschner
    0

    Thanks for the response, Paul.

    I'm using the TempStorage package to handle the local cache for Examine's temp files. I'll double check this to make sure I'm implementing it right. I noticed a line in Umbraco's logs that is inconsistent in subsequent log entries.

    2015-02-13 10:35:19,716 [54] INFO Umbraco.Web.UmbracoModule - [Thread 57] Setting OriginalRequestUrl: www.research.fsu.edu:80/umbraco

    vs

    2015-02-13 11:18:14,447 [97] INFO Umbraco.Web.UmbracoModule - [Thread 91] Setting OriginalRequestUrl: research1.magnet.fsu.edu:80/umbraco

    Shouldn't this consistently be only one of the domains?

    Just for reference, our DNS A record is research.fsu.edu, with each node having an alias of research1.magnet.fsu.edu and research2.magnet.fsu.edu. I have a URL Rewrite that directs /umbraco/ and /umbraco/default/ traffic to research1.magnet.fsu.edu. My distributed calls in umbracoSettings.config look like:

    <server serverName="Research1">research1.magnet.fsu.edu</server>
    <server serverName="Research2">research2.magnet.fsu.edu</server>
    
  • Matthew Kirschner 323 posts 611 karma points
    Feb 13, 2015 @ 20:02
    Matthew Kirschner
    0

    I looked at the Application Event Log and found a warning from ASP.NET. It seems to be thrown as a result of Umbraco attempting to load content from the disk cache. The request path comes from /umbraco/webservices/nodesorter.asmx/js

    Exception information:
        Exception type: ArgumentOutOfRangeException
        Exception message: startIndex cannot be larger than length of string.
    Parameter name: startIndex
       at System.String.Substring(Int32 startIndex, Int32 length)
       at System.Web.Compilation.DiskBuildResultCache.MarkAssemblyAndRelatedFilesForDeletion(String
    assemblyName)
       at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String
    cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart)
       at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath
    virtualPath)
       at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath
    virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean
    ensureIsUpToDate)
       at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext
    context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
       at System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(VirtualPath
    virtualPath, HttpContext context, Boolean allowCrossApp, Boolean
    throwIfNotFound)
       at System.Web.Compilation.BuildManager.GetCompiledType(VirtualPath
    virtualPath)
       at System.Web.Script.Services.WebServiceData.GetWebServiceData(HttpContext
    context, String virtualPath, Boolean failIfNoData, Boolean pageMethods, Boolean inlineScript)
       at System.Web.Script.Services.WebServiceClientProxyGenerator.GetClientProxyScript(HttpContext
    context)
       at System.Web.Script.Services.RestClientProxyHandler.ProcessRequest(HttpContext
    context)
       at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
       at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
    
    
    
    Request information:
        Request URL:
    http://research1.magnet.fsu.edu/umbraco/webservices/nodesorter.asmx/js
        Request path: /umbraco/webservices/nodesorter.asmx/js
        User host address: 146.201.224.125
        User: Kevin John
        Is authenticated: True
        Authentication Type: Forms
        Thread account name: AD\DomainDavidsonIIS
    
    Thread information:
        Thread ID: 95
        Thread account name: AD\DomainDavidsonIIS
        Is impersonating: True
        Stack trace:    at System.String.Substring(Int32 startIndex, Int32 length)
       at System.Web.Compilation.DiskBuildResultCache.MarkAssemblyAndRelatedFilesForDeletion(String
    assemblyName)
       at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String
    cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart)
       at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath
    virtualPath)
       at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath
    virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean
    ensureIsUpToDate)
       at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext
    context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
       at System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(VirtualPath
    virtualPath, HttpContext context, Boolean allowCrossApp, Boolean
    throwIfNotFound)
       at System.Web.Compilation.BuildManager.GetCompiledType(VirtualPath
    virtualPath)
       at System.Web.Script.Services.WebServiceData.GetWebServiceData(HttpContext
    context, String virtualPath, Boolean failIfNoData, Boolean pageMethods, Boolean inlineScript)
       at System.Web.Script.Services.WebServiceClientProxyGenerator.GetClientProxyScript(HttpContext
    context)
       at System.Web.Script.Services.RestClientProxyHandler.ProcessRequest(HttpContext
    context)
       at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
       at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
    
  • Matthew Kirschner 323 posts 611 karma points
    Feb 16, 2015 @ 16:13
    Matthew Kirschner
    0

    I think this issue may have been caused by leaving distributed calls in my dev environment. For now, I have commented out the servers in my umbracoSettings.config until I am ready to publish.

    This brings up a bit of a dev nuisance. Since I am using a Website Project in Visual Studio, I am unable to create transformations for any config file other than web.config (and even this has some issues). It would be nice if I could add distributed calls to my web.config as opposed to umbracoSettings. That way I only have to keep track of differences between dev, staging, and release in a single file.

  • Brad 69 posts 97 karma points
    Feb 16, 2015 @ 22:25
    Brad
    0

    Hi Matthew,

    You can move umbracoSettings.config to web.config. It would go where the configSource line is:

    <umbracoConfiguration>
         <settings configSource="config\umbracoSettings.config" />
    </umbracoConfiguration>
    

    So instead settings would contain everything in umbracoSettings.

  • Matthew Kirschner 323 posts 611 karma points
    Feb 16, 2015 @ 23:02
    Matthew Kirschner
    0

    Thanks, Brad. I'll play around with this next time I deploy to staging.

    Do you know if config transforms would work with this setup? For example, I want to add in my web.Debug.Config:

    <umbracoConfiguration>
      <settings configSource="config\umbracoSettings-debug.config" xdt:Transform="Replace" />
    </umbracoConfiguration>
    

    Obviously I would have to add "umbracoSettings-debug.config" to my config folder and replicate to this any changes I make to the regular config, but that would still be an improvement over my current workflow.

Please Sign in or register to post replies

Write your reply to:

Draft