Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Nick 6 posts 84 karma points
    Sep 05, 2014 @ 11:16
    Nick
    0

    Load balancing distributed call not updating cache (7.1.6)

    Hopefully someone can help with this?  I am having problems setting up our load balanced environment to work with umbraco (7.1.6).  I have followed the documentation but still no luck.  My setup is as follows....

    • 1x sql server (shared by both web servers)
    • 2x web servers
    • Web servers use their own local copy of the code which is synchronised but excludes the App_Data folder (i previously excluded specific files/folders but that didn't work either).
    • The web servers share an iis config, but have bindings specific for each machine to enable them to communicate internally with each other.
    • Distributed calls have been setup in the umbracoSettings.config file, this includes setting enable to true, adding the two servers using host names which are added to the hosts file and the IIS bindings, the master server has its server name property set on the server tag in the umbracoSettings.config file.
    • I am using the umbraco backoffice using server 1 only.

    I have monitored each server when i make a change using wireshark and I can see the call come in to server 2 with the correct user/page details which gives a response of 200 ok (with an empty BulkRefreshResponse node) but the cache files/folders are not changed. 

    The only unusual thing I can see in the web service call is a guidId node which is set to 00000000-0000-0000-0000-000000000000 and an intId node which is set to 0 but I am assuming that the latter is the userId?

    Looking at the logs...

    • There are no errors in the logs and I can see the log item "Submitting calls to distributed servers" on server 1. 
    • There are no items logged on server 2 at the same time as a page is published on server 1. 

    Server 2 can successfully generate its cache so if i delete the contents of the App_data folder on that server it does regenerate it but it just won't update when a page is published on server 1.

    What have I missed or is this a bug?

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    Sep 08, 2014 @ 07:55
    Shannon Deminick
    0

    Not sure, I've set this up so many times on my local environment without issue, the problem with LB is that everyone's environment is always different and there are so many moving parts.

    First, you should set the serverName on ALL servers listed in your servers config.

    Next, you can change your log4net settings to be 'Debug' to output more information, on server 2 with that setting you should at least then see the request coming in for the cacherefresher.asmx. For the time being until you iron out all the issues, might be best to set logging levels to Debug for both of your servers.

    Not all of the App_Data folder should be ignored, since you are doing file replication, here's the docs on what to ignore:

    http://our.umbraco.org/documentation/Installation/load-balancing#Nonreplicatedfiles

  • Zac 239 posts 541 karma points
    Sep 15, 2014 @ 19:01
    Zac
    0

    We're encountering the same issue and it sounds like we have a very similar setup.

    We have confirmed that upon publishing, server 1, calls umbraco/RestServices/ScheduledPublish/Index, then calls umbraco/webservices/cacheRefresher.asmx and upon publishing server 1, the cacheRefresher is called on server 2.  The cache files on server 1 are updated correctly, however, the cache files on server 2 (app_data/temp/umbraco.config and app_data/temp/examineindexes/...) are not updated as expected.

    Here's our setup 

    • 1 SQL Server db instance (SQL Server 2008 R2)
    • 2 web server instances (Windows Server 2008 R2)
    • Using file replication using DFS to sync the two web instances (umbraco.config is stored in the App_Data/TEMP folder.  App_Data/TEMP and App_Data/Logs are the only folders excluded from replication)
    • Load balancing with instance 1 designated as the back office server - host headers set appropriately
    • web.config - shared machine key
    • umbracoSettings.config - distributed calls are enabled, both servers are listed using internal DNS name and server name, i.e.ourServer1.internalip.com
    • Permissions set correctly on both servers (you can publish the cache fine individually, but distributed publishing is not working)
    • Log4Net is in debug mode
    Log file snippets (urls and ips are scrubbed)

    Server 1:
    IIS Logs

    2014-09-15 16:32:29 XXX.106 POST /umbraco/RestServices/ScheduledPublish/Index - 80 - XXX.106 - 200 0 0 156

     

    2014-09-15 16:32:59 XXX.106 POST /umbraco/webservices/cacheRefresher.asmx - 80 - XXX.106  - 200 0 0 15

    Umbraco Trace Logs 

    2014-09-15 11:32:58,949 [17] INFO  Umbraco.Core.Publishing.PublishingStrategy - [Thread 30] Content 'Distributed Test' with Id '1623' has been published.

    2014-09-15 11:32:59,027 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Performing distributed call for refresher Umbraco.Web.Cache.UnpublishedPageCacheRefresher, message type: RefreshById, servers: http://[internal1].com/umbraco/webservices/cacheRefresher.asmx;http://[internal2].com/umbraco/webservices/cacheRefresher.asmx, ids: 1623, json: 

    2014-09-15 11:32:59,027 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Invoking refresher Umbraco.Web.Cache.UnpublishedPageCacheRefresher on single server instance, message type RefreshById

    2014-09-15 11:32:59,214 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Performing distributed call for refresher Umbraco.Web.Cache.PageCacheRefresher, message type: RefreshById, servers: http://[internal1].com/umbraco/webservices/cacheRefresher.asmx;http://[internal2].com/umbraco/webservices/cacheRefresher.asmx, ids: 1623, json: 

    2014-09-15 11:32:59,214 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Invoking refresher Umbraco.Web.Cache.PageCacheRefresher on single server instance, message type RefreshById

    2014-09-15 11:32:59,230 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Invoking refresher Umbraco.Web.Cache.MacroCacheRefresher on single server instance, message type RefreshAll

    2014-09-15 11:32:59,542 [17] INFO  Umbraco.Web.BatchedServerMessenger - [Thread 30] Submitting calls to distributed servers

     

    Server 2: 
    IIS Logs
    2014-09-15 16:32:59 XXX.107 POST /umbraco/webservices/cacheRefresher.asmx 

    Umbraco Trace Log  
    No relevant logs - we find this to be very odd.  

     

    We're wondering if this error could be due to the Scheduled Publishing & Tasks update made for 7.1.5+ http://our.umbraco.org/documentation/Installation/load-balancing#UmbracoConfiguration - correct config for scheduled publishing & tasks), since it appears that the distributed publishing may now be handled by umbraco/RestServices/ScheduledPublish/Index?

    Any advise would be greatly appreciated.

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    Sep 16, 2014 @ 02:11
    Shannon Deminick
    1

    Hi , i think i found the reason for distributed calls not updating the cache in some cases from 7.1.5+, see this issue here with a work around:

    http://issues.umbraco.org/issue/U4-5491

  • Zac 239 posts 541 karma points
    Sep 16, 2014 @ 02:17
    Zac
    0

    Thank you Shannon.  This does sound like the issue we're encountering.  We tried to setup the app using the AppDomainAppId method and found out that our AppDomainAppId is the same for both web servers.

    We'll try this tomorrow and update you.

    Thanks

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    Sep 16, 2014 @ 02:21
    Shannon Deminick
    0

    Cool, just to confirm the 'appId' part in the config listed here: http://our.umbraco.org/documentation/Installation/load-balancing#Correctconfigforscheduledpublishing&tasks

    is ONLY for scheduled publishing, if you have 2+ sites being load balanced on the same server - which is generally only ever done for testing. You'd 99% of the time, use the serverName attribute.

    That said, we do use AppDomainAppId to filter out cache refresher calls so that we don't double cache refresh on the master. All the details are listed on that issue.

    Please let me know if the work around works for you (i think it should) and I'll update the LB docs. I'd rather not back port this fix and create new releases just for this one issue when there is a simple work around in place.

    Cheers!

  • Nick 6 posts 84 karma points
    Sep 16, 2014 @ 09:07
    Nick
    0

    Hi Shannon,

    Thanks for your help on this one.  It looks like this is the reason for our troubles.  As we have a shared config the sites HttpRuntime.AppDomainAppId is the same on both servers. Unfortunately this means that your workaround will not work for us but we look forward to the next release which by the look of your fix description should do the trick.

    Any ideas when this release will be out? Will it be 7.1.7 or 7.2.0?

    Cheers

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    Sep 16, 2014 @ 09:24
    Shannon Deminick
    0

    hrm, yes with shared config that won't work unless you remove the shared config to get it working - if you need it working now then i guess that's what you'll need to do.

    7.2 beta should be out in a week or two

  • Nick 6 posts 84 karma points
    Sep 16, 2014 @ 10:15
    Nick
    101

    Hello again, ok i have come up with a work around for those using a shared configuration in IIS, unfortunately i think the more servers you have in your cluster the more painful this workaround will be (we only have 2 so not able to test more)....

    1. duplicate your umbraco site in the shared config (you may need a duplicate per server?)
    2. in the original site set the bindings to be specific to the server 1 ip address
    3. in the duplicate site set the bindings to be specific to the server 2 ip address
    4. make sure that each site has its own binding for internal comunication, for me this was just setting up server1.mywebsite.local against the original site and setting up server2.mywebsite.local against the duplicate site these were then added to the hosts file and the umbracoSettings.config file distributedCall section as described in the docs.

    This means that when umbraco does its webservice call it will be calling a site with a different appId.

    I hope this helps.

     

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    Sep 17, 2014 @ 01:22
  • Zac 239 posts 541 karma points
    Sep 17, 2014 @ 03:26
    Zac
    0

    Thanks Shannon. 

    None of these will work for us since our environments have the iis meta data shared.  This prevents us from setting individual appdomainappid's.

    We're going to just create a custom 7.1.6 build incorporating your patch.

    Just to double check, this isn't already available,  correct? 

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    Sep 17, 2014 @ 07:57
  • Nick 6 posts 84 karma points
    Sep 17, 2014 @ 09:02
    Nick
    0

    Thanks for the aditional work around link Shannon.  This should work for users who do not have a shared configuration.

    @Zac, We too have the iis meta data shared so my work around should work for you but a custom build will certainly work too. :)

  • Zac 239 posts 541 karma points
    Sep 17, 2014 @ 18:13
    Zac
    2

    Thanks Shannon and Nick.  

    We have several environments, so we didn't feel comfortable with the IIS meta data workaround.

    We built a custom version of Umbraco v7.1.6 with this change (only dll changed is umbraco.dll).  It's working great for us.  We attached it to the issue here http://issues.umbraco.org/issue/U4-5491, so it will hopefully help others.

     

Please Sign in or register to post replies

Write your reply to:

Draft