Is anyone using umbraco examine in load balanced environment? If not does anyone know what would need to be done to get it to work in load balanced setup where we are not using dfs. Possibly webservice call from admin server to add published document to index on target server?
I have the following situation, one admin server behind firewall. 2 dumb servers other side of firewall. The admin server has in umbracosettings.config distrubuted setting set to true and has itself and 2 dumb server urls set. The dumb servers have umbraco_client and most of umbraco dir removed so you cannot edit on those servers only on admin server. Publish from admin also publishes to dumb servers via web services. The media files are being replicated using dfs. This setup all works.
Phase 2 of project we are looking to use Examine for search. So i guess i need to install examine on all 3 servers. On admin server on publish the index will be created /updated. Will the indexes on the dumb servers also be updated? or will i need to tap into actionhandler and then call bespoke webservice on dumb servers that will call something AddNodeToIndex(int Nodeid) then that way the indexes on those servers will also be upto date.
Ok, I think this can be done. The reason why Examine won't work out of the box in a replicated load balanced environment is because it needs one central index. In many cases replication mean replicate everything and this won't work with Examine. In your case however, this will work because you have 1 central admin server.
Indexes are only created when content/media is saved or published in the admin area so if you have only one administration server than this is fine.
However, your issue is going to be searching. You will need to actually write your own search webservice which will need to be hosted on this single admin server, then each of your servers will need to call this web service to actually do your searching. Shouldn't be very hard to create though. Your other option is to actually replicate the index to the other servers. Since I'm not sure how nicely this will work with Lucene until it's tested, i'll recommend the web service (though this does mean you have single point of failure). On the other hand, if copying indexes from the main server to other servers works without a prob, then thats definitely the way to go and all you need to do is install Examine on each server.
I know that there are issues with replicating indexes, have had issues in past using verity and am certain there will be issues with lucene as files are locked and you may end up trying to copy while an add to index is taking place. As you say "Indexes are only created when content/media is saved or published in the admin area" so admin server is fine. So for the dummy servers would either of these work:
1. On admin server i also have action handlers that call webservices on dummy servers to add to index on dummy server. so each one will have its own index which i guess gets created in same place so search code will call its own local index?
2. Create network share just for index folder and search on that.
With one i don't have single point of failure with 2 i do.
We normally run our load balanced environments either off of a SAN or a clustered file server running off a SAN so there isn't a single point of failure and doesn't require replication. If you don't have redundancy on your central file share then I'd recommend not doing it (as your no better off in regards to high availability).
Replication is awesome for HA, but needs some work in setting up to work nicely.
I'd suggest option 1 just so you don't have a single point of failure unless you have access to a clustered file share/SAN. It wouldn't be too hard to custom make an index replicator between your servers. Examine has a ton of events that fire during the indexing operation which you can easily attach to in order to call your web services on your other servers. Your web service would then just add a new index item to your dummy servers using the Examine API. Then as you've said, your dummy servers will just use their local index for searching.
Examine also optimizes the index on application startup and on every 100 index commits. So by using Option 1 and having Examine installed on each machine this will automaticaly be taken care of. Also, option 1 will allow you to change your master server to a different one quite easily.
I have a follow up question to one of your replies, Shannon :)
In order for Examine to work in a load balanced setup with a SAN where the Index is produced on the admin server (with umbraco client) and the "frontend server" (dumb server as Ismail described it). Is it just a matter of setting async=false so an index isn't produced by the dumb server as well? or is there another setting, which indicates that an instance should index? Since we are aiming for a central index hosted on a SAN I only want the index produced once = by the admin server.
No, async should never be set to false unless you are doing testing, you WILL run into problems.
If you are running a load balanced environment in a SAN (centrally shared by IIS websites) and not doing replication, then Examine will work out of the box without any work required. If you are running a replicated scenario, then you will have to write the replication handlers (though we'll hopefully be including these with the next version of Examine).
The way this works with a SAN is by checking for other hosts in the cluster which is done by creating some lock files in the ExamineIndex/ folders (which you will see). It then magically determines something called the Indexer Executive which is the single host that will do the indexing. Currently there is no control over which machine does the indexing in the cluster as it depends on up time between the machines. However, this shouldn't really make any difference since all hosts in the cluster will write to the index queue and the indexer executive is the one that picks up the queue.
Apologies for posting to such an old thread; I'll start another if that's more appropriate.
I have a scenario that's very similar to Ismail's, and I'm wondering if it could be causing problems.
We have a 3 server configuration, linked together using Umbraco's distributed calls. The filesystem is replicated using DFSR, with the notable exclusion of the ClientDependency and ExamineIndexes folders. Each node has its own index, therefore, and all are using async mode. One of the nodes is the nominated 'master'.
We have an index used by the website search functionality, as well as the internal content/member search indices.
What I gather from the above is that the distributed calls have no effect on the secondary nodes' indexing, meaning their indexes will become outdated over time. The only way to make this work is to have another set of distributed calls telling the Examine index to update itself across the other nodes.
Or am I just mixed up, and the cache refresh caused by the distrubuted call will cause the Examine index to be updated? A cursory examination of the source indicates that (for example) the AfterUpdateDocumentCache will fire - one of the events that Examine listens to. This would mean that as long as the index isn't replicated by the DFS service, it will all work.
The reason for the above is that we have had problems, similar to those described in another thread, and I'd like to try and rule out whether our replication strategy is the true source of the issue.
Umbraco examine in load balanced environment
Hello,
Is anyone using umbraco examine in load balanced environment? If not does anyone know what would need to be done to get it to work in load balanced setup where we are not using dfs. Possibly webservice call from admin server to add published document to index on target server?
Regards
Ismail
In answer to my question examine does it out of the box. woohoo!
To clarify Umbraco Examine will work in load balanced if you set it to run in async mode. This is a property on the indexer and is defaulted to true.
Shannon,
I have the following situation, one admin server behind firewall. 2 dumb servers other side of firewall. The admin server has in umbracosettings.config distrubuted setting set to true and has itself and 2 dumb server urls set. The dumb servers have umbraco_client and most of umbraco dir removed so you cannot edit on those servers only on admin server. Publish from admin also publishes to dumb servers via web services. The media files are being replicated using dfs. This setup all works.
Phase 2 of project we are looking to use Examine for search. So i guess i need to install examine on all 3 servers. On admin server on publish the index will be created /updated. Will the indexes on the dumb servers also be updated? or will i need to tap into actionhandler and then call bespoke webservice on dumb servers that will call something AddNodeToIndex(int Nodeid) then that way the indexes on those servers will also be upto date.
Regards
Ismail
Ok, I think this can be done. The reason why Examine won't work out of the box in a replicated load balanced environment is because it needs one central index. In many cases replication mean replicate everything and this won't work with Examine. In your case however, this will work because you have 1 central admin server.
Indexes are only created when content/media is saved or published in the admin area so if you have only one administration server than this is fine.
However, your issue is going to be searching. You will need to actually write your own search webservice which will need to be hosted on this single admin server, then each of your servers will need to call this web service to actually do your searching. Shouldn't be very hard to create though. Your other option is to actually replicate the index to the other servers. Since I'm not sure how nicely this will work with Lucene until it's tested, i'll recommend the web service (though this does mean you have single point of failure). On the other hand, if copying indexes from the main server to other servers works without a prob, then thats definitely the way to go and all you need to do is install Examine on each server.
Hope this helps!
Shannon,
I know that there are issues with replicating indexes, have had issues in past using verity and am certain there will be issues with lucene as files are locked and you may end up trying to copy while an add to index is taking place. As you say "Indexes are only created when content/media is saved or published in the admin area" so admin server is fine. So for the dummy servers would either of these work:
1. On admin server i also have action handlers that call webservices on dummy servers to add to index on dummy server. so each one will have its own index which i guess gets created in same place so search code will call its own local index?
2. Create network share just for index folder and search on that.
With one i don't have single point of failure with 2 i do.
Regards
Ismail
We normally run our load balanced environments either off of a SAN or a clustered file server running off a SAN so there isn't a single point of failure and doesn't require replication. If you don't have redundancy on your central file share then I'd recommend not doing it (as your no better off in regards to high availability). Replication is awesome for HA, but needs some work in setting up to work nicely.
I'd suggest option 1 just so you don't have a single point of failure unless you have access to a clustered file share/SAN. It wouldn't be too hard to custom make an index replicator between your servers. Examine has a ton of events that fire during the indexing operation which you can easily attach to in order to call your web services on your other servers. Your web service would then just add a new index item to your dummy servers using the Examine API. Then as you've said, your dummy servers will just use their local index for searching.
Examine also optimizes the index on application startup and on every 100 index commits. So by using Option 1 and having Examine installed on each machine this will automaticaly be taken care of. Also, option 1 will allow you to change your master server to a different one quite easily.
Cheers, Shannon
Shannon,
Awesome many thanks for the prompt reply. You planning on doing examine session at cg10 reckon you will have alot of takers me for one :-}
Regards
Ismail
Think Niels gave the go ahead for an Examine session, but still haven't heard any details yet... hopefully!
Shannon,
Awesome, over the year, 2 things that have really got me excited are examine and mimecloud cant wait to see and hear more about them at codegarden10.
Regards
Ismail
I have a follow up question to one of your replies, Shannon :)
In order for Examine to work in a load balanced setup with a SAN where the Index is produced on the admin server (with umbraco client) and the "frontend server" (dumb server as Ismail described it). Is it just a matter of setting async=false so an index isn't produced by the dumb server as well? or is there another setting, which indicates that an instance should index? Since we are aiming for a central index hosted on a SAN I only want the index produced once = by the admin server.
Thanks,
Morten
No, async should never be set to false unless you are doing testing, you WILL run into problems.
If you are running a load balanced environment in a SAN (centrally shared by IIS websites) and not doing replication, then Examine will work out of the box without any work required. If you are running a replicated scenario, then you will have to write the replication handlers (though we'll hopefully be including these with the next version of Examine).
The way this works with a SAN is by checking for other hosts in the cluster which is done by creating some lock files in the ExamineIndex/ folders (which you will see). It then magically determines something called the Indexer Executive which is the single host that will do the indexing. Currently there is no control over which machine does the indexing in the cluster as it depends on up time between the machines. However, this shouldn't really make any difference since all hosts in the cluster will write to the index queue and the indexer executive is the one that picks up the queue.
Apologies for posting to such an old thread; I'll start another if that's more appropriate.
I have a scenario that's very similar to Ismail's, and I'm wondering if it could be causing problems.
We have a 3 server configuration, linked together using Umbraco's distributed calls. The filesystem is replicated using DFSR, with the notable exclusion of the ClientDependency and ExamineIndexes folders. Each node has its own index, therefore, and all are using async mode. One of the nodes is the nominated 'master'.
We have an index used by the website search functionality, as well as the internal content/member search indices.
What I gather from the above is that the distributed calls have no effect on the secondary nodes' indexing, meaning their indexes will become outdated over time. The only way to make this work is to have another set of distributed calls telling the Examine index to update itself across the other nodes.
Or am I just mixed up, and the cache refresh caused by the distrubuted call will cause the Examine index to be updated? A cursory examination of the source indicates that (for example) the AfterUpdateDocumentCache will fire - one of the events that Examine listens to. This would mean that as long as the index isn't replicated by the DFS service, it will all work.
The reason for the above is that we have had problems, similar to those described in another thread, and I'd like to try and rule out whether our replication strategy is the true source of the issue.
is working on a reply...