I've been getting very frustasted trying to get Courier 2 to work on our upgraded Umbraco installation. We've gone through the pain of upgrading from 4.0.4.2 to 4.7.1.1. This has been a considerable amount of work, but all seems relatively stable....until I tried Courier 2.
First I loaded up v2.5.1 and tried to move a node from our staging to live environment. Items seem to take foreveer to package and then fail to allow me to transfer, the window just appears empty.
Then I tried the same in 2.5.3 but this time the window just locks up with a please wait. But the packaging does seem to happen in the background though.
My second approach was to create a Revision package in the main courier window. I selected no dependencies and selected all of our documents and associated files and clicked on "package selected". The tasks commences and proceeds to suck up 100% of the CPU time and takes forever to complete. Transfer from one node to the next is trivial in comparison. Unpackaging and installing on the live box is equally painful.
I also encountered a ream of errors during packaging which resulted in me either having to delete unpublished content or re-publshing content before the package would complete successfully. What annoyed me all the more was CPU utilization would stay at 100% if a package failed and I had no choice but to recycle the app-pool. Thats a total application fail!
This package in all fairness has been the shoddiest I've seen from the Umbraco team and I'm seriously un-impressed! Has too much time been focussed on Umbraco v5?
There simply not been enough testing done in scenarios with large and complicated sites. I also believe there's been little work done to ensure the packaging process is quick and efficient, the high CPU utilization, demonstrates this.
So what on earth do I do now? WIll v1.3 work in Umbraco 4.7?
Thanks for sharing your disappointments with Courier, even though its sad to hear that your can't get it to work, I appreciate you taking the time to tell us.
First of all, there has been focus on testing and pushing this with complex sites, but its a hard problem to solve so it just works for everybody, the usecases are simply too varied to find a generic deployment model that will fit any umbraco site out there.
The high cpu usage demonstates that this is actually a really really complex thing to do, sorting out all the different relations between all your items, and ensuring that they get loaded into a correct grid is very cpu intensive.
With 2.5.3 we did our first perfomance focused maintance release, we wil continue doing these, based on the feedback we're getting (like this)
If the package failed and the cpu kept at 100%, its because it's only the UI layer that failed, the dedicated thread that handles the packaging keeps on going untill its done.
There is a number of things you can do today, and a couple of things we can improve for the future releases:
1. If you need to move an entire site to a new location, don't use courier, just copy files and DB
2. Deploy in smaller chunks, this will keep the level of complexity and size down
3. Deploy using the Extraction Console Application (on nightly.umbraco.org) this will move the packaging and queuig away from the webserver and to a dedicated client, and make it more rubust, you can also do extraction from this client. Everythign is open-sourced, so you can tailor it to fit into your custom deployment needs.
For future release we can:
- If possible use a copy of your site as a test-case, we do this already with a couple of bigger, complex sites shared with us from Courier users
- look into your server logs and client-side browser error logs to determine what we can do better to avoid exceptions and unresponsive UIs
- We are working on a more fluent deployment process, where courier checks against a source during packaging, that means much lower deployment time and less data to transfer
Thanks for the quick feedback, i'll the checkout ECApp.
So why is CPU still so high when i've NOT selected any dependencies then? It's still hits the CPU really hard and still takes forever to pack. Surely you can go for a quicker process if dependencies arent selected by the user?
Can you confirm whether Courier 1.3 works in Umbraco 4.7?
So you package "selected items only" ? that should be much faster and shouldn't hit the CPU so hard as it's just fetching data and serializing it to disk
To be honest I do not know if 1.3 still works with 4.7, so can't confirm that
Our sites contains approx. 1500 documents and < 100 of each of the other datatypes/templates/scripts etc etc. I've selected them with no dependencies and clicked on "package selected". The process takes about 30 minutes, with the CPU at 100% most of the time! Transfer takes seconds, install on live node takes about another 30 minutes! So it's the packaging/unpackaging process at fault here.
1500? well that shouldn't be a problem to do, would it be possible to get a copy of your site and database (we will handle it confidently ofcourse) so we can run testing on your setup, a 30 min packaging time is not the the norm
Well umm, yea I didnt think that 1500 docs would be much of a problem either! In fact I've left out the media items because I figured that would be asking too much as we have some pretty large video files. But it still chugs away and upsets the CPU at the same time!
I'll speak to our IT Director to see if we can send anything to you.
We've just been contacted by our internal IT team over concerns of CPU utilization and they've offerred to throw more resources at it. (it's a virtual env).
It's been suggested we can scale as far as dual core/dual processors. So my question to you is, do you have any hardware recommendations that might improve things?
Our live box by the way, is not virtualized and is a quad core Xeon with 8GB RAM.
We upped the CPU's on our virtual server, solved the CPU utilization problem, but didnt make packaging go any quicker unfortunately.
Now i'm just trying to create the simplest of packages and i'm getting rediculous errors like this one...
Umbraco.Courier.Core.Exceptions.RetrieveException: Error retrieving item [23b0efc9-2a58-4354-8a65-3310012afad9] from provider [d8e6ad83-e73a-11df-9492-0800200c9a66]. ---> System.NullReferenceException: Object reference not set to an instance of an object. at Umbraco.Courier.Persistence.V4.NHibernate.Persisters.DocumentItem.RetrieveItem[T](ItemIdentifier itemId) in c:\Program Files (x86)\teamcity\buildAgent\work\3484142eca7e14a2\Umbraco.Courier.Persistence.NHibernate\Persisters\Document.cs:line 111 at Umbraco.Courier.Core.ItemCrudProvider.RetrieveItem[T](ItemIdentifier itemId) --- End of inner exception stack trace --- at Umbraco.Courier.Core.ItemCrudProvider.RetrieveItem[T](ItemIdentifier itemId) at Umbraco.Courier.ItemProviders.DocumentItemProvider.HandlePack(ItemIdentifier id) in c:\Program Files (x86)\teamcity\buildAgent\work\3484142eca7e14a2\Umbraco.Courier.Providers\ItemProviders\ContentItemProvider.cs:line 79 at Umbraco.Courier.Core.ItemProvider.Package(ItemIdentifier id) at Umbraco.Courier.Core.RevisionItem.Match() at Umbraco.Courier.Core.Revision.MatchWithServer() at Umbraco.Courier.Core.ExtractionManager.MatchRevisonWithServer() at Umbraco.Courier.UI.Pages.deployRevision.Page_Load(Object sender, EventArgs e) at System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) at System.EventHandler.Invoke(Object sender, EventArgs e) at umbraco.BasePages.BasePage.OnLoad(EventArgs e) at Umbraco.Courier.UI.Pages.CourierBasePage.OnLoad(EventArgs e) at Umbraco.Courier.UI.Pages.CourierBaseLicensedPage.OnLoad(EventArgs e) at System.Web.UI.Control.LoadRecursive() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object. at Umbraco.Courier.Persistence.V4.NHibernate.Persisters.DocumentItem.RetrieveItem[T](ItemIdentifier itemId) in c:\Program Files (x86)\teamcity\buildAgent\work\3484142eca7e14a2\Umbraco.Courier.Persistence.NHibernate\Persisters\Document.cs:line 111 at Umbraco.Courier.Core.ItemCrudProvider.RetrieveItem[T](ItemIdentifier itemId)
Some please help, If I cant get Courier 2 working, my Umbraco upgrade efforts may well be in vain!
Hi martin
I'm starting a new release cycle tomorrow and will include your issues in that, might email you with some additional questions, hope that is ok
Thanks for the feedback, you've been most helpful.
I've continued to battle with Courier but it's been a bit of an unpredictable beast!
Right click transfers initially werent working at all, i'd get the two tabs which are supposed to show whats been packaged before transfer but they'd be completely empty and no changes would show up on the live version. But in the Courier section a package would work if I kept it simple. From yesterday it's now the reverse, right click transfers now work (but still the tabs are empty) and packages dont work. It's all very odd!
Riiight! I installed 2.5.4 and had an ADO connection issue, which I found in another forum to be an accidental slip up on your part. So I downloaded and installed 2.5.4.1 this morning. This version seems more stable but there are still some fundamental problems, so there they are in detail...
1. Right click deploy. Seems to work reliably but still empty tabs (see image)
2. Direct deploy hangs (see image)
3. Packaging wont install on target server. I've checked that propertydata is packaging correctly on source and target servers by looking at the .courier files and all looks ok. But install refuses to work unless I recycle the app pool on target server. After a recycle package installs as expected.
I do appreciate that a product of this kind is an insanely complicated implementation, but I cant help but feel its still some way off being ready for customers!
Hmm, will investigate each one, Could you enable logging in Courier. Open the courier.config and have debugmode enabled
<debugMode>true</debugMode>
(remember to reset app pool after editing courier.config, and edit the config on both instances)
That will produce log files in /app_data/courier/logs including debug, error and timer info, during packaging and extraction could you forward those to me at pph at umbraco.dk
Martin, I share your concerns. I've been trying to sell Umbraco internally at my company, and Courier is central to my sales effort :^) Thus far, I haven't had any problems on the target getting documents, document types, templates, and relationships (apparently fixed in 2.5.4.1), even without recycling the app pool, I still am not able to get stylesheets and images packaged on the originating server. Apologies in advance for cross-posting.
High CPU utilization and other Courier 2 problems
I've been getting very frustasted trying to get Courier 2 to work on our upgraded Umbraco installation. We've gone through the pain of upgrading from 4.0.4.2 to 4.7.1.1. This has been a considerable amount of work, but all seems relatively stable....until I tried Courier 2.
First I loaded up v2.5.1 and tried to move a node from our staging to live environment. Items seem to take foreveer to package and then fail to allow me to transfer, the window just appears empty.
Then I tried the same in 2.5.3 but this time the window just locks up with a please wait. But the packaging does seem to happen in the background though.
My second approach was to create a Revision package in the main courier window. I selected no dependencies and selected all of our documents and associated files and clicked on "package selected". The tasks commences and proceeds to suck up 100% of the CPU time and takes forever to complete. Transfer from one node to the next is trivial in comparison. Unpackaging and installing on the live box is equally painful.
I also encountered a ream of errors during packaging which resulted in me either having to delete unpublished content or re-publshing content before the package would complete successfully. What annoyed me all the more was CPU utilization would stay at 100% if a package failed and I had no choice but to recycle the app-pool. Thats a total application fail!
This package in all fairness has been the shoddiest I've seen from the Umbraco team and I'm seriously un-impressed! Has too much time been focussed on Umbraco v5?
There simply not been enough testing done in scenarios with large and complicated sites. I also believe there's been little work done to ensure the packaging process is quick and efficient, the high CPU utilization, demonstrates this.
So what on earth do I do now? WIll v1.3 work in Umbraco 4.7?
Very dissappointed, Umbraco user!
Hi Martin
Thanks for sharing your disappointments with Courier, even though its sad to hear that your can't get it to work, I appreciate you taking the time to tell us.
First of all, there has been focus on testing and pushing this with complex sites, but its a hard problem to solve so it just works for everybody, the usecases are simply too varied to find a generic deployment model that will fit any umbraco site out there.
The high cpu usage demonstates that this is actually a really really complex thing to do, sorting out all the different relations between all your items, and ensuring that they get loaded into a correct grid is very cpu intensive.
With 2.5.3 we did our first perfomance focused maintance release, we wil continue doing these, based on the feedback we're getting (like this)
If the package failed and the cpu kept at 100%, its because it's only the UI layer that failed, the dedicated thread that handles the packaging keeps on going untill its done.
There is a number of things you can do today, and a couple of things we can improve for the future releases:
1. If you need to move an entire site to a new location, don't use courier, just copy files and DB
2. Deploy in smaller chunks, this will keep the level of complexity and size down
3. Deploy using the Extraction Console Application (on nightly.umbraco.org) this will move the packaging and queuig away from the webserver and to a dedicated client, and make it more rubust, you can also do extraction from this client. Everythign is open-sourced, so you can tailor it to fit into your custom deployment needs.
For future release we can:
- If possible use a copy of your site as a test-case, we do this already with a couple of bigger, complex sites shared with us from Courier users
- look into your server logs and client-side browser error logs to determine what we can do better to avoid exceptions and unresponsive UIs
- We are working on a more fluent deployment process, where courier checks against a source during packaging, that means much lower deployment time and less data to transfer
/per
Hi Per
Thanks for the quick feedback, i'll the checkout ECApp.
So why is CPU still so high when i've NOT selected any dependencies then? It's still hits the CPU really hard and still takes forever to pack. Surely you can go for a quicker process if dependencies arent selected by the user?
Can you confirm whether Courier 1.3 works in Umbraco 4.7?
Regards
Martin
So you package "selected items only" ? that should be much faster and shouldn't hit the CPU so hard as it's just fetching data and serializing it to disk
To be honest I do not know if 1.3 still works with 4.7, so can't confirm that
/Per
Our sites contains approx. 1500 documents and < 100 of each of the other datatypes/templates/scripts etc etc. I've selected them with no dependencies and clicked on "package selected". The process takes about 30 minutes, with the CPU at 100% most of the time! Transfer takes seconds, install on live node takes about another 30 minutes! So it's the packaging/unpackaging process at fault here.
Martin
1500? well that shouldn't be a problem to do, would it be possible to get a copy of your site and database (we will handle it confidently ofcourse) so we can run testing on your setup, a 30 min packaging time is not the the norm
/per
Well umm, yea I didnt think that 1500 docs would be much of a problem either! In fact I've left out the media items because I figured that would be asking too much as we have some pretty large video files. But it still chugs away and upsets the CPU at the same time!
I'll speak to our IT Director to see if we can send anything to you.
I'll let you know
Regards
Martin
thanks, you can contact me directly on per at umbraco.dk
We've just been contacted by our internal IT team over concerns of CPU utilization and they've offerred to throw more resources at it. (it's a virtual env).
It's been suggested we can scale as far as dual core/dual processors. So my question to you is, do you have any hardware recommendations that might improve things?
Our live box by the way, is not virtualized and is a quad core Xeon with 8GB RAM.
To Per/Anyone
We upped the CPU's on our virtual server, solved the CPU utilization problem, but didnt make packaging go any quicker unfortunately.
Now i'm just trying to create the simplest of packages and i'm getting rediculous errors like this one...
Umbraco.Courier.Core.Exceptions.RetrieveException: Error retrieving item [23b0efc9-2a58-4354-8a65-3310012afad9] from provider [d8e6ad83-e73a-11df-9492-0800200c9a66]. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at Umbraco.Courier.Persistence.V4.NHibernate.Persisters.DocumentItem.RetrieveItem[T](ItemIdentifier itemId) in c:\Program Files (x86)\teamcity\buildAgent\work\3484142eca7e14a2\Umbraco.Courier.Persistence.NHibernate\Persisters\Document.cs:line 111
at Umbraco.Courier.Core.ItemCrudProvider.RetrieveItem[T](ItemIdentifier itemId)
--- End of inner exception stack trace ---
at Umbraco.Courier.Core.ItemCrudProvider.RetrieveItem[T](ItemIdentifier itemId)
at Umbraco.Courier.ItemProviders.DocumentItemProvider.HandlePack(ItemIdentifier id) in c:\Program Files (x86)\teamcity\buildAgent\work\3484142eca7e14a2\Umbraco.Courier.Providers\ItemProviders\ContentItemProvider.cs:line 79
at Umbraco.Courier.Core.ItemProvider.Package(ItemIdentifier id)
at Umbraco.Courier.Core.RevisionItem.Match()
at Umbraco.Courier.Core.Revision.MatchWithServer()
at Umbraco.Courier.Core.ExtractionManager.MatchRevisonWithServer()
at Umbraco.Courier.UI.Pages.deployRevision.Page_Load(Object sender, EventArgs e)
at System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e)
at System.EventHandler.Invoke(Object sender, EventArgs e)
at umbraco.BasePages.BasePage.OnLoad(EventArgs e)
at Umbraco.Courier.UI.Pages.CourierBasePage.OnLoad(EventArgs e)
at Umbraco.Courier.UI.Pages.CourierBaseLicensedPage.OnLoad(EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Umbraco.Courier.Persistence.V4.NHibernate.Persisters.DocumentItem.RetrieveItem[T](ItemIdentifier itemId) in c:\Program Files (x86)\teamcity\buildAgent\work\3484142eca7e14a2\Umbraco.Courier.Persistence.NHibernate\Persisters\Document.cs:line 111
at Umbraco.Courier.Core.ItemCrudProvider.RetrieveItem[T](ItemIdentifier itemId)
Some please help, If I cant get Courier 2 working, my Umbraco upgrade efforts may well be in vain!
Martin
Hi martin I'm starting a new release cycle tomorrow and will include your issues in that, might email you with some additional questions, hope that is ok
/per
Hi Per
Thanks for the feedback, you've been most helpful.
I've continued to battle with Courier but it's been a bit of an unpredictable beast!
Right click transfers initially werent working at all, i'd get the two tabs which are supposed to show whats been packaged before transfer but they'd be completely empty and no changes would show up on the live version. But in the Courier section a package would work if I kept it simple. From yesterday it's now the reverse, right click transfers now work (but still the tabs are empty) and packages dont work. It's all very odd!
I'll continue to wrestle with it today.
Martin.
Hi Martin
What browser are you using? , if you have a javascript debugger on that browser, could you check for errors?
/per
Hi Per
I'm using Chrome 16 and IE9
No exceptions to note but lots of warnings in Chrome, mostly attributed to deprecated code used in jquery libs.
Is their anything in particular you had in mind.
Martin
Okey, new maintance release is available here:
http://nightly.umbraco.org/UmbracoCourier/2.5.4/
It might not fix your issue, but it will atleast tell you what went wrong instead of showing you an empty screen
/Per
Hi Per
Riiight! I installed 2.5.4 and had an ADO connection issue, which I found in another forum to be an accidental slip up on your part. So I downloaded and installed 2.5.4.1 this morning. This version seems more stable but there are still some fundamental problems, so there they are in detail...
1. Right click deploy. Seems to work reliably but still empty tabs (see image)
2. Direct deploy hangs (see image)
3. Packaging wont install on target server. I've checked that propertydata is packaging correctly on source and target servers by looking at the .courier files and all looks ok. But install refuses to work unless I recycle the app pool on target server. After a recycle package installs as expected.
I do appreciate that a product of this kind is an insanely complicated implementation, but I cant help but feel its still some way off being ready for customers!
Hi Martin
Hmm, will investigate each one, Could you enable logging in Courier. Open the courier.config and have debugmode enabled
<debugMode>true</debugMode>
(remember to reset app pool after editing courier.config, and edit the config on both instances)
That will produce log files in /app_data/courier/logs including debug, error and timer info, during packaging and extraction could you forward those to me at pph at umbraco.dk
Martin, I share your concerns. I've been trying to sell Umbraco internally at my company, and Courier is central to my sales effort :^) Thus far, I haven't had any problems on the target getting documents, document types, templates, and relationships (apparently fixed in 2.5.4.1), even without recycling the app pool, I still am not able to get stylesheets and images packaged on the originating server. Apologies in advance for cross-posting.
-- Paul
is working on a reply...