We are getting close to launching an upgraded CWIS instance on a new server, but seem to be running into some performance problems. It appears that if a user (or maybe the system as a whole) has been active, then things are pretty speedy. But if you go away for a while, come back, and execute a search or reload a page, the system can take several minutes or more to finish building the page.
This is running on a brand new and pretty beefy RHEL 6 server, so I don't think it's a problem with the sizing of the server, at least directly.
At one point I caught this happening, logged into the shell, and was able to use "show full processlist" to see that the script appeared to be intermittently making queries against the database, but with substantial time between them.
I'm wondering if
A) anyone has seen anything like this before, and might have suggestions, or
B) if not, if anyone might have suggestions for narrowing down what is going on.
It's frustrating, because at times system performance appears perfectly fine, and the problem is intermittent enough that it is difficult to troubleshoot.
Thanks in advance for any help you can provide,
David Dwiggins
david [at] dwiggins -dot- net
One thing you may want to try is to go into the Task Queue page (available via Administration) and uncheck Automatic Task Execution Enabled on the upper right.
If the problem disappears after you do that, it may indicate that it stems from a background task (though by their very nature, background tasks are only supposed to run after the page is loaded). You can then re-check the ATEE box to restart background task execution and keep an eye on the Task Queue page to see if a very-long-running task pops up that might be the culprit.
Other than that, if that CWIS instance is the only web-based software running on the server, my guess for the most likely cause would be something with the server configuration itself, with it possibly having to repopulate a cache or swap some chunk of Apache or another part of the web stack back in from disk.
Thanks, Ed. I'm going to try this now, and will see if the cataloguer still sees the problem this week.
When I went to look, there were around 4800 items in this queue. Is that normal? Are they perhaps building up because the server isn't in production and thus has relatively few page loads?
Can you provide a bit more information on how this works? Is there an option to set these to run via a cron job or some other mechanism so that they are not linked to user page loads?
Thanks again,
David
4,800 items is a lot to have in the queue. Tasks run off of page loads, so if you want to clear the queue up when there's little in the way of user activity, you could set up a cron job that just does a load of the home page every few minutes. (We usually use wget.) To clear the queue ASAP and keep it clear, how often you have the cron job run should correspond to what you've got Maximum Execution Time set to in the system configuration.
Ok. I'm going to hold off for a few days to see if we can verify that turning off task execution solves the performance problem. But once that experiment is complete, I will set up a periodic cron job to load the homepage (and hopefully whittle away at the queue)
What are the functions that the background tasks enable? Is there any harm in simply leaving the background execution box unchecked?
-David
Background tasks include updating of the search database, among other things, so I would not suggest leaving task execution disabled. The option to temporarily disable it was added primarily for debugging purposes.