Tweenday Two (Dec 27 2007)

("Tweenday" referring to the scant few work days between Xmas and New Year's holidays).

As we progress in our back-end scientific analysis we need to build many indexes on the science database (which vastly speed up queries). In fact, we need and hope to create 2 indexes a week for the next month or two. Seems easy, but each time you fire off such a build the science database locks up for up to 6 hours, during which there will be no assimilation and no splitting of new workunits. Well, we were planning to build another index today but with the frequent "high demand" due to our fast-return workunits the ready-to-send queue is pretty much at zero. So if we started such an index build y'all would get no work until it was done. We decided to postpone this until next week when hopefully we'll have a more user-friendly window of opportunity.

In the meantime, I've been trying to squeeze more juice out of our current servers. I'm kinda stumped as to why we are hitting this 60 MB/sec ceiling of workunit production/sending. I'm not finding any obvious I/O or network bottlenecks. However, while searching I decided to "fix" the server status page. I changed "results in progress" to "results out in the field" which is more accurate. This number never did include the results waiting for the redundant partners to return. So I added a "results returned/awaiting validation" row which also isn't exactly an accurate description either but is the shortest phrase I could think up at the time. Basically these are all the results that have been returned and have yet to enter the validation/assimilation/delete pipeline, after which it is "waiting for db purging." To use a term coined elsewhere, most of these results, if not all, are waiting for their "wingman" (should be "wingperson"). At this point if you add the results ready to send, out in the field, returned/awaiting validation, and awaiting db purging, you have an exact total of the current number of all results in the BOINC database. Thinking about this more, to get a slightly more accurate number of results waiting to reach redundancy before entering the back-end pipeline you take the "results returned/awaiting validation" and subtract 2 times the workunits awaiting validation and subtract 2 times the workunits awaiting assimilation. Whatever.. you get the basic idea. If I think of an easier/quicker way to describe all this I will.

Answering some posts from yesterday's thread:

Missing files like that prompt me to make an immediate fsck on the filesystem.

Why are the bits 'in' larger than the bits 'out'?

Thanks Matt for the update.

I trust that you & the team had a good Xmas.

Regards to everyone & a HAPPY NEW YEAR
____________

Maybe you could make a suggestion to the heavy contributors that they should set their queue's to a longer period for the period you need to build indexes and cause downtime.

Also, I thought I saw someone say something about a 60Mbit cap.

But I thought to remember you guys have Gigabit lnterlinks between all the servers and probably a Gigabit or higher backbone going almost all the way to the Internet router, that 60Mbit/s-7.5MB/s limit is really really weird.

Won't just an upgrade of NIC's on both ends of the fibre do the trick? Or is it that low quality fibre?

Let me get this right, you are paying for a gig but can only use a tenth of it? (And you are asking for donations?!)

Or is the campus is paying for 9/10 of it and you are paying for 1/10?

Or are they billing on bits transfered and not pipe width?

Is it possible to get a not campus controlled fibre? (Did they pull a backup?)

Maybe you could make a suggestion to the heavy contributors that they should set their queue's to a longer period for the period you need to build indexes and cause downtime.

I don't have exact stats here, but I believe most of our bandwidth is due to users of the "set it and forget it" variety - they never mess around with queues/caches. So if the "heavy" users did stock up it wouldn't help as much as you'd think.

Also, I thought I saw someone say something about a 60Mbit cap.

I mentioned how we seem to have a 60Mbit ceiling - that's not an enforced cap - that's due to internal disk/database/network I/O bottlenecks which are quite dynamic and always difficult to track down. In reality, we have a 1GBit connection to the world via Hurricane Electric, but alas this is constrained by a 100 Mbit fiber coming out of the lab down to campus - it will take some big $$$ to upgrade that, which may happen sooner or later (as it would not only benefit us).

But I thought to remember you guys have Gigabit lnterlinks between all the servers and probably a Gigabit or higher backbone going almost all the way to the Internet router, that 60Mbit/s-7.5MB/s limit is really really weird.

We have gigabit all over our server closet (more or less - some older servers are 100Mbit going into the 1Gbit switch). And yes, you are right - it is weird. Maybe I'll take a look at that switch now that you mention it...

- Matt

("Tweenday" referring to the scant few work days between Xmas and New Year's holidays).

As we progress in our back-end scientific analysis we need to build many indexes on the science database (which vastly speed up queries). In fact, we need and hope to create 2 indexes a week for the next month or two. Seems easy, but each time you fire off such a build the science database locks up for up to 6 hours, during which there will be no assimilation and no splitting of new workunits. Well, we were planning to build another index today but with the frequent "high demand" due to our fast-return workunits the ready-to-send queue is pretty much at zero. So if we started such an index build y'all would get no work until it was done. We decided to postpone this until next week when hopefully we'll have a more user-friendly window of opportunity.

In the meantime, I've been trying to squeeze more juice out of our current servers. I'm kinda stumped as to why we are hitting this 60 MB/sec ceiling of workunit production/sending. I'm not finding any obvious I/O or network bottlenecks. However, while searching I decided to "fix" the server status page. I changed "results in progress" to "results out in the field" which is more accurate. This number never did include the results waiting for the redundant partners to return. So I added a "results returned/awaiting validation" row which also isn't exactly an accurate description either but is the shortest phrase I could think up at the time. Basically these are all the results that have been returned and have yet to enter the validation/assimilation/delete pipeline, after which it is "waiting for db purging." To use a term coined elsewhere, most of these results, if not all, are waiting for their "wingman" (should be "wingperson"). At this point if you add the results ready to send, out in the field, returned/awaiting validation, and awaiting db purging, you have an exact total of the current number of all results in the BOINC database. Thinking about this more, to get a slightly more accurate number of results waiting to reach redundancy before entering the back-end pipeline you take the "results returned/awaiting validation" and subtract 2 times the workunits awaiting validation and subtract 2 times the workunits awaiting assimilation. Whatever.. you get the basic idea. If I think of an easier/quicker way to describe all this I will.

Answering some posts from yesterday's thread:

Missing files like that prompt me to make an immediate fsck on the filesystem.

Very true - except this is a filesystem on network attached storage. The filesystem is propietary and out of our control, therefore no fsck'ing, nor should there be a need for manual fsck'ing.

Why are the bits 'in' larger than the bits 'out'?

In regards to the cricket graphs, the in/out depends on your orientation. The bytes going into the router are coming from the lab, en route to the outside world. So this is "outbound" traffic going "into" the router. Vice versa for the inbound. Basically: green = workunit downloads, blue line = result uploads - though there is some low-level apache traffic noise mixed in there (web sites and schedulers).

- Matt

During these rather rocky times, has anyone thought of disabling the beta project(s) that share the same servers? I think astropulse uses the same hardware, so why not throttle that project down to free up the main project. I realize this may not solve the bandwidth issue, but it may help the processing issues.

Assuming that the shorties are valid wu's, (and some have validly questioned this assumption) then it is clear something is radically wrong with server architecture as it stands and some focused problem solving is needed. Doah! I guess everybody is gone/nearly gone for the long weekend.

Well today I'm going to the casino to chill out. I've had enough of slow servers at seti for a while.

During these rather rocky times, has anyone thought of disabling the beta project(s) that share the same servers? I think astropulse uses the same hardware, so why not throttle that project down to free up the main project. I realize this may not solve the bandwidth issue, but it may help the processing issues.

Let me get this right, you are paying for a gig but can only use a tenth of it? (And you are asking for donations?!)

Or is the campus is paying for 9/10 of it and you are paying for 1/10?

Or are they billing on bits transfered and not pipe width?

Is it possible to get a not campus controlled fibre? (Did they pull a backup?)

In the meantime, I've been trying to squeeze more juice out of our current servers. I'm kinda stumped as to why we are hitting this 60 MB/sec ceiling of workunit production/sending. I'm not finding any obvious I/O or network bottlenecks.

snipped....
I wonder if the priority mechanism is working on the S@H server or if the server is attempting too many simultaneous connections.