I apologize for all whose WU's were cancelled today in the process of stopping the old ones and re-sending the new fixed ones.
At the beginning I didn't realize the effect and extend of the cancellation (since it was not intended to work this way), but I see now that many hours of computation and credits were lost due to this.
It seems to have been a bug in a script that had not been seen before until now, so we will refrain from using it until it has been fixed.
I realize that the computation time is very important for everyone so we hope that such hiccups won't happen too often in the future.

Many thanks for your patience and calculations!

This morning I saw that a system had recovered from a problem and was sitting at the Windows log-on screen. When I logged on it became unresponsive, though after waiting about a minute (due to my reg settings) I got 2 app crashed error messages for SDOERR WU's:

I2HDQ_21R8-SDOERR_2HDQd-1-4-RND9506_3 4467647 18 May 2013 | 21:10:00 UTC 19 May 2013 | 11:13:06 UTC Error while computing 18,441.84 18,362.63 --- Long runs (8-12 hours on fastest card) v6.18 (cuda42)

I2HDQ_22R9-SDOERR_2HDQd-0-4-RND3283_1 4465782 18 May 2013 | 17:35:46 UTC 19 May 2013 | 11:36:31 UTC Aborted by user 33,710.79 33,611.30 --- Long runs (8-12 hours on fastest card) v6.18 (cuda42)

These 2 WU's were around 50% and 79% complete (9h). I tried Boinc exits and log -ns, cold restarts, but always got the same app crashes. Even tried a clean driver install.

2 Climate models also failed (only 87h lost this time).

When I suspended and enabled the GPUGrid WU's they showed a fixed progress (~50% and 79%), but the elapsed and remaining time kept ticking over.

No heartbeat from core client for 30 sec - exiting

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
Kernel not foundAssertion failed: a, file swanlibnv2.cpp, line 59

Whatever is wrong with these work units is bad. You would be doing the recipient of the re-sends a favor by server aborting them, even mid-run; it's better than a crash that kills the WU's and takes out other work!
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Hm Skgiven, unfortunately it is probably not related to the specific jobs. I cannot guarantee it obviously, but from the statistics I see that the jobs have a higher success rate than most others right now on GPUgrid. Also my modifications to Paola's jobs (which these jobs used to be) are essentially none except letting them run longer, meaning they have run before on GPUgrid without too grave errors.
I am really sorry for the damage done to the other projects too, it must be very bad when so much work is lost, but at this point it doesn't seem that anyone has encountered similar problems to warrant cancelling the rest. I will definitely keep an eye on it though and if more people report the same problem I might cancel them.

Stefan, I wasn't suggesting that you cancel the batch, just the individual WU's. It was just a grumpy suggestion. If you think these aren't WU specific problems then there is no point cancelling them.

I was concerned about the I2HDQ_21R8-SDOERR_2HDQd-1-4-RND9506 WU, as it has already produced 4 different errors on different systems (with varying operating systems, GPU types/generations and probably drivers). While 2 systems have a high error rate, two don't.


Stoneageman, it seems you ended up with resends for both of my failures!
http://www.gpugrid.net/workunit.php?wuid=4465782

I hope they fare better for you.
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Ah ok now I get it, sorry. I will keep the tab open and check if it fails again in which case it gets the boot. I don't mind much about canceling a single chain if it fails everywhere. Thanks for the heads up.

Completed and validated :) Yay!
http://www.gpugrid.net/workunit.php?wuid=4467647

and the other one,

http://www.gpugrid.net/workunit.php?wuid=4465782


____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help