For the moment I sent out just 100 simulations as a quick test

Nice, impressive structures to simulate.

What specific aspects of the proteins will the simulation address?

I see tasks ready to send on the server status page :-)

OPM996 is classed as a Short Run but using 980ti seems it will take around 7 hours

Yet I have it on 2 other machines as a Long Run

Only completed 6% in 30 minutes at 61% GPU utilization

Server Status calls it Short Run but in BOINC under Application it's Long Run. Got 2 already running fine.

On my 980ti it is running as short run and will take over 7 hours or longer to complete at current rate

No I am only running one at a time and it has only completed 15.57% after 1hr 11min on a 980ti utilization 60%

Projected time is increasing and it is running under short run app

2K1aR7
2K1aR5

Not working from what BOINC estimates but from real time and it will take betwen 7 to 8hrs

But never mind HeyHo

On my GTX980Ti's, the former long runs were running about 20.000 seconds, yielding 255.000 points.

Today's 2dmgR0-SDOERR_opm996-0-1-RND3256_0 was running 39.000 seconds, yielding 247.050 points.

How come?

On my GTX980Ti's, the former long runs were running about 20.000 seconds, yielding 255.000 points.

Today's 2dmgR0-SDOERR_opm996-0-1-RND3256_0 was running 39.000 seconds, yielding 247.050 points.

How come?

Yeah, they're pretty long, my 980ti is looking like it'll take about 20 hours at 70-75% usage. Don't mind the long run, I'm glad to have work to crunch, though I do hope the credit is equivalently good.

... I'm glad to have work to crunch

I crunched a WU lasting 18.1 hours for 203.850 credits on my GTX970.

This credit ratio is lower then previous "Long run" WUs

Previously I completed "Long run WUs" in 12 hours for 255.000 credits

I crunched a WU lasting 18.1 hours for 203.850 credits on my GTX970.

This credit ratio is lower then previous "Long run" WUs

Previously I completed "Long run WUs" in 12 hours for 255.000 credits

OK, Thanks Stefan

OPM996 is classed as a Short Run but using 980ti seems it will take around 7 hours

Yet I have it on 2 other machines as a Long Run

Only completed 6% in 30 minutes at 61% GPU utilization

I'm computing the WU 2kogR2-SDOERR_opm996-0-1-RND7448 but after 24hrs it is still at 15% (750ti)... Please don't say me I have to abort it!

Some of them are pretty big. Had one task task take 20 hours. Long tasks usually finish around 6.50-7 hours for reference

[...]I have 2 WU's (2b5fR8 & 2b5fR3) running, one on each of my GTX970's.[...]
Will drop the CPU usage to ~60% and give it a spring clean before starting up to see if that improves things.
- update - that allowed the cards to run at slightly higher clocks (and normally) but it's now only looking like 31.5h. That might yet drop a few hours but these are still extra long tasks.
- update - still looking like about 31.5h (55% after 17h and 51% after ~16h).

Seems like I might have underestimated the real runtime. We use a script that caclulates the projected runtime on my 780s. So it seems it's a bit too optimistic on it's estimates :/ Longer WUs (projected time over 18 hours) though should give 4x credits.

I have noticed that the last batch of WUs are worth a lot less credit per time spent crunching, but I also wanted to report that we might have some bad WUs going out. I spent quite a lot of time on this workunit, and noticed that the only returned tasks for this WU is "Error while computing." The workunit in question: https://gpugrid.net/workunit.php?wuid=11593942

It is possible that my GPU is unstable, but considering that I was crunching WUs to completion on GPUgrid before, as well as asteroids/milkyway without error, I believe the chance that the WU that was sent out was corrupt or bad. I will adjust my overclock if need be, but I have been crunching for weeks at these clocks with no problems until now.

I have noticed that the last batch of WUs are worth a lot less credit per time spent crunching, but I also wanted to report that we might have some bad WUs going out. I spent quite a lot of time on this workunit, and noticed that the only returned tasks for this WU is "Error while computing." The workunit in question: https://gpugrid.net/workunit.php?wuid=11593942

It is possible that my GPU is unstable, but considering that I was crunching WUs to completion on GPUgrid before, as well as asteroids/milkyway without error, I believe the chance that the WU that was sent out was corrupt or bad. I will adjust my overclock if need be, but I have been crunching for weeks at these clocks with no problems until now.

I have noticed that the last batch of WUs are worth a lot less credit per time spent crunching, but I also wanted to report that we might have some bad WUs going out. I spent quite a lot of time on this workunit, and noticed that the only returned tasks for this WU is "Error while computing." The workunit in question: https://gpugrid.net/workunit.php?wuid=11593942

It is possible that my GPU is unstable, but considering that I was crunching WUs to completion on GPUgrid before, as well as asteroids/milkyway without error, I believe the chance that the WU that was sent out was corrupt or bad. I will adjust my overclock if need be, but I have been crunching for weeks at these clocks with no problems until now.

You've been crashing tasks for a couple weeks, including the older 'BestUmbrella_chalcone' ones. See below.
You need to lower your GPU overclock, if you want to be stable with GPUGrid tasks!
Downclock it until you never see "The simulation has become unstable."

https://gpugrid.net/results.php?hostid=319330

https://gpugrid.net/result.php?resultid=15096591
2b6oR6-SDOERR_opm996-0-1-RND0942_1
10 May 2016 | 7:15:45 UTC

https://gpugrid.net/result.php?resultid=15086372
28 Apr 2016 | 11:34:16 UTC
e45s20_e17s22p1f138-GERARD_CXCL12_BestUmbrella_chalcone3441-0-1-RND8729_0

https://gpugrid.net/result.php?resultid=15084884
27 Apr 2016 | 23:52:34 UTC
e44s17_e43s20p1f173-GERARD_CXCL12_BestUmbrella_chalcone2212-0-1-RND5557_0

https://gpugrid.net/result.php?resultid=15084715
26 Apr 2016 | 23:30:01 UTC
e43s16_e31s21p1f321-GERARD_CXCL12_BestUmbrella_chalcone4131-0-1-RND6256_0

https://gpugrid.net/result.php?resultid=15084712
26 Apr 2016 | 23:22:44 UTC
e43s13_e20s7p1f45-GERARD_CXCL12_BestUmbrella_chalcone4131-0-1-RND8139_0

https://gpugrid.net/result.php?resultid=15082560
25 Apr 2016 | 18:49:58 UTC
e42s11_e31s18p1f391-GERARD_CXCL12_BestUmbrella_chalcone2731-0-1-RND8654_0

I will agree with the credit being much lower but pretty sure they can't change what projects are worth after the fact, so I'm okay with it.

For future reference, I have a couple of suggestions:

For WUs running 18 hours +, there should be separate category : "super long runs". I believe this was mentioned in past posts.


The future WU application version should be made less CPU dependent. The WUs are getting longer, GPU s are getting faster, but CPU speed is stagnant. Something got to give, and with the Pascal cards coming out soon, you have put out a new version anyway. So, why not do both? My GPU usage on these latest SDOERR WUs is between 70% to 80%, compared to 85% to 95% for the GERARD BestUmbrella units. I don't think this is reinventing the wheel, just updating it. This does have to be done sooner or later. The Volta cards are coming out in only a few short years.

Current (Opm996 long) batch are 10 million step simulations with a large "nAtom" variation (30k <<>> 90k).
Per step compute time "nAtom" variation create runtime differences for the same GPU's.
Expect variable results due to OPM996 changeable atom amount.
The WU's atom amount shown upon task completion in <stderr> file.

Generally the number of atoms determine GPU usage: The more atoms a simulation has - the higher GPU usage.
So far prior OPM996 WU on my 970's shown a 10% (60~70%) GPU usage (63k atoms .vs. 88k) for a ~5hr completion time variation.
My current (2) opm996 tasks difference will be near ~7hr (19 & 26hr) total runtime (69% & 62%) core usage on 970's. The GPU with lower usage (fewer atoms) finishes a WU faster.

Completed OPM996 (long WU) show >90k atoms (>24hr runtime) being rewarded with (over 630,000 credits) while a 88478 atoms task is within the 336.000 range. (<24hr runtime) 63016 atom task around 294.000 credits. 52602 atoms = 236,700.

4n6hR2-SDOERR_opm996-0-1-RND7021_0 100.440 Natoms 16h 36m 46s (59.806s) 903.900 credits (GTX980Ti)
I see 85%~89%~90% GPU usage on my WinXPx64 hosts.
BTW there's no distinction of these workunits on the performance page, that makes me pretty sad.

This one looks like it's going to take 30hrs :) 44% after 13 hours.

https://www.gpugrid.net/result.php?resultid=15094957

though should give 4x credits.

For future reference, I have a couple of suggestions:

For WUs running 18 hours +, there should be separate category : "super long runs". I believe this was mentioned in past posts.

This was previously suggested, however this issue was just an underestimate of the runtimes by the researchers and they would normally increase the credit ratio awarded for extra-long tasks. The real issue would be another queue on the server and facilitating and maintaining that; the short queue is often empty never mind having an extra long queue.

The future WU application version should be made less CPU dependent.
Generally that's the case but ultimately this is a different type of research and it simply requires that some work be performed on the CPU (you can't do it on the GPU).

The WUs are getting longer, GPU s are getting faster, but CPU speed is stagnant. Something got to give, and with the Pascal cards coming out soon, you have put out a new version anyway.
Over the years WU's have remained about the same length overall. Occasionally there are extra-long tasks but such batches are rare.
GPU's are getting faster, more adapt, the number of shaders (cuda cores here) is increasing and CUDA development continues. The problem isn't just CPU frequency (though an AMD Athlon 64 X2 5000+ is going to struggle a bit), WDDM is an 11%+ bottleneck (increasing with GPU performance) and when the GPUGrid app needs the CPU to perform a calculation it's down to the PCIE bridge and perhaps to some extent not being multi-threaded on the CPU.
Note that it's the same ACEMD app (not updated since Jan) just a different batch of tasks (batches vary depending on the research type).

My GPU usage on these latest SDOERR WUs is between 70% to 80%, compared to 85% to 95% for the GERARD BestUmbrella units. I don't think this is reinventing the wheel, just updating it. This does have to be done sooner or later. The Volta cards are coming out in only a few short years.
I'm seeing ~75% GPU usage on my 970's too. On Linux and XP I would expect it to be around 90%. I've increased my Memory to 3505MHz to reduce the MCU (~41%) @1345MHz [power 85%], ~37% @1253MHz. I've watched the GPU load and it varies continuously with these WU's, as does the Power.

Don't worry about that Stefan it's a curve ball that you've thrown us that's all.

"I expect it to be over by the end of the week"

Hi SK,
At least the only thing that was wasted there was bandwidth because they errored immediately but the biggest thing (especially but not only in a WU drought) is the person who never returns so basically the WU is in limbo for 5 days before it gets resent and then it's possible it goes to another machine that holds it for another 5 days while good machines are sat waiting for work.

It's NOT good for users
It's NOT good for scientists
It's NOT good for this project

About time we got the broom out.

Thanks Stefan,

Finally just an example of a different scenario which is frustrating,

https://www.gpugrid.net/workunit.php?wuid=11591312

User exceeds 5 days so WU gets sent to my slowest machine which completes in less than 24hrs.

However original user then sends back result after 5 days pipping my machine to the post making my effort a waste of time.

This was of course because original user downloaded 2 WUs at a time on 750ti which struggles to complete one WU in 24hrs running 24/7

On my 980ti I recently aborted a unit that was 40% complete for exactly the same reason.

Actually there is a way to send WUs only to top users by increasing the job priority over 1000. I thought I had it at max priority but I just found out that it's only set to 'high' which is 600.

I guess most of these problems would have been avoided given a high enough priority.

I highlighted a best case scenario (where several systems were involved); the WU runs soon after receipt, fails immediately, gets reported quickly and can be sent out again. Even in this situation it went to 4 clients that failed to complete the WU. To send and receive replies from 4 systems took 7h.

My second point was that the WU never really had a chance on any of those systems. They weren't capable of running these tasks (not enough RAM, too slow, oldish driver [slower or more buggy], too hot, or just not powerful enough to return an extra long WU inside 5days).

I understand the runtime was underestimated

I only sent WUs that were projected to run under 24 hours on a 780

It's not only low end cards

https://www.gpugrid.net/workunit.php?wuid=11595159

Continual errors and sometimes run a long time before error.

It's not only low end cards

https://www.gpugrid.net/workunit.php?wuid=11595159

Continual errors and sometimes run a long time before error.

Would it make sense if a moderator or staff contacted these repeat offenders with a PM, asking them if they would consider detaching or try a less strenuous project?

I think they've done this procedure on CPDN, when users find hosts that burn workunits over and over.

[...] The problem with equilibrations is that we cannot split them into multiple steps like the normal simulations so we just have to push through this batch and then we are done. I expect it to be over by the end of the week. No more simulations are being sent out a few days now so only the ones that cancel or fail will be resent automatically.

100.440 Natoms 16h 36m 46s (59.806s) 903.900 credits (GTX980Ti)

Or just deny them new tasks until they login as your previous idea. This would solve the probem for the most part. Just can't understand the project not adopting this approach as they would like WUs returned ASAP and users don't want to be sat idle while these hosts hold us up

100.440 Natoms 16h 36m 46s (59.806s) 903.900 credits (GTX980Ti)

Is 903,900 the most credit ever given to an ACEMD WU?

100.440 Natoms 16h 36m 46s (59.806s) 903.900 credits (GTX980Ti)

Is 903,900 the most credit ever given to an ACEMD WU?

I think it is so far.
There is a larger model which contains 101.237 atoms, and this could generate 911,100 credits with the +50% bonus. Unfortunately one of my slower (GTX980) host received one such task, and it has been processed under 90,589 seconds (25h 9m 49s) (plus the time it spent queued), so it has earned "only" +25% bonus, resulting in "only" 759,250 credits.

I got that beat with 113536 Natoms. It took my computer over 30 hours to complete, so I lost the 50% bonus.

Mines bigger, 117122 Natoms took around 24 hours, 878,500.00 credits with the 25% bonus.

717000 credits (with 25% bonus) was the highest I received. Would have been 860700 if returned inside 24h, but would require a bigger card.
On Linux or Win XP I'm sure a GTX970 could return some of these inside 24h.

Might have got the lowest credit though ;p
73318 Natoms 9.184 ns/day:

15097992 10 May 2016 | 20:17:32 UTC 12 May 2016 | 0:29:05 UTC 91,659.53 91,074.92 275,000.00

Mines bigger, 117122 Natoms took around 24 hours, 878,500.00 credits with the 25% bonus.

Might have got the lowest credit though ;p
73318 Natoms 9.184 ns/day:

15097992 10 May 2016 | 20:17:32 UTC 12 May 2016 | 0:29:05 UTC 91,659.53 91,074.92 275,000.00

My GPU(s) current (opm996 long WU) estimated total Runtime. Completion rate (based on) 12~24hours of real-time crunching.

2r83R1 (GTX970) = 27hr 45min (3.600% per 1hr @ 70% GPU usage / 1501MHz)
1bakR0 (GTX970) = 23hr 30min (4.320% per 1hr @ 65% / 1501MHz)
1u27R2 (GTX750) = 40hr (2.520% per 1hr @ 80% / 1401MHz)
2I35R5 (GT650m) = 70hr (1.433% per 1hr @ 75% / 790MHz)

Newer (Beta) BOINC clients introduced an (accurate) per min or hour progress rate feature - available in advanced view (task proprieties) commands bar.

I got that beat with 113536 Natoms. It took my computer over 30 hours to complete, so I lost the 50% bonus.

Wow, that would get 1.021.800 credits with the +50% bonus. I hope that one of my GTX980Ti hosts will receive such a workunit :)

My GPU(s) current (opm996 long WU) estimated total Runtime. Completion rate (based on) 12~24hours of real-time crunching.

2r83R1 (GTX970) = 27hr 45min (3.600% per 1hr @ 70% GPU usage / 1501MHz)
1bakR0 (GTX970) = 23hr 30min (4.320% per 1hr @ 65% / 1501MHz)
1u27R2 (GTX750) = 40hr (2.520% per 1hr @ 80% / 1401MHz)
2I35R5 (GT650m) = 70hr (1.433% per 1hr @ 75% / 790MHz)

Newer (Beta) BOINC clients introduced an (accurate) per min or hour progress rate feature - available in advanced view (task proprieties) commands bar.

It's there in 7.6.22 (non-beta). For the GERARD_FXCXCL tasks it's about 8% on my 970's.

The OPM WUs spelled the death knell for my last 2 super-clocked 650 Ti GPUs. They weren't too bad with the earlier WUs but were ridiculously slow with the OPMs. Probably due to having only 1GB of memory. Anyway, pulled them out of the machines. Down to a flock (perhaps: gaggle, pack, herd, swarm, pod?) of 2GB 750 Ti cards and a 670.

Clutch, brood, rookery, skulk, crash, congregation, sleuth, school, shoal, army, quiver, gang, pride, bank, bouquet.

Also noticed that machines with only 1 NV GPU processed the OPMs faster. This wasn't the case for earlier WUs.

Probably due to the increased bus and CPU usage. I tried to alleviate this by freeing up more CPU and increasing the GDDR5 freq., but the GPU clocks could run higher too, due to slightly less GPU and power usage.

I was lucky to get one such workunit.
It's 64.5% processed under 12 hours, so it will take ~18h 30m to complete.
I keep my fingers crossed :) We'll see the result in the morning.

I got that beat with 113536 Natoms. It took my computer over 30 hours to complete, so I lost the 50% bonus.

Wow, that would get 1.021.800 credits with the +50% bonus. I hope that one of my GTX980Ti hosts will receive such a workunit :)

I was lucky to get one such workunit on a GTX980Ti host.
It's 64.5% processed under 12 hours, so it will take ~18h 30m to complete.
I keep my fingers crossed :) We'll see the result in the morning.

1.021.800 credits ...

Waiting on (2) formerly Timed out OPM to finish up on my 970's (23hr & 25hr estimated runtime). OPM WU was sent after hot day then cool evening ocean breeze -97 error GERARD_FXCXCL (50C GTX970) bent the knee for May sun.

I got that beat with 113536 Natoms. It took my computer over 30 hours to complete, so I lost the 50% bonus.

Wow, that would get 1.021.800 credits with the +50% bonus. I hope that one of my GTX980Ti hosts will receive such a workunit :)

I was lucky to get one such workunit on a GTX980Ti host.
It's 64.5% processed under 12 hours, so it will take ~18h 30m to complete.
I keep my fingers crossed :) We'll see the result in the morning.

I'm happy to report that the workunit finished fine:
1sujR0-SDOERR_opm996-0-1-RND0758_1 1.021.800 credits (including 50% bonus), 67.146 sec (18h 39m 6s), 113536 atoms 6.707 ns/day

Though, this did cost me my number 1 position in the performance tab.

@Retvari

Congrats to No. 1 of the TOP Crunchers ;-)
By the way: Does anybody knows what happened to Stoneageman? Sunk silently in the ground?

Got one of the looser-files again: https://www.gpugrid.net/result.php?resultid=15103135. Only 171,150 Credits for a runtime of 64,394.64

Now I've got this one: https://www.gpugrid.net/result.php?resultid=15104507 Haw can I see if it's a good ore a bad one?
____________
Regards, Josef

Top Average Performers is a very misleading and ill-conceived chart because it is based on average performance of a user and ALL his hosts rather than a host in particular.
Retvari has a lot of hosts with a mixture of cards and arguably hosts with the fastest return and throughput on GPUGrid. This mixture of hosts/cards puts him well in front on WUs completed but because times are averaged, behind on performance in hours.
Bedrich has only 2 hosts with at least 2 980ti's and possibly 3, so, because he doesn't have any slower cards when his return time in hours is averaged over all his hosts/cards end up at the top of the chart despite producing less than half of completed WUs as Retvari.

Got one of the looser-files again: https://www.gpugrid.net/result.php?resultid=15103135. Only 171,150 Credits for a runtime of 64,394.64

Now I've got this one: https://www.gpugrid.net/result.php?resultid=15104507 Haw can I see if it's a good ore a bad one?

@Retvari

Congrats to No. 1 of the TOP Crunchers ;-)
By the way: Does anybody knows what happened to Stoneageman? Sunk silently in the ground?

Got one of the looser-files again: https://www.gpugrid.net/result.php?resultid=15103135. Only 171,150 Credits for a runtime of 64,394.64

Now I've got this one: https://www.gpugrid.net/result.php?resultid=15104507 Haw can I see if it's a good ore a bad one?

Waiting on (2) formerly Timed out OPM to finish up on my 970's (23hr & 25hr estimated runtime). OPM WU was sent after hot day then cool evening ocean breeze -97 error GERARD_FXCXCL (50C GTX970) bent the knee for May sun.

Your GPUs are too hot. Your GT 630 reaches 80°C (176°F), while in your laptop your GT650M reaches 93°C (199°F) which is crazy.
Your host with 4 GPUs has two GTX970s, a GTX 750 and a GT630.
There's no point in risking the stability of the simulations running on your fast GPUs by putting low-end GPUs in the same host.
Packing 4 GPU to a single PC for 24/7 crunching requires water cooling, (or PCIe riser cards to make breathing space between the cards).
Crunching on laptops is not recommended. But if you do, you should place your laptop on its side while not in use, to make the air outlet facing up and the bottom of the laptop vertical (so the fan could take more air in). You should also regularly clean the fan & the fins with compressed air.

Heed the good advice!

Note that 93C is the GPU's temperature cut-off point. The GPU self-throttles to protect itself because it's dangerously hot. It doesn't have a cut-off point to protect the rest of the system and GPU's are Not designed to run at high temps continuously. Use temperature and fan controlling apps such as NVIDIA Inspector and MSI Afterburner to protect your hardware.

I get one of these long runs to my notebook with GT740M. As it was slow, I stopped all other CPU projects; it was still slow. After about 60 - 70 hours it was still about %50. Anyway, it ended up with errors and I'm not getting any long runs anymore. It seems they won't finish on time...
Waiting for short runs now.

both of the below WUs crunched with a GTX980Ti:

e4s15_e2s1p0f633-GERARD_FXCXCL12R_2189739_1-0-1-RND7197_0
22,635.45 / 22,538.23 / 249,600.00

1bakR6-SDOERR_opm996-0-1-RND6740_2
41,181.13 / 40,910.27 / 236,250.00

the second one almost double crunching time, but less points earned.
what explains for this big difference?

By the way: Does anybody knows what happened to Stoneageman?

The Natom amount only known after a WU validates cleanly in (<stderr> file).

He is crunching Einstein@home for some time now.

The Natom amount only known after a WU validates cleanly in (<stderr> file)The number of atoms of a running task can be found in the project's folder, in a file named as the task plus a _0 attached to the end.
Though it has no .txt extension this is a clear text file, so if you open it with notepad you will find a line (5th) which contains this number:

# Topology reports 32227 atoms

Another way to look at it is that you are doing cutting edge theoretical/proof of concept science, never done before - it's bumpy.

This one took over 5 days to get to me https://www.gpugrid.net/workunit.php?wuid=11595181

Completed in just under 24hrs for 1,095,000

Come on admins do something about the "5 Day Timeout" and continual error machines. Next WU took over 6 days to get to me. https://www.gpugrid.net/workunit.php?wuid=11595161 and also stop people caching WUs for more than an hour.