YES! Thank you so much!
____________

I noticed the plan class is listed as "cuda1121" on the Applications page. is this a typo? will it cause any issues with getting work or running the application?

also you might need to put a cap (maybe compute capability or something) on the project server side to prevent the CUDA10.0 app from being sent to Ampere hosts. currently we saw many errors because the CUDA10.0 app was still sent to Ampere hosts. there should be a way to make sure Ampere hosts only get the 11.2 app and not try to use the cuda 10 app.
____________

So far.

Just so people are aware, CUDA 11.2 (I assume the "1121" means CUDA 11.2.1 "update 1") means you need at least driver 460.32 on Linux.
____________

Can someone confirm that the Linux cuda100 app is still sent out (and likely fail)?

T

I just got a couple tasks on my RTX 3080 Ti host, it got the new app. it failed in 2 seconds. it looks like you're missing a file, or you forgot to statically link boost into the app:

16:50:34 (15968): wrapper (7.7.26016): starting
16:50:34 (15968): wrapper (7.7.26016): starting
16:50:34 (15968): wrapper: running acemd3 (--boinc input --device 0)
acemd3: error while loading shared libraries: libboost_filesystem.so.1.74.0: cannot open shared object file: No such file or directory
16:50:35 (15968): acemd3 exited; CPU time 0.000360
16:50:35 (15968): app exit status: 0x7f
16:50:35 (15968): called boinc_finish(195)

Looks like Ubuntu 20.04.2 LTS has libboost-all-dev 1.71 installed.

I remember that Gridcoin needs libboost-all-dev 1.74 installed now also when building.

That is in 21.04.

[Edit]
Theoretically yes AFAIK anything about wrapper containers.

I just wonder if you installed the latest 1.74 libboost-all-dev environment that the tasks wouldn't fail.

https://www.boost.org/users/history/version_1_74_0.html

i think these are sandboxed in the wrapper. so packages on the system in theory shouldnt matter right?
____________

Just failed a couple more acemd3 tasks. What a waste . . . . as hard as they are to snag.

Just failed a couple more acemd3 tasks. What a waste . . . . as hard as they are to snag.

No I just have the normal CUDA 10.0 app installed. I am just investigating what would be needed to install the missing libraries.

Great to see this progress as prices of GPUs are beginning to fall and Ampere GPUs are currently dominating the market availability. I hope China's Ban on mining becomes a budgetary boon for crunchers and gamers worldwide.

The Amperes should eventually expedite this project considerably.
____________
"Together we crunch
To check out a hunch
And wish all our credit
Could just buy us lunch"

Piasa Tribe - Illini Nation

No I just have the normal CUDA 10.0 app installed. I am just investigating what would be needed to install the missing libraries.

No I just have the normal CUDA 10.0 app installed. I am just investigating what would be needed to install the missing libraries.

looks like you're actually getting the new app now: http://www.gpugrid.net/result.php?resultid=32631755

I think you’ll only get 11.2 app if you have a driver that’s compatible. Greater than 460.32. Just my guess. I’ll need to see if systems with and older driver will still get the cuda 100 app

Edit: answering my own question. I guess the driver being reported doesn’t factor into the app selection anymore. My systems reporting an older driver still received the new app. So it won’t prevent the app from being send to someone without a new enough driver.
____________

I'm still trying to figure out the best way to distribute the app. The current way has hard-coded minimum-maximum driver versions for each CUDA version and it's too cumbersome to maintain.

Suggestions are welcome. The server knows the client's CUDA version and driver version, as well as the app's CUDA plan class.

Toni, I think the first thing that needs to be fixed is the problem with boost 1.74 library not being included in the app distribution. the app is failing right away because it's not there. you either need to distribute the .so file or statically link it into the acemd3 app so it's not needed separately.

manually installing it seems to be a workaround, but that's a tall order to make every Linux user have to perform.
____________

after manually installing the required boost to get past that error, I now get this error on my 3080 Ti system:

09:55:10 (4806): wrapper (7.7.26016): starting
09:55:10 (4806): wrapper (7.7.26016): starting
09:55:10 (4806): wrapper: running acemd3 (--boinc input --device 0)
ACEMD failed:
Error launching CUDA compiler: 32512
sh: 1: : Permission denied

09:55:11 (4806): acemd3 exited; CPU time 0.479062
09:55:11 (4806): app exit status: 0x1
09:55:11 (4806): called boinc_finish(195)

Multiple failures of this task on both windows and linux

http://www.gpugrid.net/workunit.php?wuid=16517304

<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
15:19:27 (30109): wrapper (7.7.26016): starting
15:19:27 (30109): wrapper (7.7.26016): starting
15:19:27 (30109): wrapper: running acemd3 (--boinc input --device 0)
# Engine failed: Error launching CUDA compiler: 32512
sh: 1: : Permission denied

15:19:28 (30109): acemd3 exited; CPU time 0.186092
15:19:28 (30109): app exit status: 0x1
15:19:28 (30109): called boinc_finish(195)

</stderr_txt>

Why is the app launching CUDA compiler?

It was a cryptic bug in the order loading shared libraries, or something like that. Otherwise unexplainably system-dependent.

I see VERY few failures now. The new app will be a huge step forward on several aspects, not least maintainability. We'll be transitioning gradually.

I deployed the new app, which now requires cuda 11.2 and hopefully support all the latest cards. Touching the cuda versions is always a nightmare in boinc scheduler so expect problems.

sudo add-apt-repository ppa:mhier/libboost-latest
sudo apt-get update
sudo apt-get install boost1.74
reboot

I deployed the new app, which now requires cuda 11.2 and hopefully support all the latest cards. Touching the cuda versions is always a nightmare in boinc scheduler so expect problems.

Thank you so much.
Those efforts are for noble reasons.

Regarding persistent errors:
I also manually installed boost as a try at one of my Ubuntu 20.04 hosts, by means of the following commands:

sudo add-apt-repository ppa:mhier/libboost-latest
sudo apt-get update
sudo apt-get install boost1.74
reboot

But a new task downloaded after that still failed:
e3s644_e1s419p0f770-ADRIA_New_KIXcMyb_HIP_AdaptiveBandit-1-2-RND9285_4
Then, I've reset GPUGrid project, and it seems that it did the trick.
A new task is currently running on this host, instead of failing after a few seconds past:
e4s126_e3s248p0f238-ADRIA_New_KIXcMyb_HIP_AdaptiveBandit-0-2-RND6347_7
49 minutes, 1,919% progress by now.

nope, even after the project reset, still the same error

process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
11:42:55 (5665): wrapper (7.7.26016): starting
11:42:55 (5665): wrapper (7.7.26016): starting
11:42:55 (5665): wrapper: running acemd3 (--boinc input --device 0)
ACEMD failed:
Error launching CUDA compiler: 32512
sh: 1: : Permission denied

11:42:56 (5665): acemd3 exited; CPU time 0.429069
11:42:56 (5665): app exit status: 0x1
11:42:56 (5665): called boinc_finish(195)

sudo add-apt-repository ppa:mhier/libboost-latest
sudo apt-get update
sudo apt-get install libboost1.74
reboot

Maybe that your problem is an Ampere-specific one (?).

I've catched a new task in another of my hosts after applying the same remedy, and it is also running as expected.
e3s263_e1s419p0f938-ADRIA_New_KIXcMyb_HIP_AdaptiveBandit-1-2-RND6959_2
25 minutes, 0,560% progress by now for this second task.
Turing GPUs and Nvidia drivers 465.31 on both hosts.
Installing libboost1.74 didn't worked for me by itself.
Resetting project didn't worked for me by itself.
Installing libboost1.74 and resetting project afterfards, did work for both my hosts.
I've doublechecked, and the commands that I employed were the previously published at message #57064
Watching Synaptic, this lead to libboost1.74 and libboost1.74-dev were correctly installed.

Maybe that your problem is an Ampere-specific one (?).

I've catched a new task in another of my hosts after applying the same remedy, and it is also running as expected.
e3s263_e1s419p0f938-ADRIA_New_KIXcMyb_HIP_AdaptiveBandit-1-2-RND6959_2
25 minutes, 0,560% progress by now for this second task.
Turing GPUs and Nvidia drivers 465.31 on both hosts.
Installing libboost1.74 didn't worked for me by itself.
Resetting project didn't worked for me by itself.
Installing libboost1.74 and resetting project afterfards, did work for both my hosts.
I've doublechecked, and the commands that I employed were the previously published at message #57064
Watching Synaptic, this lead to libboost1.74 and libboost1.74-dev were correctly installed.

Maybe that your problem is an Ampere-specific one (?).

I've catched a new task in another of my hosts after applying the same remedy, and it is also running as expected.
e3s263_e1s419p0f938-ADRIA_New_KIXcMyb_HIP_AdaptiveBandit-1-2-RND6959_2
25 minutes, 0,560% progress by now for this second task.
Turing GPUs and Nvidia drivers 465.31 on both hosts.
Installing libboost1.74 didn't worked for me by itself.
Resetting project didn't worked for me by itself.
Installing libboost1.74 and resetting project afterfards, did work for both my hosts.
I've doublechecked, and the commands that I employed were the previously published at message #57064
Watching Synaptic, this lead to libboost1.74 and libboost1.74-dev were correctly installed.

I had this thought. I put in my old 2080ti to the problem-host, and will see if it starts processing, or if it's really a problem with the host-specific configuration. this isn't the first time this has happened though. and Toni previously fixed it with an app update. so it looks like that will be needed again even if it's Ampere-specifc.

I just used the ppa method on my other two hosts. But I did not reboot.
Picked up another task and it is running.
Waiting still on the luck of the draw for the other host without work.

well, it seems it's not Ampere specific. it failed in the same way on my 2080ti here: https://www.gpugrid.net/result.php?resultid=32632521

still the CUDA compiler error

unfortunately I can't easily move the 3080ti to another system since it's a watercooled model that requires a custom water loop.

Congratulations!
Good news...Anxious to see the performance on a 3080 Ti

Well, it will be your problem, not mine. Even having decent hard- and software I am pretty astonished how folks like cosmology and the likes seemingly do not understand how VM and the like works. I am pretty pissed as an amateur, though...

Just seen my first failures with libboost errors on Linux Mint 20.1, driver 460.80, GTX 1660 super.

Applied the PPA and reset the project - waiting on the next tasks now.

Well, it will be your problem, not mine. Even having decent hard- and software I am pretty astonished how folks like cosmology and the likes seemingly do not understand how VM and the like works. I am pretty pissed as an amateur, though...

I think I finally solved the issue! it's running on the 3080ti finally!

now to see how long it'll take a 3080ti ;). it has over 10,000 CUDA cores so I'm hoping for a fast time. 2080ti runs about 12hrs, so it'll be interesting to see how fast I can knock it out. using about 310 watts right now.

I think I finally solved the issue! it's running on the 3080ti finally!

now to see how long it'll take a 3080ti ;). it has over 10,000 CUDA cores so I'm hoping for a fast time. 2080ti runs about 12hrs, so it'll be interesting to see how fast I can knock it out. using about 310 watts right now.

This is the moment of truth we're all waiting for.
My bet is 9h 15m.

I'm curious how you have a real estimated time remaining calculated for a brand new application.

AFAIK, you JUST got the application working and I don't believe you have validated ten tasks yet to get an accurate APR which produces the accurate estimated time remaining numbers.

All my tasks are in EDF mode and multiple day estimates simply because I have returned exactly one valid task so far. A shorty Cryptic-Scout task.

I didn’t use the time remaining estimate from BOINC. I estimated it myself based on % complete and elapsed time, assuming a linear completion rate.
____________

When I checked earlier, it was tracking a 12.5hr completion time. But the 2080ti was tracking a 14.5hr completion time.

Either the new run of tasks are longer, or the CUDA 11.2 app is slower? We’ll have to see.

The PPA-reset trick worked - I have a new task running now. Another satisfied customer.

The completion estimate at 10% was 43.5 hours, both by extrapolation and by setting <fraction_done_exact/> in app_config.xml

It's a ADRIA_New_KIXcMyb_HIP_AdaptiveBandit task. I ran a couple of these about 10 days ago, under the old app: they took about 33 hours - previous 'D3RBandit_batch*' tasks had been 28 hours on average. Cards are GTX 1660 super.

So there's a possibility that the new app is slower, at least on 'before cutting edge' cards.

Some of the tasks were even longer. I have two more 2080ti reports that were 55,583 and 56,560s respectively. On Identical GPUs running the same clocks. There seems to be Some variability. If you use the slower one it puts it closer to 30%. This exposes the flaw of using a single sample to form a conclusion. More data is required.

Also note that I’ve been experiencing performance issues with this specific card. I believe it’s underperforming due to some incorrect power limiting behavior (I’ve done a lot of load testing and cross referencing benchmark results with others online). I have a replacement on the way to test.

These ADRIA tasks have hard coded reward. It isn’t necessarily based on run time. They increased the reward from the D3RBandit to these KIXcMyb tasks, but since they stopped distributing the CUDA 10 app, we can’t know for sure if the tasks are just longer or if there’s some inefficiency in the new 11.2 app that’s slowing it down. If the tasks aren’t longer, then the new app is almost 30% slower than the old CUDA app
____________

I looked back into the 'job_log_www.gpugrid.net.txt' in the BOINC data folder to get my comparison times. I haven't run many AdaptiveBandits yet, but I think the 'D3RBandit_batch*' time was a robust average over the many sub-types.

After cross referencing runtimes for various Windows hosts, I think the new app is just slower. Windows hosts haven’t experienced an app change (yet) and haven’t shown any sudden or recent change in run time with the KIX AdaptiveBandit jobs. This suggests that that tasks haven’t really changed, leading the only other cause of the longer run time to be a slower 11.2 app.

I also noticed that the package distribution is different between the CUDA 10 and 11.2 apps. 10 included some library files that are not included with 11.2 (like cudart and cudafft libraries) so the app may have been compiled in a different way.

I hope Toni can bring the app back to par. It really shouldn’t be that much slower.
____________

GTX 1080
# Speed: average 75.81 ns/day, current 75.71 ns/day

RTX 2070S
# Speed: average 134.99 ns/day, current 132.17 ns/day

RTX 3070
# Speed: average 159.15 ns/day, current 155.75 ns/day
https://www.gpugrid.net/result.php?resultid=32632515
https://www.gpugrid.net/result.php?resultid=32632513

only task yet is with 3070 and ended after 18-19 hours
3000-series looks slow with 11.2 but they works. Progressbar and estimate looks to be close expected time and 2070 could probably end after around 21 hours.

It would be great if Toni could make application to print from progress.log

Had to add PPA for libboost needed and did try update one host to 21.04 to get latest boost but did not work.

Where did you get the ns/day numbers from?

But it’s not just 3000-series being slow. All cards seem to be proportionally slower with 11.2 vs 10.0, by about 30%
____________

Go to slot folder and cat progress.log

Yes looks like all cards are affected on new application. I compare with 1000-series also but do not have numbers of ns/day for them.

Where did you get 469 driver? Can't see it on nvidia site or PPA.

It’s not real. I’ve manipulated the coproc_info file to report what I want.

Actual driver in use is 460.84
____________

RTX 3070
# Speed: average 159.15 ns/day, current 155.75 ns/day
https://www.gpugrid.net/result.php?resultid=32632515
https://www.gpugrid.net/result.php?resultid=32632513

only task yet is with 3070 and ended after 18-19 hours
3000-series looks slow with 11.2 but they works. Progressbar and estimate looks to be close expected time and 2070 could probably end after around 21 hours.

I didn’t use the time remaining estimate from BOINC. I estimated it myself based on % complete and elapsed time, assuming a linear completion rate.

RTX 3070
# Speed: average 159.15 ns/day, current 155.75 ns/day
https://www.gpugrid.net/result.php?resultid=32632515
https://www.gpugrid.net/result.php?resultid=32632513

only task yet is with 3070 and ended after 18-19 hours
3000-series looks slow with 11.2 but they works. Progressbar and estimate looks to be close expected time and 2070 could probably end after around 21 hours.

The WU you linked had one wingman run it as cuda 10.1 and the other as 11.21 with 155,037 seconds versus 68,000. Isn't that faster?
https://www.gpugrid.net/workunit.php?wuid=27075862
What does ns mean?

It’s not real. I’ve manipulated the coproc_info file to report what I want.

Actual driver in use is 460.84

It’s not real. I’ve manipulated the coproc_info file to report what I want.

Actual driver in use is 460.84

Ok why i ask was that device name is unknown for my 3080Ti and had some hope that driver you used would fix that.
So i could go coproc file and edit instead.

Greger, I just can't get my head around what it means. So out of the 8.64E13 ns in a day you only calculate for 159 ns??? I'm not familiar with that figure of merit.

BTW, my 3080 is running 465.31. Still waiting to catch a WU after the PPA, reboot & reset.

The nanoseconds will be the biochemical reaction time that we're modelling - very, very, slowly - in a digital simulation.

Greger, I just can't get my head around what it means. So out of the 8.64E13 ns in a day you only calculate for 159 ns??? I'm not familiar with that figure of merit.

BTW, my 3080 is running 465.31. Still waiting to catch a WU after the PPA, reboot & reset.

It’s not real. I’ve manipulated the coproc_info file to report what I want.

Actual driver in use is 460.84

Ok why i ask was that device name is unknown for my 3080Ti and had some hope that driver you used would fix that.
So i could go coproc file and edit instead.

What driver are you using? The 3080ti won’t be detected until driver 460.84. Anything older will not know what GPU that is.

You need driver 460.84 for 3080ti. You can use that one.

You can also use 465.31, but that driver is about a month older, 460.84 will be better unless you absolutely need some feature from the 465 branch.
____________

Greger, I just can't get my head around what it means. So out of the 8.64E13 ns in a day you only calculate for 159 ns??? I'm not familiar with that figure of merit.

BTW, my 3080 is running 465.31. Still waiting to catch a WU after the PPA, reboot & reset.

You need driver 460.84 for 3080ti. You can use that one.

You can also use 465.31, but that driver is about a month older, 460.84 will be better unless you absolutely need some feature from the 465 branch.

Yea, snagged a WU and it's running. My guesstimate is 19:44:13 on my 3080 dialed down to 230 Watts. Record breaking long heat wave here and summer peak Time-of-Use electric rates (8.5x higher) have started. Summer is not BOINC season in The Great Basin.

Rxn time, now that makes sense. Thx.

Linux Mint repository offers 465.31 and 460.84. Is it actually worth reverting to 460.84??? I wouldn't do it until after this WU completes anyway.

Linux Mint repository offers 465.31 and 460.84. Is it actually worth reverting to 460.84??? I wouldn't do it until after this WU completes anyway.

second 3080ti task completed in 11hrs

http://gpugrid.net/result.php?resultid=32632580
____________

peak 28,9°C here today so suspend during daytime after 2 task done.
I run evening and night these days if temp i high. Ambient temp was above 35 inside and fan gone up to 80% on gpu i checked.

So i manage to go to 460.84 after a few remove and --purge nvidia*. Apparently there was a libnvidia-compute left and hold it back.

Got name correct but detect vram wrong (4095MB). Lets see if it would work.

Just take in mind that any change in Nvidia driver version while a GPUgrid task is in progress, will cause it to fail when computing is restarted.
Commented in message #56909

peak 28,9°C here today so suspend during daytime after 2 task done.
I run evening and night these days if temp i high. Ambient temp was above 35 inside and fan gone up to 80% on gpu i checked.

So i manage to go to 460.84 after a few remove and --purge nvidia*. Apparently there was a libnvidia-compute left and hold it back.

Got name correct but detect vram wrong (4095MB). Lets see if it would work.

I went back to my host and driver crashed. smi unable to open and task failed on another project. Restarted it and back on track. Few minutes later it fetch new task from GPUGrid. Let's hope it does not crash again.

https://www.gpugrid.net/result.php?resultid=32634065

# Speed: average 225.91 ns/day, current 226.09 ns/day
That is more like. this much better then my 3070 and 3060Ti got.

GPU detection is handled by BOINC, not any individual projects.

Driver updates always require a reboot to take effect.
____________

Finally, my first result of a new version 2.12 task came out in my fastest card:
e4s126_e3s248p0f238-ADRIA_New_KIXcMyb_HIP_AdaptiveBandit-0-2-RND6347_7
It took 141948 seconds of total processing time. That is: 1 day 15 hours 25 minutes and 48 seconds
Predicted time in table shown at message #57091 was 142074 seconds after 61,439% done.
There is a slight difference of 126 seconds between estimated and true execution time. 0,09% deviation.
For me, it is approximate enough, and validates Ian&Steve C. theory of progress for these tasks being quite linear along their execution.

Compare old and new app on 2070S

old
52,930.87 New version of ACEMD v2.11 (cuda100)
WU 27069210 e130s1888_e70s25p0f44-ADRIA_D3RBandit_batch_nmax5000-0-1-RND2852_1

new
80,484.11 New version of ACEMD v2.12 (cuda1121)
WU: 27077230 e5s177_e4s56p0f117-ADRIA_New_KIXcMyb_HIP_AdaptiveBandit-1-2-RND4081_4

Not sure if size of units grown that much to be able compare them.

My guesstimate is 19:44:13 on my 3080 dialed down to 230 Watts.

At this moment, every of my 7 currently working GPUs have any new version 2.12 task in process.
Two tasks received today completed the quota.
Task e4s120_e3s763p0f798-ADRIA_New_KIXcMyb_HIP_AdaptiveBandit-1-2-RND9850_3, hanging from WU #27076712
Task e5s90_e4s138p0f962-ADRIA_New_KIXcMyb_HIP_AdaptiveBandit-1-2-RND6130_4, hanging from WU #27077322
Something to remark: These two tasks are repetitive resends of previously failed tasks with the following known problem:

acemd3: error while loading shared libraries: libboost_filesystem.so.1.74.0: cannot open shared object file: No such file or directory

So the number of the usable CUDA cores in the 30xx series are half of the advertised number (just as I expected), as 10240/2=5120, 5120/4352=1.1765 (so the 3080Ti has 17.65% more CUDA cores than the 2080Ti has), the CUDA cores of the 3080Ti are 1.4% faster than of the 2080Ti.

Does using half of CUDA cores have implications for BOINCing?
GG+OPNG at <cpu_usage>1.0</cpu_usage> & <gpu_usage>0.5</gpu_usage> works fine.
GG+DaggerHashimoto crashes GG instantly.
I hope to try 2xGG today.

So the number of the usable CUDA cores in the 30xx series are half of the advertised number (just as I expected), as 10240/2=5120, 5120/4352=1.1765 (so the 3080Ti has 17.65% more CUDA cores than the 2080Ti has), the CUDA cores of the 3080Ti are 1.4% faster than of the 2080Ti.

Does using half of CUDA cores have implications for BOINCing?
GG+OPNG at <cpu_usage>1.0</cpu_usage> & <gpu_usage>0.5</gpu_usage> works fine.
GG+DaggerHashimoto crashes GG instantly.
I hope to try 2xGG today.

if GPUGRID isn't seeing the effective 2x benefit of Turing vs Ampere, that tells me one of two things (or maybe some combination of both):
1. that app isn't as FP32 heavy as some have implied, and maybe has a decent amount of INT32 instructions. the INT32 setup of Ampere is the same as Turing
2. there is some additional optimization that needs to be applied to the ACEMD3 app to better take advantage of the extra FP32 cores on Ampere.

I'm not convinced that the extra cores "aren't being used" at all, ie, the cores are sitting idle 100% of the time as a direct result of the architecture or something like that. I think both the application and the hardware are fully aware of the available cores/SMs. just that the application is coded in such a way that it can't take advantage of the extra resources, either in optimization or in the number of INT instructions required.

nvidia's press notes do seem to show a 1.5x improvement in molecular modeling load for A100 vs V100, so maybe the amount of INT calls is inherent to this kind of load anyway. (granted the A100 is based on the GA100 core, which is a different architecture without the shared FP/INT cores for the doubling of FP cores like on GA102)

but in the case of GPUGRID, i think it's just their application. on folding Ampere performs much closer to the claims. a 3070 being only a bit slower than a 2080ti, which is what I would expect.
____________

The 30xx series improved gaming experience much more, than the crunching performance.

running 2x GPUGRID tasks concurrently wont make it "use more". it'll just slow both down, probably slower than half speed due to the constant resource fighting.

if GPUGRID isn't seeing the effective 2x benefit of Turing vs Ampere, that tells me one of two things (or maybe some combination of both):
1. that app isn't as FP32 heavy as some have implied, and maybe has a decent amount of INT32 instructions. the INT32 setup of Ampere is the same as Turing
2. there is some additional optimization that needs to be applied to the ACEMD3 app to better take advantage of the extra FP32 cores on Ampere.

running 2x GPUGRID tasks concurrently wont make it "use more". it'll just slow both down, probably slower than half speed due to the constant resource fighting.

if GPUGRID isn't seeing the effective 2x benefit of Turing vs Ampere, that tells me one of two things (or maybe some combination of both):
1. that app isn't as FP32 heavy as some have implied, and maybe has a decent amount of INT32 instructions. the INT32 setup of Ampere is the same as Turing
2. there is some additional optimization that needs to be applied to the ACEMD3 app to better take advantage of the extra FP32 cores on Ampere.

At less than 5% complete with two WUs running simultaneously and having started within minutes of each other:
WU1: 4840 sec at 4.7% implies 102978 sec total
WU2: 5409 sec at 4.6% implies 117587 sec total
From yesterday's singleton: 2 x 58014 sec = 116028 sec total if independent.

But since Nvidia eliminated the nvidia-settings options -a [gpu:0]/GPUGraphicsClockOffset & -a [gpu:0]/GPUMemoryTransferRateOffset that I used I haven't found a good way to do it using Linux.

It seems Nvidia chooses a performance level but I can't see how to force it to a desired level:

Nvidia also eliminated GPULogoBrightness so the baby-blinkie lights never turn off.

It sure does not look like running multiple GG WUs on the same GPU has any benefit.
My 3080 is stuck in P2. I'd like to try it in P3 and P4 but I can't make it change. I tried:
nvidia-smi -lmc 9251
Memory clocks set to "(memClkMin 9501, memClkMax 9501)" for GPU 00000000:65:00.0
All done.
nvidia-smi -lgc 240,2130
GPU clocks set to "(gpuClkMin 240, gpuClkMax 2130)" for GPU 00000000:65:00.0
All done.

But it's still in P2.

Aurum wrote:

But since Nvidia eliminated the nvidia-settings options -a [gpu:0]/GPUGraphicsClockOffset & -a [gpu:0]/GPUMemoryTransferRateOffset that I used I haven't found a good way to do it using Linux.

these options still work. I use them for my 3080Ti. not sure what you mean?

this is exactly what I use for my 3080Ti (same on my Turing hosts)

/usr/bin/nvidia-settings -a "[gpu:0]/GPUMemoryTransferRateOffset[4]=500" -a "[gpu:0]/GPUGraphicsClockOffset[4]=100"

it works as desired.

but all the slightly off-topic aside.

It was a great first step to getting the app working for Ampere. it's been long awaited and the new app is much appreciated and now many more cards can help contribute to the project, especially with these newer long running tasks lately. we need powerful cards to handle these tasks.

I think the two priorities now should be:

1. remedy the dependency on boost. either include the necessary library in the package distribution to clients, or recompile the app with boost statically linked. otherwise only those hosts who recognize the problem and know how to manually install the proper boost package will be able to contribute.

2. investigate the cause and provide a remedy for the ~30% slowdown in application performance from the older cuda100 app. this isn't just affecting Ampere, but affecting all GPUs equally it seems. maybe some optimization flag was omitted or some change to the code was made that was undesirable or unintended. just changing from cuda100 to cuda1121 should not in itself have caused this if there were no other code changes. sometimes you can see slight performance changes like 1-2%, but a 30% reduction is a sign that something is clearly wrong.
____________

Aurum wrote:

But since Nvidia eliminated the nvidia-settings options -a [gpu:0]/GPUGraphicsClockOffset & -a [gpu:0]/GPUMemoryTransferRateOffset that I used I haven't found a good way to do it using Linux.

these options still work. I use them for my 3080Ti. not sure what you mean?

this is exactly what I use for my 3080Ti (same on my Turing hosts)

/usr/bin/nvidia-settings -a "[gpu:0]/GPUMemoryTransferRateOffset[4]=500" -a "[gpu:0]/GPUGraphicsClockOffset[4]=100"

it works as desired.

How do you prove to yourself they work? They don't even exist any more. Run

nvidia-settings -q all | grep -C 10 -i GPUMemoryTransferRateOffset

and you will not find either of them.

I'm still getting the CUDA compiler permission denied error. I've added the PPA and installed libboost1.74 as above, and reset the project multiple times. But every downloaded task fails after 2 seconds.

http://www.gpugrid.net/result.php?resultid=32636087

I'm running Mint 20.1, with rtx2070 and rtx3070 cards running 465.31 drivers.

I'm still getting the CUDA compiler permission denied error. I've added the PPA and installed libboost1.74 as above, and reset the project multiple times. But every downloaded task fails after 2 seconds.

http://www.gpugrid.net/result.php?resultid=32636087

I'm running Mint 20.1, with rtx2070 and rtx3070 cards running 465.31 drivers.

Ian&Steve C wrote:

It was a great first step to getting the app working for Ampere. it's been long awaited and the new app is much appreciated and now many more cards can help contribute to the project, especially with these newer long running tasks lately. we need powerful cards to handle these tasks.

I think the two priorities now should be:

1. remedy the dependency on boost. either include the necessary library in the package distribution to clients, or recompile the app with boost statically linked. otherwise only those hosts who recognize the problem and know how to manually install the proper boost package will be able to contribute.

2. investigate the cause and provide a remedy for the ~30% slowdown in application performance from the older cuda100 app. ...

I'm still getting the CUDA compiler permission denied error. I've added the PPA and installed libboost1.74 as above, and reset the project multiple times. But every downloaded task fails after 2 seconds.

http://www.gpugrid.net/result.php?resultid=32636087

I'm running Mint 20.1, with rtx2070 and rtx3070 cards running 465.31 drivers.

How did you install the drivers? Have you ever installed the CUDA toolkit? This was my problem. If you have a CUDA toolkit installed, remove it. I would also be safe and totally purge your nvidia drivers and re-install fresh.

did you use the included removal script to remove the toolkit? or did you manually delete some files? definitely try the removal script if you havent already. good luck!
____________

...
1. remedy the dependency on boost. either include the necessary library in the package distribution to clients, or recompile the app with boost statically linked. otherwise only those hosts who recognize the problem and know how to manually install the proper boost package will be able to contribute.
...

Richard Haselgrove sait at Message #57177:

Look at that timeout: host 528201. Oh, Mr. Kevvy, where art thou? 156 libboost errors? You can fix that...

I got a note back from Mr. K - he saw the errors, and was going to check his machines. I imagine he's applied Ian's workround.

Curing the world's diseases, one computer at a time. It would be better if that bug could be fixed at source, for a universal cure.

On July 3rd 2021, Ian&Steve C. wrote at Message #57087:

But it’s not just 3000-series being slow. All cards seem to be proportionally slower with 11.2 vs 10.0, by about 30%

Also bear in mind that your first screenshot shows a D3RBandit task, and your second shows a AdaptiveBandit task.

They are different, and not directly comparable. How much of the observed slowdown is down to the data/algorithm, and how much is down to the new application, will need further examples to unravel.

Also bear in mind that your first screenshot shows a D3RBandit task, and your second shows a AdaptiveBandit task.

Is GPU grid still doing anything? I haven't gotten any work in like a month or more. And before that is was just sporadic. I used to always have work units. Now, nothing.

I am not receiving Windows tasks anymore. My configuration is
Boinc 7.16.11 GenuineIntel Intel(R) Xeon(R) CPU E5620 @ 2.40GHz [Family 6 Model 44 Stepping 2](4 processors)

NVIDIA GeForce GTX 1060 6GB (4095MB) driver: 461.40

Microsoft Windows 10 Professional x64 Edition, (10.00.19043.00)

Am I still within Spec's to get Windows acemd3 work ?

Thanks
Bill F
____________
In October of 1969 I took an oath to support and defend the Constitution of the United States against all enemies, foreign and domestic;
There was no expiration date.

there hasnt been an appreciable amount of work available for over a month.
____________

there hasnt been an appreciable amount of work available for over a month.

there hasnt been an appreciable amount of work available for over a month.

:-( :-( :-(

Yes, that is all I've been getting lately. I had 4 CRYPTICSCOUT_pocket_discovery tasks 4 days ago and I got 2 more today.

Another two more today.

what I don't understand is that there is no word whatsoever from the project team about an even tentative schedule :-(
Will there be new tasks available in say 1 week, 1 month, 3 months, ... ?
Will there be a new app which covers Ampere cards also (for both Linux and Windows) ... ?

still no tasks on the two hosts that have been having some issue getting work since the new app was released. I've set NNT on all other hosts in order to try to funnel any available work to these two hosts.

it's like they've been shadow-banned or something. even when work is available, it gets the message that no work is available. after an entire month, these hosts should have picked up at least one task.
____________

Well I stepped out on a limb and side emailed the Principal Investigator listed for the project and the University regarding the lack of any communications.

If you see a puff of smoke in the Dallas TX area and the user count goes down by one you will know that I was hit by a lighting bolt or a small thermonuclear device.

Bill F
Dallas
____________
In October of 1969 I took an oath to support and defend the Constitution of the United States against all enemies, foreign and domestic;
There was no expiration date.

It worked! And Texas is still there, the last time I checked.
http://www.gpugrid.net/forum_thread.php?id=5246

Moved to libboost thread.

Hello;
The new acemd3 app saved no provisional result, like "PrimeGrid" or "Einstein". I have a RTX3070 and i let there run with ~120W. With boinccmd and Systemdtimer, i start the GPU-Calculation for 15min and then 5min stop. At every start, the WU starts with 0%. Please look at this behaviour.
Thanks.

Hello;
The new acemd3 app saved no provisional result, like "PrimeGrid" or "Einstein". I have a RTX3070 and i let there run with ~120W. With boinccmd and Systemdtimer, i start the GPU-Calculation for 15min and then 5min stop. At every start, the WU starts with 0%. Please look at this behaviour.
Thanks.

Why are so many Acemd3 tasks failing on Windows host(s)? Mine has not succeeded on any task lately. Same errors can be seen on many other hosts too, like WU 27085364.

There's another Acemd3 test ongoing today:

TEST_24_41-RAIMIS_TEST-0-1-RND0310_0

Seems to have worked for me, at least.

all of mine failed, because they sent only the CUDA101 app to my Ampere host.
____________

all of mine failed, because they sent only the CUDA101 app to my Ampere host.

I am running my 1st New version of ACEMD 2.19 (cuda1121) task and it has reached a percentage of 35.833 and shows a Running status but the Elapsed time and Time remaining are not incrementing ?

Task properties

Application
New version of ACEMD 2.19 (cuda1121)
Name
e7s30_e1s279p0f463-ADRIA_BanditGPCR_APJ_b0-0-1-RND6681
State
Running
Received
12/1/2021 1:25:59 AM
Report deadline
12/6/2021 1:25:59 AM
Resources
0.985 CPUs + 1 NVIDIA GPU
Estimated computation size
5,000,000 GFLOPs
CPU time
09:42:23
CPU time since checkpoint
---
Elapsed time 10:11:24
Estimated time remaining 1d 10:10:29
Fraction done 35.833%
Virtual memory size 0 bytes
Working set size0 bytes
Directory slots/0
Process ID 10056
Progress rate 3.600% per hour
Executable wrapper_6.1_windows_x86_64.exe

Environment Win 10 NVIDIA GeForce GTX 1060 6GB (4095MB) driver: 472.47

Has anyone else seen this ?

Thanks
Bill F

____________
In October of 1969 I took an oath to support and defend the Constitution of the United States against all enemies, foreign and domestic;
There was no expiration date.

I am running my 1st New version of ACEMD 2.19 (cuda1121) task and it has reached a percentage of 35.833 and shows a Running status but the Elapsed time and Time remaining are not incrementing ?

Task properties

Application
New version of ACEMD 2.19 (cuda1121)
Name
e7s30_e1s279p0f463-ADRIA_BanditGPCR_APJ_b0-0-1-RND6681
State
Running
Received
12/1/2021 1:25:59 AM
Report deadline
12/6/2021 1:25:59 AM
Resources
0.985 CPUs + 1 NVIDIA GPU
Estimated computation size
5,000,000 GFLOPs
CPU time
09:42:23
CPU time since checkpoint
---
Elapsed time 10:11:24
Estimated time remaining 1d 10:10:29
Fraction done 35.833%
Virtual memory size 0 bytes
Working set size0 bytes
Directory slots/0
Process ID 10056
Progress rate 3.600% per hour
Executable wrapper_6.1_windows_x86_64.exe

Environment Win 10 NVIDIA GeForce GTX 1060 6GB (4095MB) driver: 472.47

Has anyone else seen this ?

Thanks
Bill F

Yes I updated my MS Visual C++ to the 2015 or newer level and my old NVDIA 1060 took right off !!!

Thanks
Bill F

Anybody get any new acemd3 tasks and notice if the application name changed?

Went looking for acemd3 tasks in the server and see that they changed the name to

Advanced molecular dynamics simulations for GPUs

Yes, I've had one running on a Linux machine for the last 10 hours or so. Still says v2.19 and cuda 1121. It's approaching 40%, so much the same speed as usual for a GTX 1660 Ti

Host 508381. Note that machine has an old errored task from 2020 in the list (v2.10, cuda 100), and that's got the new name too.

That's interesting.
You can track the change on the "applications" page.