Pair RTX-2070

I often see a similar problem.

https://www.gpugrid.net/result.php?resultid=33631003

Also, there workunits usually reach 100% completion for hours before they finish.

Failing for me as well on both of my systems

Linux Mint with Nvidia A4000
Windows 11 with RTX 4090

working fine on all my systems (linux + various nvidia). like a 7% error rate, which is in line with previous batches.
____________

Never have been able to decipher most of the errors on Windows hosts.

But your error on your Linux-A4000 host was simply that you interrupted the task during its run.

ATMbeta tasks can't be interrupted, paused or restarted once they have begun or the task will error out.

If you let the tasks run without any stoppage, they will complete just fine and validate.

the windows issue might be a permissions issue or something. all Ryan's windows tasks end with "cmd.exe exited" which might be due to boinc lacking permissions to call the local system's cmd.exe?

try running BOINC as Administrator. just a guess. maybe that's not it.

for his linux task. he might not have interrupted it intentionally. looks like the app (or the wrapper) hit a segfault for some reason after running for about 4 mins. then it restarted, and of course it will fail after a restart as they always do.
____________

the windows issue might be a permissions issue or something. all Ryan's windows tasks end with "cmd.exe exited" which might be due to boinc lacking permissions to call the local system's cmd.exe?

try running BOINC as Administrator. just a guess. maybe that's not it.

for his linux task. he might not have interrupted it intentionally. looks like the app (or the wrapper) hit a segfault for some reason after running for about 4 mins. then it restarted, and of course it will fail after a restart as they always do.

the windows issue might be a permissions issue or something. all Ryan's windows tasks end with "cmd.exe exited" which might be due to boinc lacking permissions to call the local system's cmd.exe?

try running BOINC as Administrator. just a guess. maybe that's not it.

for his linux task. he might not have interrupted it intentionally. looks like the app (or the wrapper) hit a segfault for some reason after running for about 4 mins. then it restarted, and of course it will fail after a restart as they always do.

Just had another one fail on the Linux box, the machine has not been touched since it was downloaded to me seeing it sat there with a computation error, it failed in 42s.

I'm seeing the same problem with ATMbeta. They error out in the first 5 minutes then go to 100% then take a long time to error out.

I just upgraded to an RTX 4080 from my GTX 980ti as I was getting a lot of failures from Einstein tasks. That has stopped but the only GPU tasks erroring are from this project.

I'm seeing the same problem with ATMbeta. They error out in the first 5 minutes then go to 100% then take a long time to error out.

I just upgraded to an RTX 4080 from my GTX 980ti as I was getting a lot of failures from Einstein tasks. That has stopped but the only GPU tasks erroring are from this project.

Yes becasue it's STILL running at 100% over an hour. It was supposed to run 28days then went to 100% in 5 minues and now is still running 1h22m at 100% complete and has not terminated. The task manager is still showing high utilization on the task.

This behavior has been known and discussed for like 8 months now.

The first segments of these tasks will show normal progress. Tasks with “0-5” or “0-10” in the filename. Subsequent segment tasks with “1-5” or “2-5” or “1-10” or “2-10” etc will all jump to 100% immediately due to a bug in how the application reports progress. If you leave it alone and don’t touch it, it will complete successfully.

The tasks you have on your host now are indeed this type with both being listed as “2-10” units. This is normal/expected. And you can expect the same with the rest of the batch all the way to the 9-10s. Just leave them be and do their thing.

It’s not an error unless BOINC says “Computation Error”, which it hasn’t.
____________

Looks like units are completing fine now.

You spoke too soon:

https://www.gpugrid.net/workunit.php?wuid=27605492

You spoke too soon:

https://www.gpugrid.net/workunit.php?wuid=27605492

Thanks for the background. I would however reconsider the term "well known" -- it's clear that it's not to the casual participant.

Has anyone considered or tabulated the wasted power, time, and corresponding CO2 emissions on these broken WUs? Wasting resources on crypto is one thing but we shouldn't be wasting volunteer's resources. These add up.

Another just hit:
https://www.gpugrid.net/workunit.php?wuid=27606226

If you are concerned, they you can stop your consternation by simply removing this project.

Done.

Thanks!

Still failing for me, blocked the 4090 machine for now, anyone on 40xx series get them working? if so what magic are you using?

Still failing for me, blocked the 4090 machine for now, anyone on 40xx series get them working? if so what magic are you using?

Yea tried running as admin, same issue, switching this machine to Linux is not an option.

do you have AV on the system? is it possible that it's blocking some activity from the app? like preventing the download of the extra packages it needs. it seems to be failing sometime between downloading the extra packages and running the app.

try disabling your AV, or whitelisting the BOINC data directory.
____________

This is a dead tasks stdeerr output (Currently 2 hours and shows 100%) and since I am going to bed for 8 hrs I am aborting this. Its only my 3rd and I have a 4th like it.

23:28:42 (16788): wrapper (7.9.26016): starting
23:28:42 (16788): wrapper: running python.exe (bin/conda-unpack)
23:28:44 (16788): python.exe exited; CPU time 0.000000
23:28:44 (16788): wrapper: running Library/usr/bin/tar.exe (xjvf input.tar.bz2)
run.bat
run.sh
tnks2_m5b_m5c_0.xml
tnks2_m5b_m5c_asyncre.cntl
tnks2_m5b_m5c.inpcrd
tnks2_m5b_m5c.prmtop
23:28:45 (16788): Library/usr/bin/tar.exe exited; CPU time 0.031250
23:28:45 (16788): wrapper: running C:/Windows/system32/cmd.exe (/c call run.bat)
Running command git clone --filter=blob:none --quiet https://github.com/raimis/AToM-OpenMM.git 'D:\data\slots\13\tmp\pip-req-build-y0gn8rc9'
Running command git rev-parse -q --verify 'sha^d7931b9a6217232d481731f7589d64b100a514ac'
Running command git fetch -q https://github.com/raimis/AToM-OpenMM.git d7931b9a6217232d481731f7589d64b100a514ac
Running command git checkout -q d7931b9a6217232d481731f7589d64b100a514ac
Warning: importing 'simtk.openmm' is deprecated. Import 'openmm' instead.


This is completed one: ( At quick glance in the opening commands I can see no difference)

<core_client_version>7.24.1</core_client_version>
<![CDATA[
<stderr_txt>
19:06:49 (13988): wrapper (7.9.26016): starting
19:06:49 (13988): wrapper: running python.exe (bin/conda-unpack)
19:06:52 (13988): python.exe exited; CPU time 0.000000
19:06:52 (13988): wrapper: running Library/usr/bin/tar.exe (xjvf input.tar.bz2)
run.bat
run.sh
tnks2_m1b_m5h_0.xml
tnks2_m1b_m5h_asyncre.cntl
tnks2_m1b_m5h.inpcrd
tnks2_m1b_m5h.prmtop
19:06:53 (13988): Library/usr/bin/tar.exe exited; CPU time 0.015625
19:06:53 (13988): wrapper: running C:/Windows/system32/cmd.exe (/c call run.bat)
Running command git clone --filter=blob:none --quiet https://github.com/raimis/AToM-OpenMM.git 'D:\data\slots\15\tmp\pip-req-build-o6pc0y1a'
Running command git rev-parse -q --verify 'sha^d7931b9a6217232d481731f7589d64b100a514ac'
Running command git fetch -q https://github.com/raimis/AToM-OpenMM.git d7931b9a6217232d481731f7589d64b100a514ac
Running command git checkout -q d7931b9a6217232d481731f7589d64b100a514ac
Warning: importing 'simtk.openmm' is deprecated. Import 'openmm' instead.
tar: run.log: file changed as we read it
03:07:01 (13988): C:/Windows/system32/cmd.exe exited; CPU time 27177.796875
03:07:01 (13988): called boinc_finish(0)
0 bytes in 0 Free Blocks.
310 bytes in 4 Normal Blocks.
1144 bytes in 1 CRT Blocks.
0 bytes in 0 Ignore Blocks.
0 bytes in 0 Client Blocks.
Largest number used: 0 bytes.
Total allocations: 427320182 bytes.
Dumping objects ->
{3078551} normal block at 0x00000261DB821170, 48 bytes long.
Data: <PATH=D:\data\slo> 50 41 54 48 3D 44 3A 5C 64 61 74 61 5C 73 6C 6F
{3078530} normal block at 0x00000261D9D674A0, 139 bytes long.
Data: <<project_prefere> 3C 70 72 6F 6A 65 63 74 5F 70 72 65 66 65 72 65
..\api\boinc_api.cpp(309) : {3078527} normal block at 0x00000261D9D35490, 8 bytes long.
Data: < &#131;&#219;a > 00 00 83 DB 61 02 00 00
{3077868} normal block at 0x00000261D9D67640, 139 bytes long.
Data: <<project_prefere> 3C 70 72 6F 6A 65 63 74 5F 70 72 65 66 65 72 65
{3077233} normal block at 0x00000261D9D34C20, 8 bytes long.
Data: < {&#148;&#219;a > 80 7B 94 DB 61 02 00 00
..\zip\boinc_zip.cpp(122) : {278} normal block at 0x00000261D9D3F160, 260 bytes long.
Data: < > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
{263} normal block at 0x00000261D9D408E0, 16 bytes long.
Data: <&#168; &#212;&#217;a > A8 19 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{262} normal block at 0x00000261D9D410B0, 16 bytes long.
Data: < &#212;&#217;a > 80 19 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{261} normal block at 0x00000261D9D41010, 16 bytes long.
Data: <X &#212;&#217;a > 58 19 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{260} normal block at 0x00000261D9D40E30, 16 bytes long.
Data: <0 &#212;&#217;a > 30 19 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{259} normal block at 0x00000261D9D40CF0, 16 bytes long.
Data: < &#212;&#217;a > 08 19 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{258} normal block at 0x00000261D9D40700, 16 bytes long.
Data: <&#224; &#212;&#217;a > E0 18 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{257} normal block at 0x00000261D9D393D0, 32 bytes long.
Data: <CUDA_DEVICE=0 PU> 43 55 44 41 5F 44 45 56 49 43 45 3D 30 00 50 55
{256} normal block at 0x00000261D9D407A0, 16 bytes long.
Data: < &#162;&#211;&#217;a > 10 A2 D3 D9 61 02 00 00 00 00 00 00 00 00 00 00
{255} normal block at 0x00000261D9D3A210, 40 bytes long.
Data: <&#160; &#212;&#217;a &#208;&#147;&#211;&#217;a > A0 07 D4 D9 61 02 00 00 D0 93 D3 D9 61 02 00 00
{254} normal block at 0x00000261D9D411F0, 16 bytes long.
Data: <&#192; &#212;&#217;a > C0 18 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{253} normal block at 0x00000261D9D406B0, 16 bytes long.
Data: < &#212;&#217;a > 98 18 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{252} normal block at 0x00000261D9D38AD0, 32 bytes long.
Data: <C:/Windows/syste> 43 3A 2F 57 69 6E 64 6F 77 73 2F 73 79 73 74 65
{251} normal block at 0x00000261D9D41060, 16 bytes long.
Data: <p &#212;&#217;a > 70 18 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{250} normal block at 0x00000261D9D391F0, 32 bytes long.
Data: <xjvf input.tar.b> 78 6A 76 66 20 69 6E 70 75 74 2E 74 61 72 2E 62
{249} normal block at 0x00000261D9D40DE0, 16 bytes long.
Data: <&#184; &#212;&#217;a > B8 17 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{248} normal block at 0x00000261D9D40F70, 16 bytes long.
Data: < &#212;&#217;a > 90 17 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{247} normal block at 0x00000261D9D40F20, 16 bytes long.
Data: <h &#212;&#217;a > 68 17 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{246} normal block at 0x00000261D9D40CA0, 16 bytes long.
Data: <@ &#212;&#217;a > 40 17 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{245} normal block at 0x00000261D9D40C50, 16 bytes long.
Data: < &#212;&#217;a > 18 17 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{244} normal block at 0x00000261D9D40C00, 16 bytes long.
Data: <&#240; &#212;&#217;a > F0 16 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{242} normal block at 0x00000261D9D40BB0, 16 bytes long.
Data: < &#165;&#211;&#217;a > 20 A5 D3 D9 61 02 00 00 00 00 00 00 00 00 00 00
{241} normal block at 0x00000261D9D3A520, 40 bytes long.
Data: <&#176; &#212;&#217;a p &#130;&#219;a > B0 0B D4 D9 61 02 00 00 70 11 82 DB 61 02 00 00
{240} normal block at 0x00000261D9D40B60, 16 bytes long.
Data: <&#208; &#212;&#217;a > D0 16 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{239} normal block at 0x00000261D9D40FC0, 16 bytes long.
Data: <&#168; &#212;&#217;a > A8 16 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{238} normal block at 0x00000261D9D39130, 32 bytes long.
Data: <Library/usr/bin/> 4C 69 62 72 61 72 79 2F 75 73 72 2F 62 69 6E 2F
{237} normal block at 0x00000261D9D411A0, 16 bytes long.
Data: < &#212;&#217;a > 80 16 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{236} normal block at 0x00000261D9D392B0, 32 bytes long.
Data: <bin/conda-unpack> 62 69 6E 2F 63 6F 6E 64 61 2D 75 6E 70 61 63 6B
{235} normal block at 0x00000261D9D40ED0, 16 bytes long.
Data: <&#200; &#212;&#217;a > C8 15 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{234} normal block at 0x00000261D9D40E80, 16 bytes long.
Data: <&#160; &#212;&#217;a > A0 15 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{233} normal block at 0x00000261D9D40B10, 16 bytes long.
Data: <x &#212;&#217;a > 78 15 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{232} normal block at 0x00000261D9D40A20, 16 bytes long.
Data: <P &#212;&#217;a > 50 15 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{231} normal block at 0x00000261D9D40610, 16 bytes long.
Data: <( &#212;&#217;a > 28 15 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{230} normal block at 0x00000261D9D40570, 16 bytes long.
Data: < &#212;&#217;a > 00 15 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{229} normal block at 0x00000261D9D41150, 16 bytes long.
Data: <&#224; &#212;&#217;a > E0 14 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{228} normal block at 0x00000261D9D40520, 16 bytes long.
Data: <&#184; &#212;&#217;a > B8 14 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{227} normal block at 0x00000261D9D40980, 16 bytes long.
Data: < &#212;&#217;a > 90 14 D4 D9 61 02 00 00 00 00 00 00 00 00 00 00
{226} normal block at 0x00000261D9D41490, 1488 bytes long.
Data: < &#212;&#217;a python.e> 80 09 D4 D9 61 02 00 00 70 79 74 68 6F 6E 2E 65
{90} normal block at 0x00000261D9D39070, 32 bytes long.
Data: <windows_x86_64__> 77 69 6E 64 6F 77 73 5F 78 38 36 5F 36 34 5F 5F
{89} normal block at 0x00000261D9D34900, 16 bytes long.
Data: <&#192;&#167;&#211;&#217;a > C0 A7 D3 D9 61 02 00 00 00 00 00 00 00 00 00 00
{88} normal block at 0x00000261D9D3A7C0, 40 bytes long.
Data: < I&#211;&#217;a p &#211;&#217;a > 00 49 D3 D9 61 02 00 00 70 90 D3 D9 61 02 00 00
{67} normal block at 0x00000261D9D34AE0, 16 bytes long.
Data: < &#234;,&#227;&#247; > 80 EA 2C E3 F7 7F 00 00 00 00 00 00 00 00 00 00
{66} normal block at 0x00000261D9D34A90, 16 bytes long.
Data: <@&#233;,&#227;&#247; > 40 E9 2C E3 F7 7F 00 00 00 00 00 00 00 00 00 00
{65} normal block at 0x00000261D9D35170, 16 bytes long.
Data: <&#248;W)&#227;&#247; > F8 57 29 E3 F7 7F 00 00 00 00 00 00 00 00 00 00
{64} normal block at 0x00000261D9D348B0, 16 bytes long.
Data: <&#216;W)&#227;&#247; > D8 57 29 E3 F7 7F 00 00 00 00 00 00 00 00 00 00
{63} normal block at 0x00000261D9D35350, 16 bytes long.
Data: <P )&#227;&#247; > 50 04 29 E3 F7 7F 00 00 00 00 00 00 00 00 00 00
{62} normal block at 0x00000261D9D34EA0, 16 bytes long.
Data: <0 )&#227;&#247; > 30 04 29 E3 F7 7F 00 00 00 00 00 00 00 00 00 00
{61} normal block at 0x00000261D9D352B0, 16 bytes long.
Data: <&#224; )&#227;&#247; > E0 02 29 E3 F7 7F 00 00 00 00 00 00 00 00 00 00
{60} normal block at 0x00000261D9D34A40, 16 bytes long.
Data: < )&#227;&#247; > 10 04 29 E3 F7 7F 00 00 00 00 00 00 00 00 00 00
{59} normal block at 0x00000261D9D353F0, 16 bytes long.
Data: <p )&#227;&#247; > 70 04 29 E3 F7 7F 00 00 00 00 00 00 00 00 00 00
{58} normal block at 0x00000261D9D35260, 16 bytes long.
Data: < &#192;'&#227;&#247; > 18 C0 27 E3 F7 7F 00 00 00 00 00 00 00 00 00 00
Object dump complete.

</stderr_txt>
]]>

its not dead. it's just not finished.

please read the lengthy ATM post in the News forum. it's known about all task segments except the first exhibiing this behavior of staying pegged to 100%. it's fine, just let it finish
____________

its not dead. it's just not finished.

please read the lengthy ATM post in the News forum. it's known about all task segments except the first exhibiing this behavior of staying pegged to 100%. it's fine, just let it finish

Thought I would try again, still failing, did anyone ever come up with a fix to get these running on 40 series cards?

failed unit https://www.gpugrid.net/result.php?resultid=33751449

use Linux seems to be the best way to run them.
____________

Thought I would try again, still failing, did anyone ever come up with a fix to get these running on 40 series cards?

failed unit https://www.gpugrid.net/result.php?resultid=33751449

I have a 5950X setup with a perfectly running 2080 Super and 3 tasks I got today (so far) are all failing...

8-)

GPUGRID 1.09 ATMbeta: Free energy calculations of protein-ligand binding (cuda1121) JNK1_m29_m27_3-QUICO_ATM_500K_dih14fit-2-5-RND8000_0 00:00:32 (00:00:00) 1/25/2024 1:16:36 PM 1/25/2024 1:19:36 PM 0.989C + 1NV 0.0 Reported: Computation error (195,) Win11-Main

I have a 5950X setup with a perfectly running 2080 Super and 3 tasks I got today (so far) are all failing...

8-)

GPUGRID 1.09 ATMbeta: Free energy calculations of protein-ligand binding (cuda1121) JNK1_m29_m27_3-QUICO_ATM_500K_dih14fit-2-5-RND8000_0 00:00:32 (00:00:00) 1/25/2024 1:16:36 PM 1/25/2024 1:19:36 PM 0.989C + 1NV 0.0 Reported: Computation error (195,) Win11-Main

Ohhh! So those were stray OLD tasks..

Gotchya!

Thanks!

8-)

Cheers, ill leave it enabled then and see what happens

I'm noticing an interesting aside wrt ATM WUs. All mine seem to be erroring out atm, but drilling into the WUs, it isn't just my computer. As an example

https://www.gpugrid.net/workunit.php?wuid=27653498

Now the odd thing is that all the other computers seem to have an Intel processor (when I inspect each computer showing the failed result), and going down the list there seems to be a fail. But then 1 result is returned successfully, which has an AMD processor. Now I know the bulk of the processing is on the GPU, but there is some CPU processing as I understand. Perhaps there's something in the code which is playing nasty with Intel's implementation of something? Thought I'd pass this observation along, it being beta and all...
____________

The one that succeeded was because it was on Linux, not because it was on AMD.


____________

On this last batch of ATMbeta, I've noticed an increase of tasks failing with ValueError: Energy is NaN
Currently about 1/3 ratio of failures.
Since January 22th, 64 valid tasks, and 20 failed with that error at my Linux hosts.
14 of the failed tasks exceeded 1 hour processing time before erroring, and in these, 9 of them exceeding 3 hours processing time.
Anyway, I still consider it as a reasonable ratio for these well-rewarded beta tasks, and I keep my hope of being contributing to Science.

On this last batch of ATMbeta, I've noticed an increase of tasks failing with ValueError: Energy is NaN
Currently about 1/3 ratio of failures.
Since January 22th, 64 valid tasks, and 20 failed with that error at my Linux hosts.
14 of the failed tasks exceeded 1 hour processing time before erroring, and in these, 9 of them exceeding 3 hours processing time.
Anyway, I still consider it as a reasonable ratio for these well-rewarded beta tasks, and I keep my hope of being contributing to Science.