I just finished crunching a task for this new application successfully.

https://www.gpugrid.net/result.php?resultid=35379717

What exactly are we crunching here?

By the name of the app, somehow uses machine learning.

I just finished crunching a task for this new application successfully.

Thanks for the progress update and explanation of just what kind of ML is being used for the ATM tasks, Steve.


I see also you released a new beta ATM app yesterday to go along with the ATMML app. Already did one of those today.

Is it Windows, Linux or both?

You can verify OS compatibility for different applications at GPUGRID apps page.

I noticed that this batch of ATMML units takes almost 3 times longer than the previous batches to complete. One of them, I suspended and when I restarted it, it would not start, I kept "running" it for over an hour, and no progress, so I had no option, but to abort it.

effectivement elles sont tres longue a calculer.Je vais les arreter aussi.
9h20 sur ma rtx 4060 et 14h20 sur rtx a2000.

They are very long to calculate. I will stop them too.
9h20 on my rtx 4060 and 14h20 on rtx a2000.
____________

j ai annulé les 4 taches ATMML que j'avais car trop longues a calculer.
entre 16 et 24 heures.MESSIEURS LES PROGRAMMEURS,j'espere que vous allez vous pencher sur ce probleme?

I cancelled the 4 ATMML stains I had because too long to calculate.
between 16 and 24 hours.PROGRAMMERS, I hope you will look into this problem?
____________

Didn't have any issues with the new ATMML tasks I received. Rescued one at "the last chance saloon" as the _7 wingman.

Don't seem to have a "unreasonable" crunch time for the hardware used. About 7 hours or so. I've had acemd that went for 12-14 hours before.

I don't recall a larger executable from a BOINC project. 4.67 GB! That is larger than some LHC VDI files.

I had 57 units so far without a single error. Great!
Fastest unit took 4,197 seconds (1,17 hours) on a 4080 Super, longest one took a bit over 30,000 seconds (8,33 hours) on a 4060ti.
More than reasonable.

Hello everyone! My four hosts running 3060, 3060ti and 3070ti were not able to complete a single unit so far. They all fail at the very beginning with the following STDERR output: "Error loading cuda module". I am running Linux Mint and Ubuntu with Nvidida driver 470. The newer drivers produce errors in other projects so I decided to stick to that driver version. I noticed that a lot of my wingmen successfully crunch the units with driver 530 or 535. is that a driver issue? All other projects run just fine on version 470.


Warning: importing 'simtk.openmm' is deprecated. Import 'openmm' instead.
Traceback (most recent call last):
File "/var/lib/boinc-client/slots/24/bin/rbfe_explicit_sync.py", line 10, in <module>
rx.setupJob()
File "/var/lib/boinc-client/slots/24/lib/python3.11/site-packages/sync/atm.py", line 85, in setupJob
self.worker = OMMWorkerATM(ommsystem, self.config, self.logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/boinc-client/slots/24/lib/python3.11/site-packages/sync/worker.py", line 34, in __init__
self.simulation = Simulation(self.topology, self.ommsystem.system, self.integrator, platform, properties)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/boinc-client/slots/24/lib/python3.11/site-packages/openmm/app/simulation.py", line 106, in __init__
self.context = mm.Context(self.system, self.integrator, platform, platformProperties)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/boinc-client/slots/24/lib/python3.11/site-packages/openmm/openmm.py", line 12171, in __init__
_openmm.Context_swiginit(self, _openmm.new_Context(*args))
^^^^^^^^^^^^^^^^^^^^^^^^^^
openmm.OpenMMException: Error loading CUDA module: CUDA_ERROR_UNSUPPORTED_PTX_VERSION (222)

with that error, yes i would assume the old driver version is the issue.

CUDA historically has not been forward compatible. as in, a CUDA10 binary could not run on a system with only CUDA 8 drivers. but the opposite was true in most cases, that backward compatibility is fine and you can run even very old CUDA code with the latest drivers.

only starting with CUDA 11.1 was forward compatibility introduced, and only within the same major version. So a system with only CUDA 11.1 drivers could still run up to CUDA 11.8 binaries. Same goes for CUDA12, where all CUDA 12 drivers will be compatible with all CUDA 12+ binaries.

I have a feeling that some parts of this new ATMML app, and probably in particular OpenMM (based on what's throwing the error) actually requires CUDA 12+ drivers. and the app is misidentified at the project as being CUDA 11 compatible.

you could test this by installing the newer drivers and see if they then run.

what other project has issue with the newer drivers?
____________

chez moi les pilotes d'origine du system fonctionne tres bien.ce sont les pilotes 535 fourni a l'install de linux mint..

https://www.gpugrid.net/results.php?userid=563937

at me the original drivers of the system works three good.this are the 535 drivers provided to install linux mint..
____________

chez moi les pilotes d'origine du system fonctionne tres bien.ce sont les pilotes 535 fourni a l'install de linux mint..

https://www.gpugrid.net/results.php?userid=563937

at me the original drivers of the system works three good.this are the 535 drivers provided to install linux mint..

chez moi les pilotes d'origine du system fonctionne tres bien.ce sont les pilotes 535 fourni a l'install de linux mint..

https://www.gpugrid.net/results.php?userid=563937

at me the original drivers of the system works three good.this are the 535 drivers provided to install linux mint..

I tried to install the 535 driver but after that my GPU is no longer recognised by Amicable, Einstein and Asteroids. GPUgrid lets me start new wus but they fail after 43 seconds saying that no Nvidia GPU was found. Do I have to install additional libraries or something like that? I also noticed that there is an open driver package from Nvidia and a regualar meta package and a server version of that driver. Which one are you guys using?

je n'utilise rien de supplemntaire comme package.
J'ai installé linux mint normalement et fais les mises a jours systeme et pilotes.
J'ai installé les pilotes 535 en passant par le gestionnaire de pilotes at tout fonctionne tres bien.
boinc reconnait ma rtx 4060 et ma rtx a2000 et ma gtx 1650 dans le meme pc.
je calcule pour gpugrid et amicable numbers sans problemes.
soit vous avez une installation systeme défaillante soit un probleme hardware.

I don’t use anything extra as a package.
I installed linux mint normally and make the system and driver updates.
I installed the 535 drivers through the driver manager and everything works fine.
boinc recognizes my rtx 4060 and my rtx a2000 and my gtx 1650 in the same pc.
I calculate for gpugrid and friendly numbers without problems.
either you have a system installation failure or a hardware problem.
____________

pour commencer,je vous conseille de tester vos barrettes de ram avec memtest free et pas un autre programme.il fonctionne tres bien et est fiable.

To start with, I advise you to test your ram strips with memtest free and not another program.it works very well and is reliable.

https://www.memtest86.com/
____________

quand vous installer de driver 535 sous linux il installe aussi tout le nécessaire pour calculer.
c'est a dire opencl nvidia et cuda donc il ne faut rien installer d'autres .
Juste le driver 535 et c'est tout

when you install driver 535 under linux it also installs everything necessary to calculate.
that is opencl nvidia and cuda so nothing else to install .
Just driver 535 and that’s it
____________

je vous conseille aussi de faire un test de votre dur pour voir s'il ny a pas de cluster defectueux.
il faut faire un test de surface avec hdtune 255 free ou un autre programme genre crystaldiskinfo.


https://www.hdtune.com/download.html

I also advise you to do a test of your hard to see if there is no bad cluster.
you have to do a surface test with hdtune 255 free or another program like crystaldiskinfo.

____________

pour dépanner un pc je commence toujours par ces 2 choses.

to troubleshoot a pc I always start with these 2 things
____________

it may be true for Mint that opencl components are installed with the normal driver package. but that is not the case for Ubuntu, and you do need to install the opencl components separately with the command in my previous post.
____________

ok, I managed to get one of my UBUNTU hosts running with the 535 driver and the additional OpenCL libraries installed like you said Ian&Steve. For 5h it has been crunching an ATMML unit so far, it's looking good! I am surpised to see that you seemingly need OpenCL to run CUDA code because that is the only difference to my previous attempts.

So far no luck with my Mint Laptop. If I change the driver to any other version than 470 the GPU is no longer detected. That may also have to do with the AMD driver from the on-board AMD GPU. Maybe they interfere with each other. Will try my other hosts this weekend. Thanks for the help.

you don't need opencl driver components to run true cuda code. but a lot of other projects are not running apps compiled in cuda, but rather OpenCL.
____________

...running 3060, 3060ti and 3070ti were not able to complete a single unit so far.

Until yesterday I received the ATMML wus and my hosts crunched them successfully with driver version 535. Today I am not getting any new ones. The servers says there were no available but on the server main page there are always around 300 available for download. And that number fluctuates so others get them. Does anybody else have that problem?

Does anybody else have that problem?

Huh... Do you know how much vram they need? My GPUs are limited to 8 GB.

Huh... Do you know how much vram they need? My GPUs are limited to 8 GB.

Is it Windows, Linux or both?

still waiting and wanting for some Windows tasks
____________

I tried to install the 535 driver but after that my GPU is no longer recognised by Amicable, Einstein and Asteroids. GPUgrid lets me start new wus but they fail after 43 seconds saying that no Nvidia GPU was found.

These tasks are running fine for me so far. The only thing I noticed was that the points awarded seem off. The second task I completed took an hour longer than the first but received 900k fewer points. So far they have been taking 5-6 hours on a 3080ti, driver version 535.183.01.

It is lower credit because it took longer from receive to return result

19 Aug 2024 | 19:41:09 UTC 21 Aug 2024 | 20:35:33 UTC
This is time and it is more then 24 hours and therefor it got reduced in points.

These tasks are running fine for me so far. The only thing I noticed was that the points awarded seem off. The second task I completed took an hour longer than the first but received 900k fewer points. So far they have been taking 5-6 hours on a 3080ti, driver version 535.183.01.

This task has been "downloading" for almost 4 hrs, has 0% completion and estimated run time of 400+ days.

Looks like it needs killing?

Running on Win 11, high-end machine.

Richard

==============================

Application
ATMML: Free energy with neural networks 1.01 (cuda1121)

Name TYK2_A02_A09_r0_2-QUICO_ATM_AF_04_Benchmark-12-20-RND2746
State Downloading
Received 2024-08-22 7:40:17 AM
Report deadline 2024-08-27 7:40:19 AM
Resources 0.996 CPUs + 1 NVIDIA GPU
Estimated computation size 1,000,000,000 GFLOPs
Executable wrapper_6.1_windows_x86_64.exe

Patience . . . . grasshopper. This is a new app for Windows hosts so you are competing for download bandwidth with the cohort of other Windows hosts. Which are many.

The task runtime estimation is not accurate until your host has returned 11 valid tasks of that type to develop a correct and accurate APR rate.

Only then will the estimated runtimes be correct.

Looks like it finally started and ran for a few minutes, then uploaded...

Richard

These tasks are running fine for me so far. The only thing I noticed was that the points awarded seem off. The second task I completed took an hour longer than the first but received 900k fewer points. So far they have been taking 5-6 hours on a 3080ti, driver version 535.183.01.

The points are accurate. You get a 50% bonus, if you finish the task successfully and return the results within 24 hours from downloading it. There is a 25% bonus if you do it within 48 hours. No bonus if you return it after 48 hours. This is an incentive for quick return of results.

Are there no checkpoints on ATMML tasks?

I was about 30% complete when I had to suspend the task and shut down the computer. When I restarted both the % done and elapsed time were zero.

Are there no checkpoints on ATMML tasks?

I was about 30% complete when I had to suspend the task and shut down the computer. When I restarted both the % done and elapsed time were zero.

I hope this doesn't backfire. This morning I see 800 tasks in progress, but zero ready to send.

My last two downloads have been replica _3 tasks, each WU having failed on three Windows machines first.

I do hope new Windows users pay attention to the 'tricks of the trade' we've learned over the years:

• small cache, especially with slower GPUs.
  
• run continuously, don't allow interruptions (especially auto-updates)
  
• don't swap to a different GPU type mid-run
  

I do hope new Windows users pay attention to the 'tricks of the trade' we've learned over the years:

My last two downloads have been replica _3 tasks, each WU having failed on three Windows machines first.

ReL the Apps Page: https://www.gpugrid.net/apps.php

I wish, for consistency, it would state:

ATMML: Free energy with neural networks for GPU

Also, when selecting projects in project preferences, it would be nice if it stated:

ATMML on GPU

ReL the Apps Page: https://www.gpugrid.net/apps.php

I wish, for consistency, it would state:

ATMML: Free energy with neural networks for GPU

Also, when selecting projects in project preferences, it would be nice if it stated:

ATMML on GPU

Despite this, there is a noticeable increase in the number of users returning ATMML results.

* small cache, especially with slower GPUs.

I just started ATMML yesterday. Out of seven starts only one completed. The rest errored-out after 1-1.5 hours. Windows11/RTX4090.

I'd like to get some actual work done...

* small cache, especially with slower GPUs.

Which cache? Where is it set?? What should it be set at???

The Windows tasks ARE NOT working as advertised....

On two 3090ti computers and one 3090 11 work units have error out between 2-4 hours of run time.

Previous successful task run times went between 17000-18500 seconds.

Errored tasks are 5000-8500 seconds.

I am killing the ap in preferences until itself out....
____________

Thanks. There are too many cache's out there. Let's call this the work queue.

The Windows tasks ARE NOT working as advertised....

On two 3090ti computers and one 3090 11 work units have error out between 2-4 hours of run time.

Previous successful task run times went between 17000-18500 seconds.

Errored tasks are 5000-8500 seconds.

I am killing the ap in preferences until itself out....

Exit status 197 (0xc5) EXIT_TIME_LIMIT_EXCEEDED

Does anyone know where the denominator comes from in this line?:

<message>
exceeded elapsed time limit 5454.20 (10000000000.00G/1712015.37G)</message>
<stderr_txt>

The numerator I believe is the fpops_bound that is set in the WU template which is controlled by us.

I'm curious why do we even bother to intentionally error out a task based on runtime at all? Usually a wrong estimate of runtime just messes with local client scheduling a bit, but tasks finish fine eventually. It's not like GPUGrid had accurate runtime estimation before, but previous tasks didn't fail.

Does this batch/app has bug that could cause it to stuck computing forever, which is why we need an additional protection to abort tasks after certain runtime?

Probably the decision is because this project depends on fast turnaround and turnover for tasks.

Science can't proceed till the earlier result is returned, validated and then iterated into the next task.

Better to fail fast and send out the next wingman task until the task gets retired at 8 fails.

Deadline is always 5 days to get through 8 tries and why they reward 50% credit bonus for returning results within 24 hours.

I've seen people misuse this "fail fast" philosophy very often. "Fail fast" makes sense only when it's going to be a failure anyway. Turning a successful result into a failure proactively is the opposite of making progress.

Look at the errors on this host. It took ~20K seconds for my host to finish, but all the prematurely killed results ended up wasting way more compute time and on average suffered another half a day delay before getting a successful return. That's not speeding up but slowing down.

That's why I'm curious what this limit is trying to protect against. Only if we know there is a chance that a task can stuck computing indefinitely, would such a limit make sense. That would generally indicate some bug needs to be fixed. Even then, given how long turnaround would be after killing an otherwise successful task, the project should have set a floor of the limit to a few hours.

In addition, "science" does not equal to this project alone. Wasting hours of compute that could have been used by other projects isn't advancing "science". It's advancing this project at the cost of other science. That would be a bit disrespectful to fellow scientists if it's done intentionally. However, I'd rather assume good intention here that this is just a misguided optimization. I know software isn't easy, so project owner should take a look at the resulting data and try to make more efficient use of available compute by reducing waste, which would also speed up the progress for this project.

I'm pretty sure the "exceeded elapsed time limit" is not because the project scientists just decided on a whim to utilize it.

It's part of the Boinc code and nothing they have control over. It's present for all projects that use the Boinc code unmodified.

Only the Boinc developers have the knowledge of how that function is implemented.

The project scientist already stated he was surprised by the errors when the exact same task template was used for the Linux tasks and they have not had any issues with elapsed time limit errors.

Something specific to Windows. And they do not develop for Windows firstly being all the tools and software they use is primarily first Linux based and where their expertise is greatest.

Some of the toolchains they use have never had Windows versions which is why it has taken so long for some Windows versions of the native Linux apps.

I agree - the runtime errors are an issue mainly of the BOINC software, but they are appearing because the GPUGrid teams - admin and research - have over the years failed to fully come to terms with the changes introduced by BOINC around 2010. We are running a very old copy of the BOINC server code here, which include the beginnings of the 2010 changes, but which makes it very difficult for us to dig our way out of the hole we're in.

But I don't agree that only the BOINC developers understand the code. It's all open-source, and other projects mange to control it reasonably well. The finer points are indeed cloaked in obscure language, but the resulting data is visible on all our machines.

Let's play with a current worked excample.

I've just downloaded a new ATMML task on host 508381. That's a Linux machine, with 2 GPUs. They are in fact identical, so for once the 'Coprocessors' line is true. It has completed 52 ATMML tasks so far, so it has had plenty of time to reach a steady state. [BOINC loves steady states - it's the edge cases, like deploying a new app_version, which cause the problems]

My key objective is to see how the runtime estimate was derived, and to see what was done well, and what was done badly. BOINC works out the runtime from the size and the speed of the task. In dimensional terms, that's

"transient upload error: server out of disk space"

the old problem which has been occurring over the years :-(
Unbelievable that this is still happening.

So this runtime exceeded failure is actually related to the absurd rsc_fpops_est setting. This number is obviously not accurate. Could the project fix the number to be more reasonable, instead of relying on client's trial and error for adjustment that wastes lots of compute power?

So this runtime exceeded failure is actually related to the absurd rsc_fpops_est setting. This number is obviously not accurate. Could the project fix the number to be more reasonable, instead of relying on client's trial and error for adjustment that wastes lots of compute power?

Yes, I found host 591089. That succeeded on its first task, but then failed five in a row on the time limit.

It's had the current one for two days now, so hopefully it'll work. One to watch.

I found a way to get the windows WUs finish without errors.
after you get a WU, go to Projects tab and select no new tasks. be sure you don't have other projects running at the same time. once is finished and uploaded you can allow for another WU and repeat.

WUs starting with "MCL1" are all erroring out in windows or linux.

Quite a few of my tasks starting with PTP1B are failing in both OSs

Quite a few of my tasks starting with PTP1B are failing in both OSs

Same issue here:

https://www.gpugrid.net/result.php?resultid=35798577



Tue 27 Aug 2024 08:34:58 PM EDT | GPUGRID | Computation for task MCL1_A23_A35_r0_4-QUICO_ATM_AF_04_Benchmark_MCL1-0-20-RND1911_1 finished
Tue 27 Aug 2024 08:34:58 PM EDT | GPUGRID | Output file MCL1_A23_A35_r0_4-QUICO_ATM_AF_04_Benchmark_MCL1-0-20-RND1911_1_0 for task MCL1_A23_A35_r0_4-QUICO_ATM_AF_04_Benchmark_MCL1-0-20-RND1911_1 absent
Tue 27 Aug 2024 08:34:59 PM EDT | GPUGRID | Started upload of MCL1_A23_A35_r0_4-QUICO_ATM_AF_04_Benchmark_MCL1-0-20-RND1911_1_1

That is a bad batch. All units error out after several weeks of trouble-free calculation under Linux. Example:
https://www.gpugrid.net/result.php?resultid=35800268

Warning: importing 'simtk.openmm' is deprecated. Import 'openmm' instead.
[W output_modules.py:45] Warning: CUDA graph capture will lock the batch to the current number of samples (2). Changing this will result in a crash (function )
[W output_modules.py:45] Warning: CUDA graph capture will lock the batch to the current number of samples (2). Changing this will result in a crash (function )
Traceback (most recent call last):
File "/var/lib/boinc-client/slots/28/bin/rbfe_explicit_sync.py", line 11, in <module>
rx.scheduleJobs()
File "/var/lib/boinc-client/slots/28/lib/python3.11/site-packages/sync/atm.py", line 142, in scheduleJobs
if isample % int(self.config['CHECKPOINT_FREQUENCY']) == 0 or isample == num_samples:
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/boinc-client/slots/28/lib/python3.11/site-packages/configobj/__init__.py", line 554, in __getitem__
val = dict.__getitem__(self, key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'CHECKPOINT_FREQUENCY'

Yes, I had 90 failed ATMML tasks overnight. The earliest was issued just after 18:00 UTC yesterday, but was created at 27 Aug 2024 | 13:28:15 UTC.

I've switched to helping with the quantum chemistry backlog for the time being.

I experienced the same problems over several days when I was suspending GPU processing because of very hot temps in Texas; the result was loss of many hours of processing until I discovered the LTIWS(leave tasks in memory while suspended) was apparently not working. I am suspending ATMML tasks until cooler weather arrives in the fall.BET

I can't remember all of the task types that allow suspending or exiting Boinc without erroring out.

The acemd tasks properly checkpoint, but you also can't allow a restarted WU to start again on a different gpu or it will also error out.

Best practice for GPUGrid generally has always been to let all tasks run to completion before exiting Boinc. No guarantees that any task will resume without loss of prior work done or just error out.

LTI[M]WS only applies to CPU tasks. GPUs don't have that much spare memory.

I appreciate the informative responses of BOTH Keith and Richard immediately below!!

Generally, if I know I must reboot the system shortly in the future I will just wait till the current tasks are finished or reboot shortly after a new task starts so I won't begrudge the little time lost it has already spent crunching and which it will have have restart again after the reboot.

It is generally safe to stop a task soon after it starts because with the exception of the acemd tasks, all the rest of the task types need several minutes to unpack the python environment in the slots and actually hasn't started calculating anything yet

You can get away with interrupting the startup process with a reboot I have found and you won't throw away the task or error it out.

The time bonus system has been in place w/ GPUGrid for years. (And yes, the GG tasks download several GIG of data... [WHY? well, another issue] and the download time does count against the deadline)
BUT the points awarded are nonetheless - shall one say - unfathomable.
Case in point: ATMML has a very, very high failure rate [yet another issue, AND an important one], and when completed usually award 300,000 points, at least to my NVIDIA which is better in some ways than this guy's... HOWEVER, host 621740 has had seven successful ATMML tasks (see below) in the last six days with EACH being awarded 2,700,000 points .... SO, what gives?? WHY a 9-fold difference???
WUid other task result
29271283 1 error, 1 abort
29270516 3 errors
29265238 1 error, 1 abort
29204796 1 time out, 5 errors (1 of these after 50,905 sec = 14+ Hrs
29268456 n/a
29267692 3 errors
29267146 2 errors

621740's specs:
GenuineIntel
Intel(R) Core(TM) i7 CPU 930 @ 2.80GHz [Family 6 Model 26 Stepping 5]
Number of processors 8
Coprocessors NVIDIA NVIDIA GeForce RTX 3060 (12287MB) driver: 560.81
Operating System Microsoft Windows 10 Professional
Memory 12279.11 MB
Cache 256 KB

My NVIDIA has
Memory 16316.07 MB
Cache 512 KB
PLUS
Swap space 45668.07 MB
Total disk space 464.49 GB

Lasslo P, PhD, Prof Engr.

Good point, but winDoze update does not make it easy to avoid IT's "decision" about updating and the time to restart my system. (Don't you hate it when big tech is so much more brilliant and all-knowing than you?)

I turn OFF updates for the 5 weeks max allowed, then as the month ends, I pick the time when I will download and install OS updates.
Even then, I set the "active hours" to the times LEAST likely my PC is in use, usually including late PM to early AM

Lasslo P, PhD, Prof Engr

you're comparing multiple levels of apples vs oranges.

a GTX1660Ti is in no ways better than a RTX3060, it's older gen, less than half the CUDA cores, no tensor cores (which ATMML will use), slower clock speed, slower memory speed, truly basically every performance metric favors the 3060.

your task you completed for 300,000cr was ACEMD3, not ATMML, and you also need to consider that the ATMML tasks run much longer than ACEMD3 and use more resources, so the higher credit reward is appropriate.

your 1x ACEMD3 task if you were to crunch 24/7 would come up with a production of a little over 1,000,000 points per day.

your competitor also completed one ACEMD3 task recently, and scaling that to 24/7 production comes out to around 1,500,000 points per day.

it takes them about 4x longer to run ATMML. and based on their recent production, including the failure, is about 3,600,000 ppd.

the project admins have resolved a lot of the problems with ATMML, if you want to have better success with this project, and GPUGRID in general, you can consider switching to Linux. otherwise, maybe investigate what's going wrong with your system to cause the failures. looks like a permissions issue to me since your failed tasks have a bunch of Access is denied errors in the WU log. possibly an over zealous AV software. that could be the reason for your download errors also, or just spotty internet
____________

GTX1660Ti is in no ways better

maybe investigate what's going wrong with your system to cause the failures.

that could be the reason for your download errors also

GTX1660Ti is in no ways better

"in no way" is an absolute statement, and is false: my NVIDIA has 33% more memory and double the cache.
But, admittedly it is not in general as powerful

maybe investigate what's going wrong with your system to cause the failures.

how bizarre ... Batting 1000, the GPUGrid tasks which fail on my system have ALSO failed on several, perhaps even 6 or 7, other systems (except when I take a newly issued task, and when I check those they also fail after bombing on my system.) So, if the problem is on my end and not in any way on GPUGrid's end, then there must be dozens and dozens (and dozens) of other systems which apparently need to "investigate what's going wrong" with them...

that could be the reason for your download errors also

I have no "Download errors" except when I abort the download of a task which has already had repeated compute errors.

Anyway, thanks for your feedback

your 1660Ti has half the memory of a 3060.
your 1660Ti also has half the cache of the 3060.
GTX 1660Ti has 6GB
RTX 3060 has 12GB (you can see this in the host details you referenced)
not sure where you're getting your information, but it's wrong or missing context (like comparing to a laptop GPU or something)

your 1660Ti has half the memory of a 3060.
your 1660Ti also has half the cache of the 3060.
GTX 1660Ti has 6GB
RTX 3060 has 12GB (you can see this in the host details you referenced)
not sure where you're getting your information, but it's wrong or missing context (like comparing to a laptop GPU or something)

My information is from GPUGrid's host information,
https://gpugrid.net/show_host_detail.php?hostid=613323 which states 16GB, but this may be unreliable as TechPowerUp GPU-Z does give the NVIDIA site's number of 6GB
My numbers for the cache also come from gpugrid.net/show_host_detail.php as indeed all the memory figurers in my original post so I guess my mistake was trusting gpugrid.net/show_host_detail info.

And no, this is not a laptop, but a 12-core desktop.

windows users to have issues with BOINC projects

AV = Anti Virus software.
____________

Switching back to BOINC software and (specifically) ATMML tasks. I've posted extensively in this thread about the problems of task duration estimation at this project. I've got some new data, which I can't explain.

Last week, I added a new Linux host (host 625407). It's a pretty plain vanilla Intel i5 with a single RTX 3060 - should be fairly average for this project. It's completed 17 tasks so far, with number 18 in progress - around 3 per day. I attached it to a venue with only ATMML tasks allowed.

Given my interest in BOINC's server-side task duration estimation for GPUs, I've been logging the stats. Here's what I've got so far:

I didn't run ATMML but I'm currently running Qchem on Tesla P100 with short run times (averaging somewhere around 12 mins or so per task). I see this similar behavior/pattern when starting a new instance. If I were to guess, your DCF will eventually go down from the last value of 0.7215 to 0.01 after running 100+ tasks and your final estimated run time could be about 1.16 days which is still higher than your average expected run time for your card. However if you run cpu benchmark, then the DCF number will go up from 0.01 to something higher and will take another 100+ tasks for the DCF to go down to 0.01 again but this time the estimated run time will go below 1.16 days.

I didn't take any note but just making observation only, so I could be wrong. My wild guess is that when running gpu only task there is associated cpu % required to run that gpu task and running benchmark will take care of the cpu portion needed for the gpu task. My one cent.

Well, I said that this is going to be difficult ...

My knowledge and understanding comes from already being an active volunteer at SETI@home on 18 Dec 2008, when BOINC was first announced as being able to manage and use CUDA on GPUs for scientific computing - GPUGrid is also mentioned as becoming CUDA-enabled on the same day. We spent the following months and years knocking the rough edges off the initial launch code. I think the names and features I was referring to in my post were introduced in a sort-of relaunch around 2011. That's still a long way back in the memory bank, and that makes it difficult to find precise references in code or documentation.

My understanding is that the system was designed to be as easy as possible for researchers to implement. I believe the only key information required is the rsc_fpops_est - the estimated size of the task. From your comments on the 4090, and my logging of the early tasks, I think we can accept the current figure as being 'near enough right', and that's all it needs to be.

I think that the flops value is - since 2011-ish - reverse-engineered from your estimate of fpops_est and the measured runtime of the task on the volunteer's hardware. Pre 2011, BOINC took more notice of the 'peak flops' calculated by our computers from the speed and internal geometry of the GPU in use. BOINC guessed a 'fiddle factor' - I think something like 5% - as the ratio between the maximum usable speed on real-life jobs, and the calculated peak speed. But that was abandoned, except possibly for use as an initial seeding value.

Once the real-life data is available from the initial tasks run on a new computer, the server should maintain a running average value for flops for each computer attached to the project. That should wobble with small changes in actual task duration, which is why I was surprised to see it remained identical to the last significant digit in my run so far.

All the data necessary to calculate flops is returned as each task is reported complete. It's stored in the result table on the server, and should be transferred/averaged to the host table. I should be able to point you to the current code for managing that transfer, and the variable names and db field names used - though I may not be able to post them until after the weekend. Perhaps we could compare notes once I've found them?

08/09/2024 21:08:27 | GPUGRID | [error] Error reported by file upload server: Server is out of disk space

08/09/2024 21:08:27 | GPUGRID | [error] Error reported by file upload server: Server is out of disk space

Well. How does one managed to complete unit in 5 min ?

Im sitting on quite more then ok PC with decent 4090 card. and me units are close to 5 hours .)

Well. How does one managed to complete unit in 5 min ?

Im sitting on quite more then ok PC with decent 4090 card. and me units are close to 5 hours .)

Could it be that my Quadro P5000 is unable to crunch ATMMLs?
Several days ago, I tried it twice, and each time the tasks errored out after a few minutes (I guess, but cannot tell for sure: at the moment the GPU was supposed to start working after the initial steps).
BTW: the CPU is Intel Xeon E5 2667 v4 (two such CPUs are in the box).

Any ideas ?

I think a P5000 should work, given that it is the same generation as a 1080 which I have confirmed to work. It may be that your drivers are too old. I have a 1080 with driver version 536.

Hi,

Why do I receive such an error messages in ATMML tasks recently?

You have to read a long way further down to find the real answer to your question!

In the one I picked, I see:

Today, I've so far had 3 tasks which were "aborted by server" after several hours' runtime. Which is not really nice.
If the team decides that certain types of tasks are no longer needed, they should delete them from the distribution pool before they are sent out, instead of having us volunteeers crunch them for hours before stopping them.

Today, I've so far had 3 tasks which were "aborted by server" after several hours' runtime. Which is not really nice.
If the team decides that certain types of tasks are no longer needed, they should delete them from the distribution pool before they are sent out, instead of having us volunteeers crunch them for hours before stopping them.

Today, I've so far had 3 tasks which were "aborted by server" after several hours' runtime. Which is not really nice.
If the team decides that certain types of tasks are no longer needed, they should delete them from the distribution pool before they are sent out, instead of having us volunteeers crunch them for hours before stopping them.

Today, I've so far had 3 tasks which were "aborted by server" after several hours' runtime. Which is not really nice.
If the team decides that certain types of tasks are no longer needed, they should delete them from the distribution pool before they are sent out, instead of having us volunteeers crunch them for hours before stopping them.

I came here to report exactly the same thing.

Two were a relatively quick 'Error while computing' - perhaps around the 5% mark.

There was a message from Quico posted on the GPUGrid Discord server. They cancelled some parts of the project as they need to finish other parts quickly. They will resend those later.

I also lost quite a bit of run time, but they didn't have a better way to do it.

They cancelled some parts of the project as they need to finish other parts quickly.

They cancelled some parts of the project as they need to finish other parts quickly.

Let's hope the ones they need to finish quickly aren't the most recent ones that end with 'Error while computing'.

My 'Error while computing' today are all replication _0, type CDK2 or - a recent one - NEW_CDK2.

It doesn't really feel like that sort of problem, but I'll keep an eye on it and restart if it happens again.

At the moment, the machines seem happy on QC tasks - and that seems to be the work that they want crunching, judging by the SSP. When I see the RTS queue filling up again, I'll try one or two to see what happens.

At the moment, the machines seem happy on QC tasks - and that seems to be the work that they want crunching

Today, I've so far had 3 tasks which were "aborted by server" after several hours' runtime. Which is not really nice.

Another indication for this intention seems to be that they reduced the credit points for ATMML by 50 %

135 hours is a drag! Sorry to hear. We want our GPUs productive and not just hamster wheels.

Another indication for this intention seems to be that they reduced the credit points for ATMML by 50 %

I've seen that well before the cancellations started - but only for tasks which ran significantly quicker that the ones we're used to. I think that's fair.

135 hours is a drag! Sorry to hear. We want our GPUs productive and not just hamster wheels.

I just processed my first one after getting my computer back online.
It errored out.
There is a lot of depreciation going on in the task.
What is that all about?

12.9 hours run time.
https://www.gpugrid.net/results.php?userid=107556&offset=0&show_names=0&state=5&appid=