So far I've had
one long wu error after 13s

one short wu complete OK
one long wu stall, ran for 2hr on a 660 but little progress, so got the bullet

Be on your guard!

Encountering issues with Noelia WUs:

http://www.gpugrid.net/result.php?resultid=7218125

http://www.gpugrid.net/result.php?resultid=7218124

http://www.gpugrid.net/result.php?resultid=7218111

All had the same error, swanMemset failed

I've had 5 NOELIA's fail in the past 24 hours.

Hi, for some reason some of you are having problems with this WUs in the new application and we've moved them to the beta queue to have a proper look. I've also just sent 50 WU under the name KLEBEbeta with a much simpler configuration file. These simulations are really important, and fixing this bug will also help for future similar projects in drug discovery. Please report any problems you might have on groups KLEBEs and KLEBEbeta.

Noelia,

Thanks for moving to beta to try to fix the problems!
I'm noticing a "new" problem with the NOELIA_KLEBEbeta tasks, though.

Essentially, they get assigned to a GPU, and then when they try to "get going", BOINC shows that they run for about 15-40 seconds, and then the task resets back to the beginning (with Elapsed back to 0 seconds), and it retries.
It just keeps retrying until failure.
Additionally, if the user closes BOINC, the acemd.800-55.exe process for that task does not close properly (it still remains in the Task Manager's process list, even though all other related BOINC processes have exited normally).

Also, looking at stderr.txt for one of the tasks that I aborted (http://www.gpugrid.net/result.php?resultid=7221709), said the following lines that might give a hint as to what's happening:
swanMemset failed
Can't acquire lockfile - exiting
FILE_LOCK::unlock(): close failed.: No error

I have not seen this behavior before today, so I think there is at least 1 new bug here.

This happens both on my GTX 660 Ti, as well as my GTX 460, in Windows 8.1 Preview x64.

The current task exhibiting this behavior is:
109nx4-NOELIA_KLEBEbeta-0-3-RND0846_0

I hope this information helps you to track it down to correct the problem(s) quickly, as right now my GPU is spinning in circles and doing no work. Are you able to reproduce the problem in your testing?

If there's anything else you might need, please let us know.

Thanks,
Jacob

Lol What a "luck" ^^

http://www.gpugrid.net/workunit.php?wuid=4729004

One of my maschine failed this, and i got exactly this wu to my next machine where it stucks O.o saw it now after 2 hours. Puh earliy enough before weekend ^^
____________
DSKAG Austria Research Team: http://www.research.dskag.at

Thanks for the prompt response. I got an 801 KLEBEbeta, and it's at least getting off the ground now.

I hope that you can see that it is very difficult for us to know if the problem is in the task set, or if the problem is in the application.

Will continue to monitor...

I tested suspending one of these KLEBEbeta tasks, and it caused a driver reset. So, the problem still persists.

Can you please look into it more closely? The issue has to deal with how the KLEBE tasks are exiting - it seems they are not releasing the GPU in a timely fashion, as compared to every other GPU task I run (across all my GPU projects).

Maybe compare the exit logic of a KLEBE task, versus the exit logic of other GPUGrid task types?

On my Titan box I've gotten two of these NOELIAs.

Both exhibited the same behavior.

"8/29/2013 7:36:55 AM | GPUGRID | Task 063px38-NOELIA_KLEBEs-1-3-RND3786_0 exited with zero status but no 'finished' file"

Over and over again without making much progress.

So I pulled the trigger to kill them.

So it's back to babysitting to make sure I only get NATHAN longs for the time being.

That NOELIA sure has a reputation! ;-}

Operator

____________

Jacob,

The current beta addresses the "swanMemset failed" and "access violation" errors.
The suspend problem I have not yet investigated. (Is it with 'suspend to memory' or 'suspend and exit'? )

MJH

I have two NOELIA KLEEBEbeata on my 770 and they start, then when 0.021% complete, no more progress but they keep running, 2h57m16s elapsed and 0h0m0s remaining. This was in app 8.00 and I have now aborted these WU's and try the new 8.01 app.

I have now 1 with the 8.01 (cuda55) app (NOELIA KLEEBEbeta) and it is running normal. Twelve hour to finish, progress runs up, elapsed time runs up, and remaining runs down. Win7 x64, BOINC 7.0.64, driver 326.80
____________
Greetings from TJ

I have two NOELIA KLEEBEbeata on my 770 and they start, then when 0.021% complete, no more progress but they keep running, 2h57m16s elapsed and 0h0m0s remaining. This was in app 8.00 and I have now aborted these WU's and try the new 8.01 app.

Would anyone with a cc 1.3 card - Geforce GTX 200 series - please try some of the current acemdbeta v801 Noelia-KLEBE WUs and report back here?

MJH

With the new 8.01 app they run normal!

I noticed the following:
On the 770 I have this one: 063px79-NOELIA_KLEEBEbeta2-0-3-RND678_0
MEM use: 1003MB
Clock: 1097MHz (however I have set the clock to 1060MHz!)
GPU load: 87%
Temp: 65°C
7.5% done in 40 minutes

On the 660 I have this one: 109nx37-NOELIA_KLEEBEbeta-0-3-RND0283_0
MEM use: 779MB
Clock: 1045MHZ (as I set it)
GPU load: ~88%
Temp: 67°C
5.6% done in 40 minutes

I now these are not the same WU's and the GPU's are not the same as well. But it is strange that the WU can manage to get the clock higher, or this must have been the result of the faulty WU that I aborted and not reboot afterwards.

Due to the difference in memory load it can also be that cards with only 1MB can not do these WU's as before. That may result in some comments ;-)
____________
Greetings from TJ

Would anyone with a cc 1.3 card - Geforce GTX 200 series - please try some of the current acemdbeta v801 Noelia-KLEBE WUs and report back here?

MJH

Ok i started one with 8.01. But this can take some time even on my 670mhz 285gtx..I normaly dont run gpugrid on this anymore. It will need about 33hours. The short run 8.00 was ok on this card. I dont think anybody still uses a powerhungry 200series on long runs O.o

I have noticed that, using the 8.01 app on a NOELIA_KLEBEbeta task, on my GTX 660 Ti, the process does not utilize a full CPU core (like other GPUGrid tasks normally do for that GPU). It's like SWAN_SYNC is not set correctly. Though I'm still getting good (85-91%) GPU utilization for the task.

Is this behavior new? Also, is it expected?

Oh and when somebody started one too on 200series, plz tell me, got my energybill today, so i would love to stop it the next hours when not needed :p

If it is still running then that's plenty long enough to demonstrate that all is well, thanks. You can kill it off.

Matt

I have noticed that, using the 8.01 app on a NOELIA_KLEBEbeta task, on my GTX 660 Ti, the process does not utilize a full CPU core (like other GPUGrid tasks normally do for that GPU). It's like SWAN_SYNC is not set correctly. Though I'm still getting good (85-91%) GPU utilization for the task.

Is this behavior new? Also, is it expected?

That's fine if that's the case. I don't know, which is why I asked. I'm "used" to seeing tasks on my "Kepler" (GTX 660 Ti) taking a full CPU core (via SWAN_SYNC) automatically. Maybe NOELIA tasks work differently.

ID 156955: i7-3770K (no HT, 4 CPU cores) GTX560Ti, W7 64bit driver 328.80.
NOELIA_KLEBEbeta 8.02 CUDA55 application
GPU [GeForce GTX 560 Ti] Platform [Windows] Rev [3182M] VERSION [55]
# SWAN Device 0 :
# Name : GeForce GTX 560 Ti
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:01:00.0
# Device clock : 1720MHz
# Memory clock : 2100MHz
# Memory width : 256bit
# Driver version : r325_00
GPU load 91 - 94%, process priority tamed to realtime, GPU load is 87 - 91% if not tamed
CPU load 20% of one core (5% in all)

That's fine if that's the case. I don't know, which is why I asked. I'm "used" to seeing tasks on my "Kepler" (GTX 660 Ti) taking a full CPU core (via SWAN_SYNC) automatically. Maybe NOELIA tasks work differently.

:) Yeah, thanks. I've helped Einstein fix a bug, MindModeling fix a bug, GPUGrid fix a couple things, Test4Theory fix a bug, Rosetta fix their app, SETI fix a GPU estimate problem, got nVidia to fix a monitor-sleep issue, and more. And I also do alpha/beta testing of the actual BOINC software, and have worked directly with the BOINC devs.

Regarding this particular case, I believe I was aware of some tasks "not using a full CPU core on my Kepler card", but I did not know it was NOELIA ones. I'll try to keep that in mind.

Thanks again.

That's fine if that's the case. I don't know, which is why I asked. I'm "used" to seeing tasks on my "Kepler" (GTX 660 Ti) taking a full CPU core (via SWAN_SYNC) automatically. Maybe NOELIA tasks work differently.

Have revved the beta app to 8.02. This might also fix the driver-hang-on-suspend problem.

MJH

That's fine if that's the case. I don't know, which is why I asked. I'm "used" to seeing tasks on my "Kepler" (GTX 660 Ti) taking a full CPU core (via SWAN_SYNC) automatically. Maybe NOELIA tasks work differently.

I still consider this different CPU load as a malfunction.
However, with this low CPU load the GPU load is still above 95%, so we can turn this question the way around: is it sure that the other tasks need a full CPU thread to feed a Kepler GPU?

8.02 beta tasks seem to work ok on 780s, but now all other tasks fail

I still consider this different CPU load as a malfunction.
However, with this low CPU load the GPU load is still above 95%, so we can turn this question the way around: is it sure that the other tasks need a full CPU thread to feed a Kepler GPU?

On the Folding forum, there have been extended discussions of Nvidia CPU core usage under CUDA. It contrasts to the case of AMD cards running OpenCL, which typically require only a few percent of a CPU core.

As I recall, Nvidia provides the option to the developers to reserve a full CPU core when running under CUDA using spin states, which I don't understand anyway. If the application developers want to ensure that they have enough CPU support, they can reserve it, even though typically not all of it is actually in use

So maybe the other tasks don't really require a full core, except that it may be useful to reserve it for stability or performance or whatever.

EDIT: To further complicate matters, Nvidia cards running OpenCL always require a full CPU core; there is no option not to.

Watching two different third-party developers working on SETI (one specialising in CUDA, the other in OpenCL), we get the opposite outcome: OpenCL on ATI is inefficient unless a spare CPU core is available, but CUDA on Nvidia requires very little CPU.

I'm not a developer myself (at least, not at the level these guys program), but from the peanut gallery it looks as if CPU usage is very much down to the skill of the developer, and how well they know their platform and tools.

But I'm interested by the OpenCL on Nvidia point. That does seem to be a common observation - I wonder if it has necessarily to be so? Or maybe Mvidia didn't port some of their synch technology from CUDA to the OpenCL toolchain yet?

But I'm interested by the OpenCL on Nvidia point. That does seem to be a common observation - I wonder if it has necessarily to be so? Or maybe Mvidia didn't port some of their synch technology from CUDA to the OpenCL toolchain yet?

All I know is that on Folding with their newest OpenCL core_17, which runs on both AMD and Nvidia, the situation is reversed. It requires only 1 or 2 percent on AMD cards (e.g., my HD 7870 on an i7-3770), whereas on an Nvidia card it reserves a full core (e.g., on my GTX 660 Ti). The question has been asked on the Folding forum as to whether that is necessary, and the answer is that Nvidia has not implemented the option in OpenCL to use less than a full core. Apparently they could if they wanted to, but maybe for performance reasons (so the speculation goes) they want their cards to perform the best they can, so they just grab the whole core. It helps solve the problem you mentioned above, where users don't always know to leave a core free I suppose.

Perhaps you might create a new thread devoted toward finding the OpenCL/CUDA information.
This thread is for "WU: NOELIA_KLEBEs" :)

Perhaps you might create a new thread devoted toward finding the OpenCL/CUDA information.
This thread is for "WU: NOELIA_KLEBEs" :)

http://www.gpugrid.net/result.php?resultid=7221215

Would progress to 0.21% and then sit while elapsed time increased. Switched from 660Ti to 580 with same result. Aboirted after 3 hrs 38 min on 580.
____________

Thank you.

However;

i. I would have to be convinced that this my problem?

ii. Why are deprecated WU being dispatched? That certainly is not a problem caused by me.

____________

IM getting zero output file errors every few minutes and it slams the fan on my card and resets the task.

It is also showing my video cards in like slots 7 and 8. I believe the should be slot 0 and 1 or is that moot?

This has been going on a few days, I reset the project and aborted a few tasks thinking they were the problem to find it's ongoing.

I see this is an issue that is known?? Any etr?

Thank you
Aaron

How can you all see that a full core is used with one GPUGRID WU?
____________
Greetings from TJ

How can you all see that a full core is used with one GPUGRID WU?

Thanks for the info, skgiven.

I have overridden the CPU requirements, via app_config.xml. Because I have 2 GPUs that do GPUGrid (1 Fermi, 1 Kepler), I had set cpu_usage to 0.5 for all GPUGrid app types, so that when both cards are working on GPUGrid, BOINC reserves 1 total CPU core for them, keeping the CPU slightly above saturation. I've since changed my logic a bit so as to slightly undersaturate the CPU; I accomplished that by changing cpu_usage to 1.0 for all GPUGrid app types, so a logical CPU core is reserved for each, which I think is what you guys always recommended anyway.

Long story short, I used Process Explorer to confirm that NOELIA_KLEBE units strangely do not use a full CPU on my Kepler card, whereas it seems to me that every other GPUGrid task does use a full CPU on my Kepler card. It matters to me since they are "mixed in" with other tasks in the "long" app, and my cpu_usage setting now applies to some tasks that won't use a full core. In a perfect world, and if I were an admin, I might consider placing "strange types" like this in a separate app queue, maybe.

Thank you very much for confirming this is "normal" for NOELIA_KLEBE on Kepler.
Jacob.

How can you all see that a full core is used with one GPUGRID WU?

It can be checked in the Windows Task Manager: look for the acemd.80x-55.exe (or acemd.80x-42.exe) on the "Processes" tab. If its CPU usage is 1-2%, then it's not using a full core, otherwise the CPU usage is 100/the number of your CPU's threads (12-13% on a 8-threaded CPU, 8% on a 12-threaded CPU). You can check the past workunits' CPU usage at your hosts' task list: if the "CPU time" (almost) equals the "run time", then the task used a full core, if the "CPU time" is significantly less than the "run time", then it didn't use a full core.

I had a NOEL_KLEBEbeta WU error out because of this:

8/31/2013 3:59:44 PM | GPUGRID | Aborting task 063px53-NOELIA_KLEBEbeta2-2-3-RND7138_0: exceeded elapsed time limit 4172.44 (250000000.00G/2062.63G)
8/31/2013 3:59:47 PM | GPUGRID | Computation for task 063px53-NOELIA_KLEBEbeta2-2-3-RND7138_0 finished


Here is the link:


http://www.gpugrid.net/result.php?resultid=7230639

The unit was good to that point.

I had another NOEL_KLEBEbeta WU error out.


http://www.gpugrid.net/result.php?resultid=7231327


8/31/2013 7:16:37 PM | GPUGRID | Aborting task 063px63-NOELIA_KLEBEbeta2-2-3-RND4579_0: exceeded elapsed time limit 3857.75 (250000000.00G/64804.69G)
8/31/2013 7:16:41 PM | GPUGRID | Computation for task 063px63-NOELIA_KLEBEbeta2-2-3-RND4579_0 finished
8/31/2013 7:16:41 PM | GPUGRID | Output file 063px63-NOELIA_KLEBEbeta2-2-3-RND4579_0_1 for task 063px63-NOELIA_KLEBEbeta2-2-3-RND4579_0 absent
8/31/2013 7:16:41 PM | GPUGRID | Output file 063px63-NOELIA_KLEBEbeta2-2-3-RND4579_0_2 for task 063px63-NOELIA_KLEBEbeta2-2-3-RND4579_0 absent
8/31/2013 7:16:41 PM | GPUGRID | Output file 063px63-NOELIA_KLEBEbeta2-2-3-RND4579_0_3 for task 063px63-NOELIA_KLEBEbeta2-2-3-RND4579_0 absent

I updated to the 326.98 driver last night. The 1 NOEL_KLEBEbeta WU I received failed with 197 (0xc5) EXIT_TIME_LIMIT_EXCEEDED error message.
http://www.gpugrid.net/result.php?resultid=7232695

The 20 or so MJHARVEY_TEST14 betas I received all finished and validated.

I updated to the 326.98 driver last night. The 1 NOEL_KLEBEbeta WU I received failed with 197 (0xc5) EXIT_TIME_LIMIT_EXCEEDED error message.
http://www.gpugrid.net/result.php?resultid=7232695

The 20 or so MJHARVEY_TEST14 betas I received all finished and validated.