Getting a lot of failures on app 815(Cuda60)

Realize the effort and focus of the 8.15 and cuda60 is probably for maxwell, but just wanted to ensure you knew WinXP seems to error out on every one of these tasks while Win7/8 is ok.

See below thread
http://www.gpugrid.net/forum_thread.php?id=3671

Hi,

Just confirmation the new 8.15/cuda60 app on XP x86 with a GTX 650 ti crash with a popup message:
"The procedure entry point GetTickCount64 could not be located in the dynamic link library KERNEL32.dll"
Performed a project as suggested to no avail.

x86 bug?

cuda55 app was working fine.
Until the issue is resolved is there a way of identifying cuda60 tasks by name so I can write a script to abort them immediately?

Thanks

Downgrade video driver and you won't get Cuda 6

Downgrade video driver and you won't get Cuda 6

Matt,

I reset the project for both machine, and they are both still erroring out on every cuda60 WU. I went through the top 320 performing computers on the volunteer list last night, and did not find a single XP machine with drivers 334 or 335 that completed a cuda60 WU. The other XP machines just did not receive the WU's with the older drivers. Only a handful of the XP machines were on 334/335. The rest were on older drivers and just running the cuda55/42 WU's.

I am changing back to the 331 driver to avoid the cuda60 WU's for now. I can try the 335 driver again later if needed.

Thank you for the feedback and suggestion.

Regards,
Jeremy

Would it be possible to add a checkbox to the 'Run only the selected application' named 'Cuda 6' and one named 'Cuda < 6' ?
Using these choices should solve most probles with these version incompatibilities.

Alexander

I got the first tasks cuda 60 on March 19 and all were Calculate flawlessly. Already about 20 + already completed okay.
Times is about 25000sec, cpu 8500 but it can be for each user due to hw different one..
Good job GPUGRID, cheers

Would it be possible to add a checkbox to the 'Run only the selected application' named 'Cuda 6' and one named 'Cuda < 6' ?
Using these choices should solve most probles with these version incompatibilities.

Alexander

Will this also be compatible for WinXP 64bit? From the leader list the top machines are using it.

Hi,

Updated 8.15-cuda60 on the short Q (win XP x86, gtx 650ti) appears to getting further however running very slowly with CPU utilisation <1% (typically 8-10%) (Currently a SANTI_MAR419cap310 after 2hrs 22min is only at 47%)

Ah, noticed I am still getting the error "..GetTickCount64 could not be located in KERNEL32.DLL" msg which pops up as soon as I start the task :(

Update - rebooted and performed a project reset to no avail.

I have just completed my first 8.20 short run on a GTX 660 under WinXP 32-bit.
No problems or restarts at all at 3 hours 7 minutes total elapsed time; good work.
(And please take some time off during the week.)

Name 990x-SANTI_MAR423cap310-45-84-RND7785_1
Workunit 5346410
Created 23 Mar 2014 | 15:20:02 UTC
Sent 23 Mar 2014 | 15:20:14 UTC
Received 23 Mar 2014 | 18:30:08 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x0)
Computer ID 168614
Report deadline 28 Mar 2014 | 15:20:14 UTC
Run time 11,222.14
CPU time 1,951.00
Validate state Valid
Credit 18,300.00
Application version Short runs (2-3 hours on fastest card) v8.20 (cuda60)

I can confirm . 8.20 cuda 6 short run ; ran one each on gtx770 & gtx780 WinXP Pro 64bit with the 335.28 driver all went fine. Only just slightly longer GPU run time than observed with 5.5 and 332.21 driver.



PS: I have this wild dream that one day Nvidia comes out with a new driver and it actually shortens GPU run time ! DARE TO DREAM :)

Try the new version 820

Finally got some of the new 820(CUDA60) units on my XP32 cruncher and they started ok without erroring out :-)

Thanks for sorting out the problem with the app!!!

Update on 820 Cuda60 on XP Pro32

Computer 1 completed another for 2 good and 0 errors
Computer 2 I switched this morning to 335.28 completed 1, 1 in progress, and 0 errors.

Looks resolved. Again, thanks.

I'm still getting the occasional v8.15 allocated to my Maxwell - which is a bit of a waste for all concerned.

Task 8064144

Same here: My 750 Ti is getting Cuda55 long work units and they all error out... the short units are Cuda60 and are successful. A few days ago I got some long run Cuda60 units which were also sucessful - but I'm not getting those anymore.

Is there anything I can do to force Cuda 6 long run units?

http://www.gpugrid.net/results.php?hostid=170512

I'm still getting the occasional v8.15 allocated to my Maxwell

Don't know what's going wrong there. Try a project reset?

Matt

Fields for NVIDIA GPU apps:

<max_nvidia_compcap>0499</max_nvidia_compcap>
maximum compute capability

I've had 6 App820(CUDA60) units successfully complete on my XP32 cruncher... Looking good!!!

Agree; 820 now looking good on win x86 - gtx 650ti

Thanks

I'm still getting the occasional v8.15 allocated to my Maxwell - which is a bit of a waste for all concerned.

Task 8064144

Should we detach until this is fixed? Half my units are erroring due to 8.15.

This poor unit keeps being sent as 8.15 http://www.gpugrid.net/workunit.php?wuid=5508762

I guess no one cares enough. I'll detach, back to PrimeGrid with me.

I would keep running the WU's, but I would abort any 5.5 I see downloading. The WU's that fail do so very quickly but there is a lot of wasted Internet usage.

One day, my heterogeneous PC will have: 1 Maxwell GPU, 1 Kepler GTX 660 Ti, and 1 Fermi GTX 460... while running the latest NVIDIA drivers. I hope that, whatever scheduler changes you make, will still allow for heterogeneous scenarios such as mine.

I think the scheduler is written to 'test' sub-optimal plan_classes randomly, to avoid a client getting stuck indefinitely in a one-off rogue allocation.

That means that the periodic 'ping' with cuda55 is deliberate, and has to be actively prevented when, as now, there's an app or two which are incompatible with a certain class of cards or (more commonly) drivers.

Use that max_compute_capability (check the tag name - it's probably not that) on the lower plan classes to prevent them being sent to Maxwell or above.

I don't receive any long tasks on my 750 Ti. Should I be getting them ?
Win7 x64, driver 337.50

short and long..

@Mumak, I dont get any either (Ubuntu 12.04.4, GTX750Ti) since about two days, but the server status says always only 15, 21, 27 are available, so I ***guess*** there are simply much less long runs available than there is demand.

Doesn't seem that long cuda60 WUs are being sent out any more. Any particular reason? Has the bug with the cuda55 app that causes WU crashes when a machine is rebooted or BOINC is restarted been fixed?

My research indicates that I am still being given 8.15 apps that still infuriatingly crash. It seems that they never updated the non-cuda6 applications to the fixed 8.20 version :(

I continue to lose massive amounts of work every 3 days or so, as my computer usage habits require that I shutdown BOINC for a couple hours, and then the 8.15 tasks fail. It sucks.

Please try to suspend all work first and to then the reboot. That works for me, no errors when I start all projects again after booting. I am only getting 8.15 apps as I don't upgrade my 331.82 drivers yet.
____________
Greetings from TJ

Doesn't seem that long cuda60 WUs are being sent out any more. Any particular reason?

For Windows, I am still regularly getting 8.15 tasks. It's almost as if the non-cuda60 app versions were not rebuilt for 8.20. Any hopes of seeing it get fixed (since the 8.15's have the restart/resume bug?)

For Windows, I am still regularly getting 8.15 tasks. It's almost as if the non-cuda60 app versions were not rebuilt for 8.20. Any hopes of seeing it get fixed (since the 8.15's have the restart/resume bug?)

To clarify what I meant: There was an 8.20 cuda60 app on the Long Queue (proof pasted below), but now that app is gone, leaving only the buggy 8.15 apps. The applications page doesn't even indicate 8.20 on Long at all, and is a bit misleading.

I don't know what's going on, really. I just know that 8.20 seems more stable, yet I'm still being given 8.15's on the Long Queue, and they end up wasting work. :(

Yes, we appear to be in a transitional state. It just doesn't make sense why we are.

Proof:
Name I1100R11-SDOERR_BARNA-3-4-RND7766_0
Workunit 6406602
Created 9 Apr 2014 | 11:43:56 UTC
Sent 9 Apr 2014 | 14:54:40 UTC
Received 10 Apr 2014 | 8:49:06 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x0)
Computer ID 153764
Report deadline 14 Apr 2014 | 14:54:40 UTC
Run time 48,792.57
CPU time 7,842.47
Validate state Valid
Credit 137,700.00
Application version Long runs (8-12 hours on fastest card) v8.20 (cuda60)

Worse yet, I think I had managed to, at some point, get an 8.20 task to error out with the "The file exists. (0x50) - exit code 80 (0x50)" error I had been seeing with the 8.15's. Saddening and maddenning.

Name e5s7_e3s9f68-GIANNI_trypben1MCavg09-0-1-RND4755_1
Workunit 6406263
Created 9 Apr 2014 | 10:04:32 UTC
Sent 9 Apr 2014 | 12:34:36 UTC
Received 9 Apr 2014 | 14:53:42 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 80 (0x50) Unknown error number
Computer ID 153764
Report deadline 14 Apr 2014 | 12:34:36 UTC
Run time 7,386.87
CPU time 1,743.23
Validate state Invalid
Credit 0.00
Application version Long runs (8-12 hours on fastest card) v8.20 (cuda60)
Stderr output

<core_client_version>7.3.15</core_client_version>
<![CDATA[
<message>
The file exists.
(0x50) - exit code 80 (0x50)
</message>
<stderr_txt>
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : r337_00 : 33750
# GPU 0 : 67C
# GPU 1 : 57C
# GPU 2 : 74C
# GPU 0 : 68C
# GPU 1 : 60C
# GPU 0 : 69C
# GPU 1 : 63C
# GPU 1 : 65C
# GPU 0 : 70C
# GPU 1 : 66C
# GPU 0 : 71C
# GPU 1 : 68C
# GPU 1 : 69C
# GPU 0 : 72C
# GPU 1 : 70C
# GPU 1 : 71C
# GPU 2 : 75C
# GPU 1 : 72C
# GPU 2 : 76C
# GPU 2 : 77C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : r337_00 : 33750
# GPU 0 : 66C
# GPU 1 : 58C
# GPU 2 : 70C
# GPU 0 : 67C
# GPU 1 : 61C
# GPU 2 : 72C
# GPU 0 : 68C
# GPU 1 : 63C
# GPU 2 : 73C
# BOINC suspending at user request (exit)

</stderr_txt>
]]>

Thanks skgiven,
finally I know why I dont get any long WU's anymore :)

Hurray!! Thanks!! [I'll be sure to test the normal scenarios of exiting/restarting BOINC, and suspending/resuming tasks without restarting... as I rely on those scenarios all of the time!]

There will be an update for the older versions of the windows application coming tomorrow.

Matt

Thank you so much Matt. I know this will help to improve stability, although I think there is still some lingering issue, even in 8.20. :) I'm glad we are moving forward. Is there any way you would consider including additional debug in the stderr.txt, especially during suspends/resumes, especially so we might be better able to figure out why a task might abruptly just quit/exit with an error?

I'd want anything that would indicate WHY a task ended prematurely. I'd want it printed in stderr.txt.

Quite right, G2xx/sm13 is what I meant.

Checking now, I see sm13 cards now amount to <2% of compute capacity.

I shouldn't have bothered.

Matt

Mayday mayday mayday
The 8.40 CUDA5.5 app isn't working on my hosts.
GTX 680 / WinXPx64
GTX 780Ti / WinXPx64

EDIT: it's not working on any Windows host I've checked

EDIT2: I've suspended requesting new work, as I don't want my hosts get banned.

Same here - I have churned out 2 pages of failed v8.20's and 8.40's. Looks like it started with units sent around 1140 UTC. Wingmen are also erroring out.


XP Pro 332.21 GTX780ti and Titan

Same here! 8.40 does not work on w7 and driver 332.21.

Same here. All 8.40 failing on Win7-64 with driver 331.82.

http://www.gpugrid.net/results.php?hostid=165832&offset=0&show_names=1&state=0&appid=

I see that there is a recommended driver for the new app of 334.21 or newer.
http://www.gpugrid.net/forum_thread.php?id=3733#36505

I think I will go ahead and update this pc when I get a chance to see if it will work.

In the mean time, it may be best to not send 8.40 WU's to at least Win7-64 331.82 drivers. Not sure if that is possible.

My WinXP machines are currently crunching on the new WU's without errors. I see a 8.40 Cuda55 on a GTX460 and a 8.40 Cuda60 on a GTX680 on http://www.gpugrid.net/results.php?hostid=149863&offset=0&show_names=1&state=0&appid=.

My other WinXP machine, a 8.40 Cuda60 is on a GTX680 with no issues so far. This is on http://www.gpugrid.net/results.php?hostid=158717&offset=0&show_names=1&state=0&appid=

Both WinXP systems are on driver 335.28.

Yes, don't know what's gone wrong here. I've reverted back to 815 for now.

Matt.

Yes, don't know what's gone wrong here. I've reverted back to 815 for now.

See if you can get a new 841.

Me too! now as a Cuda 4.2.

[Edit] Started in this very moment with version 8.40 and Cuda 4.2. 2 minutes working without problems. [Edit]

See if you can get a new 841.

See if you can get a new 841.

I got a 8.15, and then a 8.41 on my another host.

Is 8.41 the same as 8.15?

Or is 8.41 the same as 8.40 with a fix? If so, what was the fix?

I put the 335.23 driver on my Win7-64 machine, and it is now crunching the 8.40 Cuda55 tasks. They are short and not long, but now crunching and not getting errors. Will leave this driver on for a while. I had a soft spot for the 331.82 driver. Was able to get the better utilization with 331.82 according to Precision as compared to the 335.23.

841, 840 and 820 are all the same code, I'd just made a mistake in the deployment (calling a cuda60 version cuda55), which only showed up when it hit hosts that weren't cuda 6 capable.

To summarise the current app version: cuda42 and cuda55 are version 841, cuda60 is 840. These are all the same code, just for different cuda revisions, and all include with the fix that prompted this all.

Hohum.

Matt

Hang in there, Matt. Thanks for the prompt response, for keeping us informed, and for moving forward with the apps. Much respect.

I put the 335.23 driver on my Win7-64 machine, and it is now crunching the 8.40 Cuda55 tasks. They are short and not long, but now crunching and not getting errors. Will leave this driver on for a while. I had a soft spot for the 331.82 driver. Was able to get the better utilization with 331.82 according to Precision as compared to the 335.23.

Remember SWAN_SYNC? Next time I rev the app, I'll introduce the ability to force busy waiting, which ought to give best performance, at the cost of CPU.

In the other thread you said 8.41 would already have it. I have such a CUDA 6 WU running under Win 8.1. I created a user or system environment variable called SWAN_SYNC, set it to 0 and rebooted. However, CPU usage remains at 1.4 - 1.8% (8 threads), so it's obviously not working yet. Did I do anything wrong?

MrS

Remember SWAN_SYNC? Next time I rev the app, I'll introduce the ability to force busy waiting, which ought to give best performance, at the cost of CPU.

In the other thread you said 8.41 would already have it. I have such a CUDA 6 WU running under Win 8.1. I created a user or system environment variable called SWAN_SYNC, set it to 0 and rebooted. However, CPU usage remains at 1.4 - 1.8% (8 threads), so it's obviously not working yet. Did I do anything wrong?

MrS

I was so happy receiving just cuda55/cuda42 wus for the GTX660Ti and now I'm receiving cuda60 units that are all failing since my driver is old but was perfect for my cards and ubuntu system.

Hello guys,

there is also something wrong with some long WU's on Maxwell cards,

I'm on GTX750Ti here, Driver 334.21, Ubuntu 12.04.4 64 bit.

I have a A2ART4Ex04x21-GERARD_A2ART4E-1-14-RND8074_3 task, cuda 60 8.21 and after half an hour its at 0.2% done, CPU is at almost 100% while GPU is almost idle.

skgiven,

SWAN_SYNC it has different meaning now - it merely needs to be present (set to any value at all, including 0) to have an effect.

Matt

Have had SWAN_SYNC set to 0 on all my machines. Also running with the <cpu_usage>1.0</cpu_usage> line in my app_config. Woke up today to see each machine has completed tasks on the 335 driver (XP and Win7) where CPU time is nearly equal to GPU time.

Each thread given to a GPUGrid task is now actually using the whole thread now. Yup, it is working. Thanks.

All tasks v8.21(cuda60) are failing in my Linux host with driver 304.88. My cards are GTx 660Ti.

What have I to change to run them properly?

Thanks for your help.

Thanks, major issue I see. no time for it now.

Thank you skgiven,
it seemed to have been a one time issue, other longs are doing well, will report if I encounter it again. And I run tasks on CPU but have a core reserved for GPUGrid always.

Peter_M, you completed other work with the 8.21 app, so it's probably not an app issue. It sounds like the WU is/was just running very slow. Check the GPU clocks in NVidia X server and if need be restart the application, system or even abort the WU. Are you running CPU tasks as well as GPU tasks? It doesn't look like it but if you are try to suspend the CPU work.

So now it seems established that SWAN_SYNC reserves a whole CPU core. But is it any faster? If so, how much?

It's very easy to test, just close BOINC, create or remove the System Variable, and restart BOINC.

On one of my tests, where I temporarily suspended all projects except GPUGrid (such that no CPU tasks were interfering), I showed GPU Usage go from 93% without SWAN_SYNC, to 96% with SWAN_SYNC. So, that's ~3% speedup, for that sample, which should translate into a similar amount of task completion throughput. But since I run other CPU projects, I have decided to not use SWAN_SYNC.

It's very easy to test, just close BOINC, create or remove the System Variable, and restart BOINC.

On one of my tests, where I temporarily suspended all projects except GPUGrid (such that no CPU tasks were interfering), I showed GPU Usage go from 93% without SWAN_SYNC, to 96% with SWAN_SYNC. So, that's ~3% speedup, for that sample, which should translate into a similar amount of task completion throughput. But since I run other CPU projects, I have decided to not use SWAN_SYNC.

So now it seems established that SWAN_SYNC reserves a whole CPU core. But is it any faster? If so, how much?