Message boards : Server and website : Apache HTTP/1.1 413 Request Entity Too Large
Author | Message |
---|---|
Could somebody please attend to this? I've posted full details of this before, at message 57397. 06/10/2021 08:10:27 | GPUGRID | Started upload of e2s61_e1s44p0f535-ADRIA_AdB_KIXCMYB_HIP-0-2-RND3591_2_9 06/10/2021 08:10:28 | GPUGRID | [http] [ID#15096] Received header from server: Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_auth_gssapi/1.3.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5 06/10/2021 08:10:29 | GPUGRID | [http] [ID#15096] Sent header to server: Content-Length: 535846469 06/10/2021 08:10:29 | GPUGRID | [http] [ID#15096] Received header from server: HTTP/1.1 413 Request Entity Too Large (WU 27079900) Obviously, the server doesn't care what operating system is trying to contact it. But it does show that the combination of new applications and new data is capable of producing output files larger than the current server configuration is able to handle. That's a waste of our resources, and may deprive the project of research results. | |
ID: 57496 | Rating: 0 | rate: / Reply Quote | |
Apache HTTP/1.1 413 Request Entity Too Large +1 I've also been affected once by the same problem, commented at Message #57194 | |
ID: 57497 | Rating: 0 | rate: / Reply Quote | |
See my reply here: https://www.gpugrid.net/forum_thread.php?id=5239&nowrap=true#57187 | |
ID: 57502 | Rating: 0 | rate: / Reply Quote | |
See my reply here: https://www.gpugrid.net/forum_thread.php?id=5239&nowrap=true#57187 some bug/idiosyncrasy on your system ? PS. 5 successful and currently running #6 and #7 cuda101 tasks on Ampere. | |
ID: 57503 | Rating: 0 | rate: / Reply Quote | |
... cuda101 tasks on Ampere. I am really wondering how come | |
ID: 57504 | Rating: 0 | rate: / Reply Quote | |
See my reply here: https://www.gpugrid.net/forum_thread.php?id=5239&nowrap=true#57187 could be, or on Servic's system. no evidence either way. just observed behavior from two systems running the same task with different results. ____________ | |
ID: 57505 | Rating: 0 | rate: / Reply Quote | |
... cuda101 tasks on Ampere. technically, one could make a copy of the working 1121 app (it's in the project folder), rename it to the same name as the not-working cuda101 app. and replace the file. in this case, you will have two identical apps, under different names, and BOINC will treat them separately as if they were different (and mark tasks as such). you might have to set the <dont_check_file_sizes>1</dont_check_file_sizes> flag to off in the cc_config.xml file so it doesn't get detected/replaced by the project. at the very least this could be a workaround for people who are being affected by the wrong app being sent and causing errors. I've done this on other projects before. I haven't tried on GPUGRID so there could be a complication with their wrapper setup. tasks destined for the cuda101 app would pull up the replaced version which is really the cuda1121 app under the hood and it should work. this is all off-topic for the issue brought up in this thread though. ____________ | |
ID: 57506 | Rating: 0 | rate: / Reply Quote | |
Hi Ian&Steve C., | |
ID: 57538 | Rating: 0 | rate: / Reply Quote | |
glad it's working as I theorized :) | |
ID: 57539 | Rating: 0 | rate: / Reply Quote | |
technically, one could make a copy of the working 1121 app (it's in the project folder), rename it to the same name as the not-working cuda101 app. and replace the file. in this case, you will have two identical apps, under different names, and BOINC will treat them separately as if they were different (and mark tasks as such). you might have to set the <dont_check_file_sizes>1</dont_check_file_sizes> flag to off in the cc_config.xml file so it doesn't get detected/replaced by the project. I have now done this, but due to lack of new WUs, it cannot be testet :-( | |
ID: 57540 | Rating: 0 | rate: / Reply Quote | |
Thanks everyone for keeping the Apache problem at the top of this message board for admin to notice! ;-) | |
ID: 57541 | Rating: 0 | rate: / Reply Quote | |
technically, one could make a copy of the working 1121 app (it's in the project folder), rename it to the same name as the not-working cuda101 app. and replace the file. in this case, you will have two identical apps, under different names, and BOINC will treat them separately as if they were different (and mark tasks as such). you might have to set the <dont_check_file_sizes>1</dont_check_file_sizes> flag to off in the cc_config.xml file so it doesn't get detected/replaced by the project. I got downloaded a WU with cuda101, it startet and has been running for several minutes now. So far, the steps as described above seem to be successful :-) Remains just to hope that after about 14 hours, the WU will deliver a valid result. | |
ID: 57543 | Rating: 0 | rate: / Reply Quote | |
Apache HTTP/1.1 413 Request Entity Too Large Regarding the original topic, I see that your mentioned task e2s61_e1s44p0f535-ADRIA_AdB_KIXCMYB_HIP-0-2-RND3591_2 is still stalled trying to upload unsuccessfully. It will be reaching its deadline tomorrow on 7:39:06 UTC After that, a new task e2s61_e1s44p0f535-ADRIA_AdB_KIXCMYB_HIP-0-2-RND3591_3 hanging from WU #27079900 will be generated and sent to another user. And so on, wasting processing time at every new user... unless the problem is solved on server side, or maximum allowed number of tasks is reached, whatever comes first. Earlier reference at Message #57321 | |
ID: 57545 | Rating: 0 | rate: / Reply Quote | |
Yes, I can see it uploading on the screen in front of me, and the deadline is 08:39:06 local (British time - UTC+1). | |
ID: 57546 | Rating: 0 | rate: / Reply Quote | |
[http] [ID#116] Sent header to server: Content-Length: 540810854
[http] [ID#116] Sent header to server: Content-Type: application/x-www-form-urlencoded
[http] [ID#116] Sent header to server: Expect: 100-continue
[http] [ID#116] Sent header to server:
[http] [ID#116] Received header from server: HTTP/1.1 413 Request Entity Too Large
[http] [ID#116] Received header from server: Date: Sun, 10 Oct 2021 07:14:13 GMT
[http] [ID#116] Received header from server: Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_auth_gssapi/1.3.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
[http] [ID#116] Received header from server: Connection: close
[http] [ID#116] Received header from server: Content-Type: text/html; charset=iso-8859-1
[http] [ID#116] Received header from server:
[http_xfer] [ID#116] HTTP: wrote 360 bytes
[http_xfer] [ID#116] HTTP: wrote 527 bytes
[http] [ID#116] Info: Closing connection 252
[http] [ID#116] Info: TLSv1.2 (OUT), TLS alert, Client hello (1):
[file_xfer] http op done; retval -224 (permanent HTTP error)
[file_xfer] file transfer status -224 (permanent HTTP error)
Backing off 04:15:23 on upload of e2s67_e1s44p0f1240-ADRIA_AdB_KIXCMYB_HIP-0-2-RND1963_3_9
work fetch suspended by user I put my hosts back to FAH until it gets fixed. There are many ways to fix it on the project's side. | |
ID: 57561 | Rating: 0 | rate: / Reply Quote | |
I probed the actual cutoff point of the transfer length. [http] [ID#122] Sent header to server: Content-Length: 536000482
[http] [ID#122] Sent header to server: Content-Type: application/x-www-form-urlencoded
[http] [ID#122] Sent header to server: Expect: 100-continue
[http] [ID#122] Sent header to server:
[http] [ID#122] Received header from server: HTTP/1.1 413 Request Entity Too Large So I tried 530.000.000 bytes, and it's uploading: [http] [ID#123] Sent header to server: Content-Length: 530000482
[http] [ID#123] Sent header to server: Content-Type: application/x-www-form-urlencoded
[http] [ID#123] Sent header to server: Expect: 100-continue
[http] [ID#123] Sent header to server:
[http] [ID#123] Received header from server: HTTP/1.1 100 Continue I think my result will be invalid, as the result file is truncated. ---------- EDIT ---------- The upload stucked at 98% (505.45MB). [http] [ID#123] Info: We are completely uploaded and fine
[http] [ID#123] Received header from server: HTTP/1.1 200 OK
[http] [ID#123] Received header from server: Date: Sun, 10 Oct 2021 07:40:22 GMT
[http] [ID#123] Received header from server: Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_auth_gssapi/1.3.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
[http] [ID#123] Received header from server: Transfer-Encoding: chunked
[http] [ID#123] Received header from server: Content-Type: text/plain; charset=UTF-8
[http] [ID#123] Received header from server:
[http_xfer] [ID#123] HTTP: wrote 135 bytes
[http] [ID#123] Info: Connection #260 to host www.gpugrid.net left intact
[file_xfer] http op done; retval 0 (Success)
[error] Error reported by file upload server: EOF on socket read : asked for 16382, got 9536
[file_xfer] parsing upload response: <data_server_reply> <status>1</status> <message>EOF on socket read : asked for 16382, got 9536</message></data_server_reply>
[file_xfer] parsing status: -127
[file_xfer] file transfer status -127 (transient upload error)
Temporarily failed upload of e2s67_e1s44p0f1240-ADRIA_AdB_KIXCMYB_HIP-0-2-RND1963_3_9: transient upload error
Backing off 03:20:09 on upload of e2s67_e1s44p0f1240-ADRIA_AdB_KIXCMYB_HIP-0-2-RND1963_3_9 | |
ID: 57562 | Rating: 0 | rate: / Reply Quote | |
The first thing to do is to establish communication between the research teams and the server administration. The research team's assumption about what the server can handle is clearly false - one side or the other has to make the next move. | |
ID: 57563 | Rating: 0 | rate: / Reply Quote | |
And another one: WU 27079900. This one's 535,845,928 bytes | |
ID: 57568 | Rating: 0 | rate: / Reply Quote | |
And another one: WU 27079900. This one's 535,845,928 bytes I would have thought that was down to whether one_result_per_host_per_wu or one_result_per_user_per_wu had been specified ? This WU only wants one valid result, if the server says it is valid nothing else matters does it ? If I see I have one of these tasks that are too big I will abort it. Can't see the point in wasting time on the off-chance that it just might upload, especially when it can be easily fixed by the powers that be. | |
ID: 57569 | Rating: 0 | rate: / Reply Quote | |
Regarding the original topic, I see that your mentioned task e2s61_e1s44p0f535-ADRIA_AdB_KIXCMYB_HIP-0-2-RND3591_2 is still stalled trying to upload unsuccessfully. I was feeling bored, so I tried Retvari Zoltan's method on it. I tried a size which turned out to be 532,721,637 bytes, roughly half way between his attempts. [I was using a line-based editor, so no point in trying for round numbers]. And - and I think this is crucial - I changed the value stored in client_state.xml to the same value. Just look for an _9 file file with lots of upload attempts - there won't be many around. Just change the nbytes value. And it uploaded, and the task reported. And, not surprisingly, it was declared invalid with a Validate error. Which is as it should be. Don't try this at home, kiddies! | |
ID: 57581 | Rating: 0 | rate: / Reply Quote | |
I can understand you, I've experienced a similar situation... | |
ID: 57582 | Rating: 0 | rate: / Reply Quote | |
I tried a size which turned out to be 532,721,637 bytes. And - and I think this is crucial - I changed the value stored in client_state.xml to the same value. Just look for an _9 file file with lots of upload attempts - there won't be many around. Just change the nbytes value.This file is actually a compressed text file, so it will be always invalid if you truncate the compressed stream like that, as you mess up the CRC and the stream length etc. So we can't fix this issue this way, only figure out the exact limit of the upload. I still think that the most straightforward fix is to release this batch as a 3 fragment simulation (it's a 2 fragment batch at the moment as there is -0-2- and -1-2- in their names). This would make 2/3 the processing time of the workunits and the upload size of the result compared to the present ones. | |
ID: 57585 | Rating: 0 | rate: / Reply Quote | |
That was my next thought - though it would be tricky to shorten the de-compressed file so that it re-compressed to exactly xxx,000,000. | |
ID: 57586 | Rating: 0 | rate: / Reply Quote | |
@ Retvari Zoltan: | |
ID: 57590 | Rating: 0 | rate: / Reply Quote | |
What archive manager did you use to decompress the file?WinRAR. It's not in a standard gzip or zip format, and I don't see any plaintext clues inside the file itself.True. It's zipped I guess, but it has no filename(s) and other header information inside, as it is unnecessary for a single file. My Linux Archive Manager won't open it, even when I try to fool it with .gz / .zip / .7z extensions on the file name.I couldn't open it with WinRAR either. EDIT: I hope it's scrambled with a password :) | |
ID: 57591 | Rating: 0 | rate: / Reply Quote | |
I got stuck with one of these units | |
ID: 57622 | Rating: 0 | rate: / Reply Quote | |
I got stuck with one of these units I don't believe this. I couldn't upload the file, so I aborted the upload, and the server still gave me credit. This is just plain weird!!!!!! | |
ID: 57623 | Rating: 0 | rate: / Reply Quote | |
Message boards : Server and website : Apache HTTP/1.1 413 Request Entity Too Large