Multi million file upload to Google Cloud Storage occasionally file fails stalls process.
I'm uploading a lot of individual small files to Google Cloud Storage (S3 type) using Version: 5.19.4 – see attachment.
I started it Wednesday, and it is now Sunday.
There are a lot of files, and it should take approximately 1 day I guess.
The reason it is taking so long is that apparently randomly and periodically Google servers return
So WinSCP then halts the upload process with a dialogue (attached)
If I click "Retry", it always works on this attempt and thousand more files are uploaded until Google throws the 503 again. This, of course, happens at 2am and I don't see the error until 9am. So those intervening hours have been lost.
Eventually I will be able to upload all these files. But what would be better if there was a setting and countdown. "Automatically retry failed uploads after X seconds" and WinSCP would continue and automatically close the dialogue and retry itself.
I've attached a short video of the clicking "Retry", and that the file that previously failed then works no problem on the reattempt.
I started it Wednesday, and it is now Sunday.
There are a lot of files, and it should take approximately 1 day I guess.
The reason it is taking so long is that apparently randomly and periodically Google servers return
6gb WinSCP log file so far.... I've just copied the end of it:503 service unavailable
. 2021-11-21 04:04:57.621 Header Name: [content-length], Value: [19] . 2021-11-21 04:04:57.621 Header Name: [date], Value: [Sun, 21 Nov 2021 04:04:57 GMT] . 2021-11-21 04:04:57.621 Header Name: [server], Value: [UploadServer] . 2021-11-21 04:04:57.621 Header Name: [connection], Value: [close] . 2021-11-21 04:04:57.621 End of headers. . 2021-11-21 04:04:57.622 sess: Closing connection. . 2021-11-21 04:04:57.622 sess: Connection closed. . 2021-11-21 04:04:57.622 Request ends, status 503 class 5xx, error line: . 2021-11-21 04:04:57.622 503 Service Unavailable < 2021-11-21 04:04:57.622 Service Unavailable . 2021-11-21 04:04:57.622 Request ends. . 2021-11-21 04:04:57.622 sess: Destroying session. * 2021-11-21 04:04:57.622 (ExtException) **XmlParseFailure** . 2021-11-21 04:04:57.622 Asking user: . 2021-11-21 04:04:57.622 Error transferring file 'D:\landcycle\Tiles\landregistry\16\32392\21852.pbf'. (XmlParseFailure)
If I click "Retry", it always works on this attempt and thousand more files are uploaded until Google throws the 503 again. This, of course, happens at 2am and I don't see the error until 9am. So those intervening hours have been lost.
Eventually I will be able to upload all these files. But what would be better if there was a setting and countdown. "Automatically retry failed uploads after X seconds" and WinSCP would continue and automatically close the dialogue and retry itself.
I've attached a short video of the clicking "Retry", and that the file that previously failed then works no problem on the reattempt.
- number_of_files.png (63.81 KB)
- dialog.png (25.12 KB)
