Post a reply

Options
Add an Attachment

If you do not want to add an Attachment to your Post, please leave the Fields blank.

(maximum 10 MB; please compress large files; only common media, archive, text and programming file formats are allowed)

Options

Topic review

martin

The GUI and this code works completely differently. You cannot compare it. Though I'd say that the GUI might be even slower. It just does not have the delays between individual folders, but one huge delay at the very beginning. If you have different experiences, we would need some log files.
Guest

Thanks, I have made that change in the script and so far it appears to not be timing out anymore, but I do often see a transfer/delay lag as it moves between the different folders in the collection. I see that winscp will download across the batch sizes for all the say 1000 files in the folder but then it will lag on I guess the MoveNext in the batch that dives into the next folder. I definitely don't see this delay in the GUI. I am wondering if its something to do with memory management when using powershell and the .net assembly. Any thoughts? Thanks for all the support up to this point!
Guest

I have been testing with the .NET assembly via PowerShell. I am using 5 batches and it seems to be working but at various points in running through a couple of transfers I get this following error.
MethodInvocationException:

Line |
  16 |                              if (!($using:filesEnumerator).MoveNext())
     |                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "MoveNext" with "0" argument(s): "Timeout waiting for WinSCP to respond"

My data is like 30+ folders with each folder roughly having 1000 files in each. I think the fileEnumerator is bogged down and not responding fast enough and eventually times out. I looked at the IEnumerator.MoveNext and it doesn't appear that I can feed any parameters to extend the timeout. Any thoughts?
jwomackgsa

Using background queue parallel transfers via scripting or other options

I am building a script that will go out and pull / delete all the files on the remote endpoint that were posted that day. It usually somewhere around 40,000 files no larger then a couple MB. When I run this in the GUI, I can use the 8 threads queue setting to make the download go much faster, however when I pass this queue setting as a raw setting it doesn't seem to do anything. What are my options here? I have no issues with doing PowerShell if it can be accomplished that way. I would like to have like 8 download threads at a time. If my current in-progress WinSCP script has something like
get -delete *

Can I just run the same winscp.com commandline in parallel 8 times to achieve the same effect?
The data is structure like.
\<timestamped_folder_name>

  - all the files
\<another_folder_name>
  - lots more files