Filemask not Working on Folders :: Support Forum

Jan1111
Joined:: 2022-06-09
Posts:: 11
Location:: Germany

Filemask not Working on Folders

2022-06-23 11:17

Hello Guys,

Im using WinSCP with PowerShell-script to download a couple of folders with files in it.
So FTP looks like this:

/remotepath/folder1/file1.xml
/remotepath/folder1/file1.pdf
/remotepath/folder2/file2.xml
/remotepath/folder2/file2.pdf

... and so on

Im running the command twice. First run to download the complete folders including the files. The second run with filemask and "true" parameter, to only delete the files in the remote folders. And don't know any better way but I think it should work...

The second run is the one not working fine, as it tries to delete the folders (what's not allowed per accessrights) and then the script breaks after the first folder.

My Filter is this: -filemask="*.xml,*.pdf"
The filter itself is working fine, it deltes xml or pdfs and ignores other files

My Script is just like this:

# Load WinSCP .NET assembly
Add-Type -Path "D:\Script\WinSCPnet.dll"
 
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Sftp
    HostName = "server"
    UserName = "user"
    Password = "pass"
    SshHostKeyFingerprint = "ssh-rsa 2048 lalalala"
}
 
$session = New-Object WinSCP.Session
$session2 = New-Object WinSCP.Session
$session2.SessionLogPath = "D:\Script\log\sessionlog.txt"
 
 
# Connect
$session.Open($sessionOptions)
$session2.Open($sessionOptions)
 
#Transfer Options
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary
 
# Download files    
$session.GetFiles("/remotePath/*", "D:\localPath\", $False, $transferOptions)
 
#Transfer Options2
$transferOptions2 = New-Object WinSCP.TransferOptions
$transferOptions2.TransferMode = [WinSCP.TransferMode]::Binary
$transferOptions2.FileMask = "*.xml,*.pdf"
 
# Delete *.xml   
$session2.GetFiles("/remotePath/*", "D:\localPath\", $True, $transferOptions2)

As you can see from the log, it tries to delete the folders as well:

> 2022-06-23 11:48:26.811 Script: get -delete -nopermissions -preservetime -transfer="binary" -filemask="*.xml,*.pdf"  -- "/RemotePath/*" "C:\LocalPath\"
. 2022-06-23 11:48:26.811 Listing directory "/RemotePath".
> 2022-06-23 11:48:26.811 Type: SSH_FXP_OPENDIR, Size: 42, Number: 267
< 2022-06-23 11:48:26.826 Type: SSH_FXP_HANDLE, Size: 41, Number: 267
> 2022-06-23 11:48:26.826 Type: SSH_FXP_READDIR, Size: 41, Number: 524
< 2022-06-23 11:48:26.826 Type: SSH_FXP_NAME, Size: 283, Number: 524
> 2022-06-23 11:48:26.842 Type: SSH_FXP_READDIR, Size: 41, Number: 780
. 2022-06-23 11:48:26.842 Reading symlink ".".
> 2022-06-23 11:48:26.842 Type: SSH_FXP_READLINK, Size: 44, Number: 1043
> 2022-06-23 11:48:26.842 Type: SSH_FXP_STAT, Size: 48, Number: 1297
< 2022-06-23 11:48:26.842 Type: SSH_FXP_STATUS, Size: 42, Number: 780
. 2022-06-23 11:48:26.842 Storing reserved response
< 2022-06-23 11:48:26.842 Type: SSH_FXP_NAME, Size: 82, Number: 1043
. 2022-06-23 11:48:26.842 Link resolved to "/../../applications/.../remotepath".
< 2022-06-23 11:48:26.857 Type: SSH_FXP_ATTRS, Size: 59, Number: 1297
> 2022-06-23 11:48:26.857 Type: SSH_FXP_CLOSE, Size: 41, Number: 1540
. 2022-06-23 11:48:26.857 .;L;0;2022-04-26T14:02:18.000Z;3;"lalala" [0];"lala" [0];rwxrwxrwx;1
. 2022-06-23 11:48:26.857 ..;D;0;2022-05-26T22:57:16.000Z;3;"lalala" [0];"lala" [0];rwxr-xr-x;0
. 2022-06-23 11:48:26.857 Folder1;D;0;2022-06-23T09:33:32.000Z;3;"lalala" [0];"lala" [0];rwxr-xr-x;0
. 2022-06-23 11:48:26.857 20220620-012887;D;0;2022-06-23T08:34:09.000Z;3;"lalala" [0];"lala" [0];rwxr-xr-x;0
. 2022-06-23 11:48:26.857 Copying 2 files/directories to local directory "C:\LocalPath\"
. 2022-06-23 11:48:26.857   PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 0100; Resume: S (102400); CalcS: No; Mask: 
. 2022-06-23 11:48:26.857   TM: B; ClAr: No; RemEOF: No; RemBOM: No; CPS: 0; NewerOnly: No; EncryptNewFiles: Yes; ExcludeHiddenFiles: No; ExcludeEmptyDirectories: No; InclM: *.xml,*.pdf; ResumeL: 0
. 2022-06-23 11:48:26.857   AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; *.js; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml
. 2022-06-23 11:48:26.857 File: '/RemotePath/Folder1' [2022-06-23T09:33:32.000Z] [0]
. 2022-06-23 11:48:26.857 Listing directory "/RemotePath/Folder1".
> 2022-06-23 11:48:26.857 Type: SSH_FXP_OPENDIR, Size: 58, Number: 1803
< 2022-06-23 11:48:26.873 Type: SSH_FXP_STATUS, Size: 17, Number: 1540
. 2022-06-23 11:48:26.873 Discarding reserved response
< 2022-06-23 11:48:26.873 Type: SSH_FXP_HANDLE, Size: 41, Number: 1803
> 2022-06-23 11:48:26.873 Type: SSH_FXP_READDIR, Size: 41, Number: 2060
< 2022-06-23 11:48:26.904 Type: SSH_FXP_NAME, Size: 293, Number: 2060
> 2022-06-23 11:48:26.904 Type: SSH_FXP_READDIR, Size: 41, Number: 2316
. 2022-06-23 11:48:26.904 Reading symlink "..".
> 2022-06-23 11:48:26.904 Type: SSH_FXP_READLINK, Size: 43, Number: 2579
> 2022-06-23 11:48:26.904 Type: SSH_FXP_STAT, Size: 47, Number: 2833
< 2022-06-23 11:48:26.920 Type: SSH_FXP_STATUS, Size: 42, Number: 2316
. 2022-06-23 11:48:26.920 Storing reserved response
< 2022-06-23 11:48:26.920 Type: SSH_FXP_NAME, Size: 82, Number: 2579
. 2022-06-23 11:48:26.920 Link resolved to "/../../applications/.../remotepath".
< 2022-06-23 11:48:26.920 Type: SSH_FXP_ATTRS, Size: 59, Number: 2833
> 2022-06-23 11:48:26.920 Type: SSH_FXP_CLOSE, Size: 41, Number: 3076
. 2022-06-23 11:48:26.920 .;D;0;2022-06-23T09:33:32.000Z;3;"lalala" [0];"lala" [0];rwxr-xr-x;0
. 2022-06-23 11:48:26.920 ..;L;0;2022-04-26T14:02:18.000Z;3;"lalala" [0];"lala" [0];rwxrwxrwx;1
. 2022-06-23 11:48:26.920 test.pdf;-;10784;2022-06-23T09:33:32.000Z;3;"lalala" [0];"lala" [0];rw-r--r--;0
. 2022-06-23 11:48:26.920 File123.xml;-;10784;2022-06-23T09:33:32.000Z;3;"lalala" [0];"lala" [0];rw-r--r--;0
. 2022-06-23 11:48:26.920 File: '/RemotePath/Folder1/test.pdf' [2022-06-23T09:33:32.000Z] [10784]
. 2022-06-23 11:48:26.920 Copying "/RemotePath/Folder1/test.pdf" to local directory started.
. 2022-06-23 11:48:26.920 Binary transfer mode selected.
. 2022-06-23 11:48:26.920 Opening remote file.
> 2022-06-23 11:48:26.920 Type: SSH_FXP_OPEN, Size: 80, Number: 3331
< 2022-06-23 11:48:26.936 Type: SSH_FXP_STATUS, Size: 17, Number: 3076
. 2022-06-23 11:48:26.936 Discarding reserved response
< 2022-06-23 11:48:26.967 Type: SSH_FXP_HANDLE, Size: 41, Number: 3331
> 2022-06-23 11:48:26.967 Type: SSH_FXP_FSTAT, Size: 45, Number: 3592
< 2022-06-23 11:48:26.982 Type: SSH_FXP_ATTRS, Size: 67, Number: 3592
. 2022-06-23 11:48:26.982 Confirming overwriting of file.
> 2022-06-23 11:48:26.982 Type: SSH_FXP_READ, Size: 53, Number: 3845
. 2022-06-23 11:48:27.014 Storing reserved response
< 2022-06-23 11:48:27.014 Status code: 1
. 2022-06-23 11:48:27.014 5 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.
> 2022-06-23 11:48:27.014 Type: SSH_FXP_CLOSE, Size: 41, Number: 4612
. 2022-06-23 11:48:27.014 1 requests to fill 1 data gaps were issued.
. 2022-06-23 11:48:27.014 Preserving timestamp [2022-06-23T09:33:32.000Z]
. 2022-06-23 11:48:27.014 Transfer done: '/RemotePath/Folder1/test.pdf' => 'C:\LocalPath\Folder1\test.pdf' [10514]
. 2022-06-23 11:48:27.014 Deleting file "/RemotePath/Folder1/test.pdf".
> 2022-06-23 11:48:27.014 Type: SSH_FXP_REMOVE, Size: 67, Number: 4877
< 2022-06-23 11:48:27.029 Type: SSH_FXP_STATUS, Size: 17, Number: 4612
. 2022-06-23 11:48:27.029 Discarding reserved response
< 2022-06-23 11:48:27.029 Type: SSH_FXP_STATUS, Size: 17, Number: 4877
< 2022-06-23 11:48:27.029 Status code: 0
. 2022-06-23 11:48:27.029 File: '/RemotePath/Folder1/File123.xml' [2022-06-23T09:33:32.000Z] [10784]
. 2022-06-23 11:48:27.029 Copying "/RemotePath/Folder1/File123.xml" to local directory started.
. 2022-06-23 11:48:27.029 Binary transfer mode selected.
. 2022-06-23 11:48:27.029 Opening remote file.
> 2022-06-23 11:48:27.029 Type: SSH_FXP_OPEN, Size: 88, Number: 5123
< 2022-06-23 11:48:27.076 Type: SSH_FXP_HANDLE, Size: 41, Number: 5123
> 2022-06-23 11:48:27.076 Type: SSH_FXP_FSTAT, Size: 45, Number: 5384
< 2022-06-23 11:48:27.092 Type: SSH_FXP_ATTRS, Size: 67, Number: 5384
. 2022-06-23 11:48:27.092 Confirming overwriting of file.
> 2022-06-23 11:48:27.092 Type: SSH_FXP_READ, Size: 53, Number: 5637
. 2022-06-23 11:48:27.107 Storing reserved response
< 2022-06-23 11:48:27.107 Status code: 1
. 2022-06-23 11:48:27.107 5 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.
> 2022-06-23 11:48:27.107 Type: SSH_FXP_CLOSE, Size: 41, Number: 6404
. 2022-06-23 11:48:27.107 1 requests to fill 1 data gaps were issued.
. 2022-06-23 11:48:27.107 Preserving timestamp [2022-06-23T09:33:32.000Z]
. 2022-06-23 11:48:27.107 Transfer done: '/RemotePath/Folder1/File123.xml' => 'C:\LocalPath\Folder1\File123.xml' [10514]
. 2022-06-23 11:48:27.107 Deleting file "/RemotePath/Folder1/File123.xml".
> 2022-06-23 11:48:27.107 Type: SSH_FXP_REMOVE, Size: 75, Number: 6669
< 2022-06-23 11:48:27.139 Type: SSH_FXP_STATUS, Size: 17, Number: 6404
. 2022-06-23 11:48:27.139 Discarding reserved response
< 2022-06-23 11:48:27.139 Type: SSH_FXP_STATUS, Size: 17, Number: 6669
< 2022-06-23 11:48:27.139 Status code: 0
. 2022-06-23 11:48:27.139 Deleting file "/RemotePath/Folder1".
> 2022-06-23 11:48:27.139 Type: SSH_FXP_RMDIR, Size: 58, Number: 6927
< 2022-06-23 11:48:27.154 Type: SSH_FXP_STATUS, Size: 34, Number: 6927
< 2022-06-23 11:48:27.154 Status code: 3, Message: 6927, Server: Permission denied, Language:  
. 2022-06-23 11:48:27.154 Asking user:
. 2022-06-23 11:48:27.154 Error deleting file '/RemotePath/Folder1'. ("Permission denied.
. 2022-06-23 11:48:27.154 Error code: 3
. 2022-06-23 11:48:27.154 Error message from server: Permission denied")
< 2022-06-23 11:48:27.154 Script: Error deleting file '/RemotePath/Folder1'.
< 2022-06-23 11:48:27.154 Script: Permission denied.
< 2022-06-23 11:48:27.154 Error code: 3
< 2022-06-23 11:48:27.154 Error message from server: Permission denied
. 2022-06-23 11:48:27.154 Answer: Abort
. 2022-06-23 11:48:27.154 Script: Failed

I would be fine with two solutions, either the filemask starts working fine and ignores the folders or the script just continues on errors.

Thanks alot

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,478
Location:: Prague, Czechia

Re: Filemask not Working on Folders

2022-06-27

First, if you want to remove files, use Session.RemoveFile. That's the only correct solution.
Anyway, if you want skip subfolders, see:
How do I transfer (or synchronize) directory non-recursively?
Btw, your script should not fail on any errors, as you are not checking for errors at all. Are you sure the logs comes from the code you have posted?

Reply with quote

Jan1111
Joined:: 2022-06-09
Posts:: 11
Location:: Germany

Re: Filemask not Working on Folders

2022-09-07 12:31

martin wrote:

Btw, your script should not fail on any errors, as you are not checking for errors at all. Are you sure the logs comes from the code you have posted?

Yes, I am sure and it fails because I'm not allowed to delete the folder. I found something else out.
If you have different types of files in the subfolder, lets say .xml and .pdf and you use the file filter for .xml ($transferOptions.FileMask = "*.xml"), it's processing all the folders cause in every folder, there is at least one file left with the extension .pdf.
It only crashes when you delete the last file in a folder, then (for some reason), it tries to delete the subfolder as well and is not caring about the FileMask.

Reply with quote

Jan1111
Joined:: 2022-06-09
Posts:: 11
Location:: Germany

Re: Filemask not Working on Folders

2022-09-07 12:36

martin wrote:

First, if you want to remove files, use Session.RemoveFile. That's the only correct solution.

To be honest, I have no idea how to do this. I would have to enumerate all files in all subfolders or at least all subfolders on the server. Do you have a peace of code as an example?
Thanks alot

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,478
Location:: Prague, Czechia

Re: Filemask not Working on Folders

2022-09-08

Jan1111 wrote:

If you have different types of files in the subfolder, lets say .xml and .pdf and you use the file filter for .xml ($transferOptions.FileMask = "*.xml"), it's processing all the folders cause in every folder, there is at least one file left with the extension .pdf.
It only crashes when you delete the last file in a folder, then (for some reason), it tries to delete the subfolder as well and is not caring about the FileMask.

Sorry, I'm lost. If you want me to investigate this, please post the code and session log file.

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,478
Location:: Prague, Czechia

Re: Filemask not Working on Folders

2022-09-08

Jan1111 wrote:

To be honest, I have no idea how to do this. I would have to enumerate all files in all subfolders or at least all subfolders on the server. Do you have a peace of code as an example?
Thanks alot

The Session.GetFiles returns list of all transferred files in TransferOperationResult.Transfers. Just iterate it, and call Session.RemoveFile for each transferred files.

Btw, note that this all makes sense only, if you goal is to delete the files only after all files are transferred. If you are ok with deleting the files as they are transferred, just use Session.GetFiles with remove = false straight away.

Reply with quote

Jan1111
Joined:: 2022-06-09
Posts:: 11
Location:: Germany

Re: Filemask not Working on Folders

2022-09-13 10:49

martin wrote:

Btw, note that this all makes sense only, if you goal is to delete the files only after all files are transferred. If you are ok with deleting the files as they are transferred, just use Session.GetFiles with remove = false straight away.

I think you mean remove = true. That's not working because it tries to delete the folder as well after deleting the last file in the folder even when using file filter.

Im using this solution with PowerShell and it works perfectly:

#Enum Files
$RemoteFilesPDF = $session.EnumerateRemoteFiles("/remotePath/", "*.pdf", [WinSCP.EnumerationOptions]::AllDirectories)
$RemoteFilesXML = $session.EnumerateRemoteFiles("/remotePath/", "*.xml", [WinSCP.EnumerationOptions]::AllDirectories)
   
# PDF delete
foreach ($fileToDelete in $RemoteFilesPDF)
{
    $session.RemoveFile($fileToDelete.FullName)
} # end foreach
        
# XML delete
foreach ($fileToDelete2 in $RemoteFilesXML)
{
    $session.RemoveFile($fileToDelete2.FullName)
} # end foreach

Many thanks for your help and patience!

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,478
Location:: Prague, Czechia

Re: Filemask not Working on Folders

2022-09-13

Your code "works". But it is inefficient and it is not transactionally safe. You enumerate remote folder three times. And if new file is added to the remote folders between GetFiles and EnumerateRemoteFiles, you may delete a file that was not downloaded. At the very least, do EnumerateRemoteFiles before GetFiles. Though once you already enumerate files, instead of separate GetFiles and RemoveFile do GetFileToDirectory with remove = true:

foreach ($file in $RemoteFilesPDF)
{
    $session.GetFileToDirectory($fileToDelete.FullName, "D:\localPath", $True)
} # end foreach

Reply with quote

Jan1111

Re: Filemask not Working on Folders

2022-09-13 12:56

Yes, I enumerate the files before downloading and I check the Download for errors first. But your code is a very good idea. I just don't know if this creates the subfolders before copying the files. I'm gonna test this the next days. Thanks

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,478
Location:: Prague, Czechia

Re: Filemask not Working on Folders

2022-09-15

If there are subfolders, then the code needs to be more complicated.

But then it is still better to do what I have suggested originally:

$result = $session.GetFiles("/remotePath/*", "D:\localPath\")
$result.Check()
 
foreach ($transfer in $result.Transfers)
{
    # Not sure if this condition is needed
    if ($transfer.FileName.EndsWith(".jpg") -or
        $transfer.FileName.EndsWith(".pdf"))
    {
        $session.RemoveFile($transfer.FileName)
    }
}