PS script to synchronize and S3 bucket to a local share :: Support Forum

Guest

PS script to synchronize and S3 bucket to a local share

2021-05-03 17:04

I'm trying to write a script that will synchronize a folder on AWS to my local server. I've got the script to open the secure session but can't seem to find any documentation that works with the synchronize function. I've generated the code in WinSCP but not sure of the syntax to use for synchronizing. I don't want it to delete any files/folders wither, just synch the two folders. Here's the script I have so far:

& "C:\Program Files (x86)\WinSCP\WinSCP.com" `
  /log="C:\winscplogs\WinSCP.log" /ini=nul `
  /command `
    "open s3://AKIA2XNQT5DDVSBAGMZL:TBdRqCTOwA4r5cA3t8c3KbSg8ijN3yCoCLgGpMrt@s3.amazonaws.com/ao-prod-sea-storage/SEA_4598723/" `
    -synchronize local [ e:\call center\call center recordings [ /ao-prod-sea-storage/SEA_4598723/ ] ] `
    "exit"
$winscpResult = $LastExitCode
if ($winscpResult -eq 0)
{
  Write-Host "Success"
}
else
{
  Write-Host "Error"
}
exit $winscpResult

Right now it opens the session and just sits there. I'm not getting any information in the log files. It just shows the session has been started and is active. It doesn't error out or give any info in the log file.

Any ideas what I'm doing wrong?

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,478
Location:: Prague, Czechia

Re: PS script to synchronize and S3 bucket to a local share

2021-05-04

    "synchronize local `"`"e:\call center\call center recordings`"`" `"`"/ao-prod-sea-storage/SEA_4598723/`"`" `

See https://winscp.net/eng/docs/commandline#syntax

Though in PowerShell, it's easier to use WinSCP .NET assembly:
https://winscp.net/eng/docs/library_powershell

Reply with quote

roameeo
Joined:: 2021-05-03
Posts:: 3

Same thing

2021-05-04 13:15

I used:

winscp.exe /synchronize // "C:\Users\martin\Documents\MySite" /root

and am still not seeing any synchs. Here's the log file:

2021-05-04 07:04:28.432 --------------------------------------------------------------------------
. 2021-05-04 07:04:28.432 WinSCP Version 5.17.10 (Build 11087) (OS 10.0.14393 - Windows Server 2016 Datacenter)
. 2021-05-04 07:04:28.432 Configuration: nul
. 2021-05-04 07:04:28.432 Log level: Normal
. 2021-05-04 07:04:28.432 Local account: AFFILIATEDBANK\rccallsrvacct
. 2021-05-04 07:04:28.432 Working directory: C:\Users\rccallsrvacct\Desktop
. 2021-05-04 07:04:28.432 Process ID: 3252
. 2021-05-04 07:04:28.432 Command-line: "C:\Program Files (x86)\WinSCP\WinSCP.exe" /console=5.17.10 /consoleinstance=_6220_419 "/log=C:\winscplogs\WinSCP.log" "/ini=nul" "/command" "open s3://AKIA2XNQT5DDVSBAGMZL:***@s3.amazonaws.com/ao-prod-sea-storage/SEA_4598723/"
. 2021-05-04 07:04:28.432 Time zone: Current: GMT-5, Standard: GMT-6 (Central Standard Time), DST: GMT-5 (Central Daylight Time), DST Start: 3/14/2021, DST End: 11/7/2021
. 2021-05-04 07:04:28.432 Login time: Tuesday, May 4, 2021 7:04:28 AM
. 2021-05-04 07:04:28.432 --------------------------------------------------------------------------
. 2021-05-04 07:04:28.432 Script: Retrospectively logging previous script records:
> 2021-05-04 07:04:28.432 Script: open s3://AKIA2XNQT5DDVSBAGMZL:***@s3.amazonaws.com/ao-prod-sea-storage/SEA_4598723/
. 2021-05-04 07:04:28.432 --------------------------------------------------------------------------
. 2021-05-04 07:04:28.432 Session name: AKIA2XNQT5DDVSBAGMZL@s3.amazonaws.com (Ad-Hoc site)
. 2021-05-04 07:04:28.432 Host name: s3.amazonaws.com (Port: 443)
. 2021-05-04 07:04:28.432 User name: AKIA2XNQT5DDVSBAGMZL (Password: Yes, Key file: No, Passphrase: No)
. 2021-05-04 07:04:28.432 Transfer Protocol: S3
. 2021-05-04 07:04:28.432 Proxy: None
. 2021-05-04 07:04:28.432 HTTPS: Yes
. 2021-05-04 07:04:28.432 S3: URL Style: Virtual Host
. 2021-05-04 07:04:28.432 TLS/SSL versions: TLSv1.0-TLSv1.2
. 2021-05-04 07:04:28.432 Local directory: default, Remote directory: /ao-prod-sea-storage/SEA_4598723/, Update: Yes, Cache: Yes
. 2021-05-04 07:04:28.432 Cache directory changes: Yes, Permanent: Yes
. 2021-05-04 07:04:28.432 Recycle bin: Delete to: No, Overwritten to: No, Bin path: 
. 2021-05-04 07:04:28.432 --------------------------------------------------------------------------
. 2021-05-04 07:04:28.432 Trying to open directory "/ao-prod-sea-storage/SEA_4598723/".
. 2021-05-04 07:04:28.432 Unknown bucket "ao-prod-sea-storage", will detect its region (and service endpoint)
. 2021-05-04 07:04:28.432 HTTP session to https://ao-prod-sea-storage.s3.amazonaws.com:443 begins.
. 2021-05-04 07:04:28.432 ssl: SNI enabled by default.
. 2021-05-04 07:04:28.432 Sending request headers:
. 2021-05-04 07:04:28.432 GET /?prefix=SEA_4598723%2F&delimiter=%2F&max-keys=1 HTTP/1.1
 
. 2021-05-04 07:04:28.432 User-Agent: WinSCP/5.17.10 neon/0.31.1
 
. 2021-05-04 07:04:28.432 Keep-Alive: 
 
. 2021-05-04 07:04:28.432 Connection: TE, Keep-Alive
 
. 2021-05-04 07:04:28.432 TE: trailers
 
. 2021-05-04 07:04:28.432 Host: ao-prod-sea-storage.s3.amazonaws.com
 
. 2021-05-04 07:04:28.432 Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
. 2021-05-04 07:04:28.432 x-amz-date: 20210504T120428Z
 
. 2021-05-04 07:04:28.432 x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
. 2021-05-04 07:04:28.432 Sending request-line and headers:
. 2021-05-04 07:04:28.432 Doing DNS lookup on ao-prod-sea-storage.s3.amazonaws.com...
. 2021-05-04 07:04:28.497 req: Connecting to 52.218.170.65:443
. 2021-05-04 07:04:28.552 Doing SSL negotiation.
. 2021-05-04 07:04:28.667 Identity match for 's3.amazonaws.com': good
. 2021-05-04 07:04:28.667 Verifying certificate for "Amazon.com, Inc., Seattle, Washington, US" with fingerprint 90:e0:af:dc:fa:f7:0b:ac:50:bb:fa:43:e1:ec:e2:3d:ce:91:90:47 and 08 failures
. 2021-05-04 07:04:28.727 Certificate verified against Windows certificate store
. 2021-05-04 07:04:28.727 Using TLSv1.2, cipher TLSv1.2: ECDHE-RSA-AES128-GCM-SHA256, 2048 bit RSA
. 2021-05-04 07:04:28.727 Request sent; retry is 0.
. 2021-05-04 07:04:28.777 [status-line] < HTTP/1.1 400 Bad Request
. 2021-05-04 07:04:28.777 Header Name: [x-amz-bucket-region], Value: [us-west-2]
. 2021-05-04 07:04:28.777 Header Name: [x-amz-request-id], Value: [14QPW46F9SYTZADA]
. 2021-05-04 07:04:28.777 Header Name: [x-amz-id-2], Value: [FIoI85Pbcq+u2F3/eSp1PfhwbcJN5RD6TfGOA8JzBcKMrBfWZv8fMKKlixPrZX76O2Z7A9Ajkbo=]
. 2021-05-04 07:04:28.777 Header Name: [content-type], Value: [application/xml]
. 2021-05-04 07:04:28.777 Header Name: [transfer-encoding], Value: [chunked]
. 2021-05-04 07:04:28.777 Header Name: [date], Value: [Tue, 04 May 2021 12:04:28 GMT]
. 2021-05-04 07:04:28.777 Header Name: [server], Value: [AmazonS3]
. 2021-05-04 07:04:28.777 Header Name: [connection], Value: [close]
. 2021-05-04 07:04:28.777 End of headers.
. 2021-05-04 07:04:28.777 End of headers.
. 2021-05-04 07:04:28.777 sess: Closing connection.
. 2021-05-04 07:04:28.777 sess: Connection closed.
. 2021-05-04 07:04:28.777 Request ends, status 400 class 4xx, error line:
. 2021-05-04 07:04:28.777 400 Bad Request
. 2021-05-04 07:04:28.777 The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
. 2021-05-04 07:04:28.777 Extra Details: Region: us-west-2, RequestId: 14QPW46F9SYTZADA, HostId: FIoI85Pbcq+u2F3/eSp1PfhwbcJN5RD6TfGOA8JzBcKMrBfWZv8fMKKlixPrZX76O2Z7A9Ajkbo=
< 2021-05-04 07:04:28.777 <?xml version="1.0" encoding="UTF-8"?>
< 2021-05-04 07:04:28.777 <Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'</Message><Region>us-west-2</Region><RequestId>14QPW46F9SYTZADA</RequestId><HostId>FIoI85Pbcq+u2F3/eSp1PfhwbcJN5RD6TfGOA8JzBcKMrBfWZv8fMKKlixPrZX76O2Z7A9Ajkbo=</HostId></Error>
. 2021-05-04 07:04:28.777 Request ends.
. 2021-05-04 07:04:28.777 sess: Destroying session.
. 2021-05-04 07:04:28.777 Will use region "us-west-2" for bucket "ao-prod-sea-storage" from now on.
. 2021-05-04 07:04:28.777 HTTP session to https://ao-prod-sea-storage.s3.amazonaws.com:443 begins.
. 2021-05-04 07:04:28.777 ssl: SNI enabled by default.
. 2021-05-04 07:04:28.777 Sending request headers:
. 2021-05-04 07:04:28.777 GET /?prefix=SEA_4598723%2F&delimiter=%2F&max-keys=1 HTTP/1.1
 
. 2021-05-04 07:04:28.777 User-Agent: WinSCP/5.17.10 neon/0.31.1
 
. 2021-05-04 07:04:28.777 Keep-Alive: 
 
. 2021-05-04 07:04:28.777 Connection: TE, Keep-Alive
 
. 2021-05-04 07:04:28.777 TE: trailers
 
. 2021-05-04 07:04:28.777 Host: ao-prod-sea-storage.s3.amazonaws.com
 
. 2021-05-04 07:04:28.777 Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
. 2021-05-04 07:04:28.777 x-amz-date: 20210504T120428Z
 
. 2021-05-04 07:04:28.777 x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
. 2021-05-04 07:04:28.777 Sending request-line and headers:
. 2021-05-04 07:04:28.777 Doing DNS lookup on ao-prod-sea-storage.s3.amazonaws.com...
. 2021-05-04 07:04:28.777 req: Connecting to 52.218.170.65:443
. 2021-05-04 07:04:28.832 Doing SSL negotiation.
. 2021-05-04 07:04:28.947 Identity match for 's3.amazonaws.com': good
. 2021-05-04 07:04:28.947 Verifying certificate for "Amazon.com, Inc., Seattle, Washington, US" with fingerprint 90:e0:af:dc:fa:f7:0b:ac:50:bb:fa:43:e1:ec:e2:3d:ce:91:90:47 and 08 failures
. 2021-05-04 07:04:28.972 Certificate verified against Windows certificate store
. 2021-05-04 07:04:28.972 Using TLSv1.2, cipher TLSv1.2: ECDHE-RSA-AES128-GCM-SHA256, 2048 bit RSA
. 2021-05-04 07:04:28.972 Request sent; retry is 0.
. 2021-05-04 07:04:29.182 [status-line] < HTTP/1.1 200 OK
. 2021-05-04 07:04:29.182 Header Name: [x-amz-id-2], Value: [Hp42CLM/8tbwWekZVNTMst/FTQ313S+DOo1jPI8TP8PJ34DWKJEIwEgfYU/FjOxxT/4e3E2Cl0I=]
. 2021-05-04 07:04:29.182 Header Name: [x-amz-request-id], Value: [GS2R457RSHV2NQ71]
. 2021-05-04 07:04:29.182 Header Name: [date], Value: [Tue, 04 May 2021 12:04:30 GMT]
. 2021-05-04 07:04:29.182 Header Name: [x-amz-bucket-region], Value: [us-west-2]
. 2021-05-04 07:04:29.182 Header Name: [content-type], Value: [application/xml]
. 2021-05-04 07:04:29.182 Header Name: [transfer-encoding], Value: [chunked]
. 2021-05-04 07:04:29.182 Header Name: [server], Value: [AmazonS3]
. 2021-05-04 07:04:29.182 Header Name: [connection], Value: [close]
. 2021-05-04 07:04:29.182 End of headers.
. 2021-05-04 07:04:29.182 End of headers.
. 2021-05-04 07:04:29.182 sess: Closing connection.
. 2021-05-04 07:04:29.182 sess: Connection closed.
. 2021-05-04 07:04:29.182 Request ends, status 200 class 2xx, error line:
. 2021-05-04 07:04:29.182 200 OK
< 2021-05-04 07:04:29.182 <?xml version="1.0" encoding="UTF-8"?>
< 2021-05-04 07:04:29.182 <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>ao-prod-sea-storage</Name><Prefix>SEA_4598723/</Prefix><Marker></Marker><NextMarker>SEA_4598723/</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>SEA_4598723/</Key><LastModified>2019-12-05T20:51:02.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>1ad43af4e8fe98b03b543cf927c1821997991f7660fad47fd44cc3b4b2617006</ID><DisplayName>aws-ic-prod</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>
. 2021-05-04 07:04:29.182 Request ends.
. 2021-05-04 07:04:29.182 sess: Destroying session.
. 2021-05-04 07:04:29.182 --------------------------------------------------------------------------
. 2021-05-04 07:04:29.182 Using S3 protocol.
. 2021-05-04 07:04:29.182 Doing startup conversation with host.
. 2021-05-04 07:04:29.182 Changing directory to "/ao-prod-sea-storage/SEA_4598723/".
. 2021-05-04 07:04:29.182 Trying to open directory "/ao-prod-sea-storage/SEA_4598723/".
. 2021-05-04 07:04:29.182 HTTP session to https://ao-prod-sea-storage.s3.amazonaws.com:443 begins.
. 2021-05-04 07:04:29.182 ssl: SNI enabled by default.
. 2021-05-04 07:04:29.182 Sending request headers:
. 2021-05-04 07:04:29.182 GET /?prefix=SEA_4598723%2F&delimiter=%2F&max-keys=1 HTTP/1.1
 
. 2021-05-04 07:04:29.182 User-Agent: WinSCP/5.17.10 neon/0.31.1
 
. 2021-05-04 07:04:29.182 Keep-Alive: 
 
. 2021-05-04 07:04:29.182 Connection: TE, Keep-Alive
 
. 2021-05-04 07:04:29.182 TE: trailers
 
. 2021-05-04 07:04:29.182 Host: ao-prod-sea-storage.s3.amazonaws.com
 
. 2021-05-04 07:04:29.182 Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
. 2021-05-04 07:04:29.182 x-amz-date: 20210504T120429Z
 
. 2021-05-04 07:04:29.182 x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
. 2021-05-04 07:04:29.182 Sending request-line and headers:
. 2021-05-04 07:04:29.182 Doing DNS lookup on ao-prod-sea-storage.s3.amazonaws.com...
. 2021-05-04 07:04:29.182 req: Connecting to 52.218.170.65:443
. 2021-05-04 07:04:29.237 Doing SSL negotiation.
. 2021-05-04 07:04:29.352 Identity match for 's3.amazonaws.com': good
. 2021-05-04 07:04:29.352 Verifying certificate for "Amazon.com, Inc., Seattle, Washington, US" with fingerprint 90:e0:af:dc:fa:f7:0b:ac:50:bb:fa:43:e1:ec:e2:3d:ce:91:90:47 and 08 failures
. 2021-05-04 07:04:29.377 Certificate verified against Windows certificate store
. 2021-05-04 07:04:29.377 Using TLSv1.2, cipher TLSv1.2: ECDHE-RSA-AES128-GCM-SHA256, 2048 bit RSA
. 2021-05-04 07:04:29.377 Request sent; retry is 0.
. 2021-05-04 07:04:29.565 [status-line] < HTTP/1.1 200 OK
. 2021-05-04 07:04:29.565 Header Name: [x-amz-id-2], Value: [3RidIUTr9vkB3GVgXAepyd/yDySJLJrxwok2rFSVWqBVTIYh37iRVbMDAvf+w5rsvlYAhkMq/zY=]
. 2021-05-04 07:04:29.565 Header Name: [x-amz-request-id], Value: [GS2VWNVSRCE45YZ9]
. 2021-05-04 07:04:29.565 Header Name: [date], Value: [Tue, 04 May 2021 12:04:30 GMT]
. 2021-05-04 07:04:29.565 Header Name: [x-amz-bucket-region], Value: [us-west-2]
. 2021-05-04 07:04:29.565 Header Name: [content-type], Value: [application/xml]
. 2021-05-04 07:04:29.565 Header Name: [transfer-encoding], Value: [chunked]
. 2021-05-04 07:04:29.565 Header Name: [server], Value: [AmazonS3]
. 2021-05-04 07:04:29.565 Header Name: [connection], Value: [close]
. 2021-05-04 07:04:29.565 End of headers.
. 2021-05-04 07:04:29.565 End of headers.
. 2021-05-04 07:04:29.565 sess: Closing connection.
. 2021-05-04 07:04:29.565 sess: Connection closed.
. 2021-05-04 07:04:29.565 Request ends, status 200 class 2xx, error line:
. 2021-05-04 07:04:29.565 200 OK
< 2021-05-04 07:04:29.565 <?xml version="1.0" encoding="UTF-8"?>
< 2021-05-04 07:04:29.565 <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>ao-prod-sea-storage</Name><Prefix>SEA_4598723/</Prefix><Marker></Marker><NextMarker>SEA_4598723/</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>SEA_4598723/</Key><LastModified>2019-12-05T20:51:02.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>1ad43af4e8fe98b03b543cf927c1821997991f7660fad47fd44cc3b4b2617006</ID><DisplayName>aws-ic-prod</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>
. 2021-05-04 07:04:29.565 Request ends.
. 2021-05-04 07:04:29.565 sess: Destroying session.
. 2021-05-04 07:04:29.565 Getting current directory name.
. 2021-05-04 07:04:29.565 Startup conversation with host finished.
< 2021-05-04 07:04:29.565 Script: Active session: [1] AKIA2XNQT5DDVSBAGMZL@s3.amazonaws.com

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,478
Location:: Prague, Czechia

Re: Same thing

2021-05-07

The log file is for winscp.com ... /command "open ..." not for winscp.exe /synchronize ....
Please post consistent information.
It also completely misses any synchronize command.
Please post your new complete PowerShell script.

Reply with quote

roameeo
Joined:: 2021-05-03
Posts:: 3

Script

2021-05-07 13:13

Here's the script I'm attempting to get to work:

& "C:\Program Files (x86)\WinSCP\WinSCP.com" `
  /log="C:\winscplogs\WinSCP.log" /ini=nul `
  /command `
    "open s3://<my access key>:<My secret key>@s3.amazonaws.com/ao-prod-sea-storage/SEA_4598723/" 
 
    /synchronize local // "e:\call center\call center recordings"  /ao-prod-sea-storage/SEA_4598723/  `
 
    "exit"
 
$winscpResult = $LastExitCode
if ($winscpResult -eq 0)
{
  Write-Host "Success"
}
else
{
  Write-Host "Error"
}
 
exit $winscpResult

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,478
Location:: Prague, Czechia

Re: Script

2021-05-11

I've already wrote above, that is should be:

    "synchronize local `"`"e:\call center\call center recordings`"`" `"`"/ao-prod-sea-storage/SEA_4598723/`"`" `

And remove the empty lines (or escape them).

Reply with quote

roameeo

Got it working

2021-05-11 13:05

I was able to get it working using AWS CLI. Thank you for all the input!

Reply with quote

PS script to synchronize and S3 bucket to a local share