AWS S3: listing directories with tailored AWS S3 API

We started to generate partitioned datasets to S3, so they are now spread in a great number of folders. And usually have 1 file in them.

And now I want to download/delete these folders and "listing" stage takes too much time going through folders 1-by-1.

AWS has an API ListObjectsV2 https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
That should speed things up a lot. And I think it is also suitable/tailored to get all the keys from "sub-folders" quite easily.

Thanks for the great tool, btw.

Thanks for your suggestion.
We will see, if more people ask for this.
Btw, I do not see how ListObjectsV2 helps here, comparing to ListObjects that WinSCP us using now. Can you explain that?

Hey, Martin.

good question about v1 vs v2.
aws suggests using v2 over v1:

This action has been revised. We recommend that you use the newer version, ListObjectsV2, when developing applications. For backward compatibility, Amazon S3 continues to support ListObjects.

apparently,

This (v1) operation is not supported by directory buckets.

---

In my code I use ListObjectsV2 everywhere. i.e. I follow the recommendation and it is "still" supported with the new features.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html

I don't use directory buckets (yet), but AWS writes directory buckets are kinda not replicated across availability zones and are much faster. (probably for temp distributed storage and AI stuff)

And anyway, V1orV2 I don't mind, but listing all keys in a prefix for a "delete/download" - should significantly speed things up. For now I switch to aws-console to delete a hefty folder.

Sure, I understand that retrieving "recursive" listing using one call will make this quite faster. But it is always difficult to use protocol-specific hacks/improvements/optimizations in multi-protocol application like WinSCP (comparing to dedicated aws).