Downloading folders from aws s3, cp or sync?

Using aws s3 cp from the AWS Command-Line Interface (CLI) will require the --recursive parameter to copy multiple files.

aws s3 cp s3://myBucket/dir localdir --recursive

The aws s3 sync command will, by default, copy a whole directory. It will only copy new/modified files.

aws s3 sync s3://mybucket/dir localdir

Just experiment to get the result you want.

Documentation:

cp command

sync command

---

Just used version 2 of the AWS CLI. For the s3 option, there is also a --dryrun option now to show you what will happen:

aws s3 --dryrun cp s3://bucket/filename /path/to/dest/folder --recursive

---

In the case you want to download a single file, you can try the following command:

aws s3 cp s3://bucket/filename /path/to/dest/folder

---

In case you need to use another profile, especially cross account. you need to add the profile in the config file

[profile profileName]
region = us-east-1
role_arn = arn:aws:iam::XXX:role/XXXX
source_profile = default

and then if you are accessing only a single file

aws s3 cp s3://crossAccountBucket/dir localdir --profile profileName

---

You've many options to do that, but the best one is using the AWS CLI.

Here's a walk-through:

Download and install AWS CLI in your machine: Install the AWS CLI using the MSI Installer (Windows). Install the AWS CLI using the Bundled Installer for Linux, OS X, or Unix. Configure AWS CLI:

https://i.stack.imgur.com/1UvKq.png

Make sure you input valid access and secret keys, which you received when you created the account.

Sync the S3 bucket using: aws s3 sync s3://yourbucket/yourfolder /local/path

In the above command, replace the following fields:

yourbucket/yourfolder >> your S3 bucket and the folder that you want to download.

/local/path >> path in your local system where you want to download all the files.

---

sync method first lists both source and destination paths and copies only differences (name, size etc.).

cp --recursive method lists source path and copies (overwrites) all to the destination path.

If you have possible matches in the destination path, I would suggest sync as one LIST request on the destination path will save you many unnecessary PUT requests - meaning cheaper and possibly faster.

---