If you are faced to transfer large amounts of data into the AWS cloud, you have to make sure that the data somehow get into S3 when you switch to a cloud service like S3. There are several ways to do this, but one of them is to copy the files using the AWS CLI. There are two commands that enable the data transfer: copy and sync.

While files are simply copied using “aws s3 cp”, “aws s3 sync” allows, for example, synchronization of a local directory with AWS S3. During synchronization, only changed files are transferred, which saves considerable resources with large amounts of data. The basic call for this is:

aws s3 sync . s3://<bucket-name>

This copies all data of the current directory (.) to the specified bucket. Before you do this you have to be connected to the S3 Bucket (“aws configure”). Like all parameters, “sync” also allows other functions to be called, for example to explicitly consider or omit certain files and directories. Here is an example in which only all files / directories are transferred that start with “1”:

aws s3 sync . s3://<bucket-name> --exclude '*' --include "1*"

Incidentally, all S3 commands of the AW CLI can be viewed in the official documentation. For some applications it can also be useful to throttle the maximum transmission speed to S3.