File Synchronisation Between Mac & S3

In this tutorial you will learn how create your own file synchronisation service that syncs files from your local directory on your Mac to an s3 bucket. Along the way you’ll learn how to use cron to schedule jobs based on a time interval you set. Ever wondered how Dropbox & OneDrive work ? Lets give it a go and we’ll create something similar.

Step 1: Create an AWS IAM user

-> Select IAM under Security, Identity & Compliance from the AWS Management Console.
-> Select Users from the left hand pane.
-> Click “Add user”, enter a username, for this tutorial use “AWS_S3_SYNC”
-> Select Programmatic access
-> Click “Next: Permissions”
-> Click “Attach existing policies directly”
-> Select “Administrator Access” then click “Next: Tags”
-> Click “Next: Review”
-> Click “Create user”
-> Click download.csv to download the newly created user security credentials.

Step 2: Install and configure the AWS ClI

-> Open terminal
-> If you have yet to install the AWS CLI follow the steps from the below link:

https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html#install-bundle-other-os

-> Type aws configure and press enter
-> Open the csv you had previously downloaded in Step 1
-> Copy & paste the Access Key ID from the csv into the terminal window and press enter
-> Copy & paste the Secret Access Key from the csv into the terminal window and press enter
-> Press Enter when prompted for “Default region name” & “Default output format”

Step 3: Using the AWS CLI with S3 & setting up a cron job

-> To create a bucket using the cli use the below command :

aws s3 mb s3://bucketname

-> For this tutorial use s3://architect-yourname-backup-bucket

aws s3 mb s3://architect-munashe-backup-bucket

-> Create a new folder that will be used to store files that will be synced to S3, I created a folder under documents named AWS Sync Folder, you can name the folder accordingly.

-> In the terminal window type crontab -e
-> type i, this will allow you to insert text

* * * * * /usr/local/bin/aws s3sync s3://architect-munashe-backup-bucket /Users/munashe/Documents/AWS\ Sync\ Folder


* * * * * /usr/local/bin/aws s3 sync /Users/munashe/Documents/AWS\ Sync\ Folder s3://architect-munashe-backup-bucket

-> Press the escape key, type “:wq” and press enter

Congratulations, you’ve successfully setup asynchronous file synchronisation between a local folder on your mac and the cloud (s3). You’ve replicated a Dropbox or OneDrive like service. I hope you’ve learnt something new. You can use this service to backup important files, pictures that you feel that you can’t afford to lose.

Please delete the cron as this will run every minute, every hour, every day, every month, every day of the week and you’ll incur quite a hefty bill from AWS. Alternatively you can adjust the cron to run less frequently to reduce the costs.

See below link for an explanation of cron:

https://en.wikipedia.org/wiki/Cron


Comments: