By Thomas Underwood 🕤 Updated on June 16, 2022 at 9:30 am
Suppose that there are two separate Amazon AWS accounts, each with its own S3 storage. These two S3 buckets need to be synchronized between Amazon accounts. To do this, we will use Automation Workshop no-code solution. No scripting, no PowerShell, no AWS CLI required.
The files are synchronized between buckets automagically like when using Dropbox or Google Drive's Backup and Sync.
Overview
Usually, syncing two Amazon S3 accounts involves changing access permissions for one or even both accounts, however, with Automation Workshop it is very easy to synchronize two buckets in two separate accounts without changing object access permissions or any AWS policies.
We are going to do the syncing process in two steps. In the first step we are going to copy files from the first Amazon account to a local HDD. Then, in the second step, we auto-sync files from the local folder to a second Amazon account.
To save the network bandwidth, Automation Workshop is only syncing files that was modified since the last session. This mechanism is similar to the popular Linux utility rsync, but this one is designed especially for Amazon S3 buckets.
How to create a Task, that will sync two Amazon S3 buckets automatically? First, create a new Task and add built-in Task Scheduler Trigger, that will launch the job hourly. Then, add Connect to Amazon S3 Action, give it a name First S3 Account, and use Access key ID and Secret access key from the first account you want to synchronize.
Then, add Synchronize Directories Action, set Local folder to some existing location on your HDD. Note that this location must exist, it should be empty, and it must be the same in both steps. Also, set the Remote directory to the bucket and path you want to synchronize. Set transfer direction to Download.
Now, that we have synced files from the first account to local computer, use the similar steps to sync files back to cloud—to the second account. Add another Connect to Amazon S3 Action, provide credentials from the second account, and name it Second S3 Account.
Then, add another Synchronize Directories Action, provide the same Local folder as in previous step. Also, specify /bucket-name/path to the second account, and change Transfer setting to Upload, and select Second S3 Account under Use connection setting.
How to send an email on successful or failed S3 sync? Optionally, you can set up email notifications. The notifications will be sent in two cases—when synchronization task finishes successfully, and when it fails. In case it fails, the email will contain the last error message, such as no internet connection or similar. More detailed information about all the events is available in the Log Manager.
Summary
The task is up and running! It consists of 1 Trigger and 4 Actions. Task Scheduler Trigger is set to start the whole process automatically every hour, no user interaction is needed. Once set up, Automation Workshop runs fully unattended in the background as a Windows Service without requiring the user to log in.
Synchronize Directories Action automatically scans for new, changed, and deleted files and folders on the first remote server. It synchronizes content to the local folder we have specified. The second Synchronize Directories Action syncs the local folder to the second Amazon S3 account.
The email notifications are used to notify system administrators about successful or failed synchronization process. It is also possible to retry all the process automatically when an unexpected error occurs.
Notes
The schedule of the Task can be changed to exclude weekends or nighttime. Also, it is possible to add Amazon S3 Watcher Trigger, to launch the synchronization process as soon as there are changes at the remote server.
Connection to Amazons S3 bucket requires Access key ID and Secret access key. Learn more on how to create Access key ID and Secret access key.
Alternatively, use the Synchronize Folders Action to sync (push, pull, or two-way syncing modes) local and shared folders.
Need a helping hand?
If you have any questions, please do not hesitate to contact our support team.