RClone is an excellent third-party tool which allows users to perform file-based operations between a large variety of supported endpoints. This is achieved by creating configurations referred to as “remotes” then copying or syncing your files between them, as well as many other supported operations. The setup part of the process takes a little care, but follow these steps and you should be good.
Step 0 - Create a Linode (Optional, but HIGHLY recommended)
The act of syncing huge amounts of data between endpoints requires that you download every single file and upload it again to the desired destination (the files are not held on disk though, so you don’t need a lot of storage). You could do this on your home internet connection, but it would take far longer than doing it via a Linode VM, which is the recommended approach. Any flavour of Linux is fine for this, though I tested it on Ubuntu 23.04. Also, make sure to create your Linode in the same region as the Object Store bucket you want to upload the content to. That will greatly reduce the round trip time for your uploads, and make the whole thing way faster.
Follow these instructions to get up and running with a Linode:
Step 1 - Install RClone
This part is super-easy. Just pick the command for your OS:
Linux (Debian-based):
sudo apt install rclone
Linux (Red Hat-based):
sudo yum install rclone
Windows:
winget install Rclone.Rclone
MacOS:
brew install rclone
Step 2 - Create API credentials
You need both a NetStorage Upload Account (with HTTP API access) and a Linode Object Store key pair. The following guides should show you how to do this:
Remember you need an HTTP API access token for your Upload Account, and that when you are finished editing the account and assigning it to a storage group, you still have to wait several hours for the changes to propagate.
Step 3 - Add NetStorage to RClone
Once you have installed RClone you need to configure a remote. This is done with the following command:
rclone config- Type n to create a new remote
- Give your remote a name. I suggest netstorage
- Select the option for Akamai NetStorage. At time of writing (v1.63.0) it is option 2
- For the host value, enter the HTTP API host of your NetStorage group followed by the CP Code upload directory, e.g. example-nsu.akamaihd.net/123456. You can get the upload directory from the NS Storage Groups UI
- Enter the ID of your Upload Account. This is the name you can see in the NS Upload Accounts UI
- At the next screen type y, confirming Yes, type in my own password
- Paste in the HTTP API key from your Upload Account, then confirm it
- Decline the option to Edit advanced config
- Finally, select y to keep the new remote, then q to quit the config
Now, just to make sure it actually works you should run the following (replacing the remote name with whatever you chose if you didn’t go for ‘netstorage’ like I told you to):
rclone ls netstorage:
Step 4 - Add Linode to RClone
This part relies on the fact that Linode Object Store does a very good impression of AWS S3, so you can treat it in almost entirely the same way. Unfortunately, the guide on the Linode doc site isn’t entirely accurate, so stick with this method instead.
rclone config- Type n to create a new remote
- Give your remote a name. I suggest linode
- Select the option for ‘Amazon S3 Compliant Storage Providers’. At time of writing (v1.63.0) it is option 5
- Select the option for ‘Any other S3 compatible provider’, at time of writing this is option 27
- At this point you have the option to input your credentials or instruct RClone to use environment variables. Environment variables are more secure, but the setup is slightly more complex so I will stick with the config file. However, feel free to update your config to do it the more secure way once you get the hang of it.
- Select 1 for entering your credentials
- Select 2 for using env variables, if ye dare
- Paste in the access_key and secret_key from your Linode Access Key (if so inclined)
- Select 1 for ‘Will use v4 signatures and an empty region’
- For endpoint, find the name of your OS bucket and enter it without the bucket name (this is really important). For example, if your bucket URL is example.us-east-1.linodeobjects.com enter JUST us-east-1.linodeobjects.com
– For location_constraint just leave blank and press enter - Select whichever access option is best, though I recommend option 1 for No one else has access rights if you are unsure
- Decline the option to Edit advanced config
- Finally, select y to keep the new remote, then q to quit the config
Now, we repeat the ls operation we did with NetStorage, but with a slight difference. When listing (and with all other commands actually) we need to supply the bucket name as a path prefix, since we set the endpoint to be just the OS region, like this:
rclone ls linode:<bucket name>/
Without this, ls operations will not function, and thus the sync command will not know which files it needs to re-upload when catching up.
Step 5 - Sync
Now we have the config completed (it is just written to a local text file if you wanted to back it up) we need to sync the contents. Run the following command, editing the source and destination paths as required:
rclone sync netstorage:/<source> linode:<bucket name>/destination --progress
e.g.
rclone sync netstorage:/files linode:example/files --progress
The –progress option will display the progress of the command, which is recommended if you are moving any number of files. You can also configure the number of concurrent transfers by adding –transfers . In my testing I found 16 (the default is 4) was the sweet spot.
When the command is complete, I recommend running it again in exactly the same way. The sync command should check all files on the destination and only upload any updated files, if any. In most cases the 2nd run should complete in a few seconds without having done any work, but will confirm that everything is good.