How-to Articles
      Back to home
      1. Impossible Cloud Knowledge Base
      2. How-to Articles

      How to copy data from a 3rd-Party S3 bucket to an Impossible Cloud bucket using rclone

      Detailed steps for copying data from a 3rd-party S3-compatible bucket to an Impossible Cloud bucket using rclone.

      Overview

      This guide provides step-by-step instructions for copying data from a third-party S3-compatible bucket, such as AWS S3, to an Impossible Cloud bucket using rclone.

      Data traffic will flow through the local machine running rclone. If you are copying large datasets, be aware that this may impact transfer speed and time.

      Prerequisites

      1. Source bucket permissions

      Ensure that you have the following permissions on the source bucket (e.g., AWS S3 or another S3-compatible service):

        • S3:ListBucket: Allows rclone to view the contents of the source bucket.

        • S3:GetObject: Allows rclone to access each file for transfer.

      For additional help with configuring permissions, please refer to your storage provider’s documentation.

      2. Target bucket permissions

      Ensure that you have the following permissions in a policy on the target bucket (Impossible Cloud):

      • s3:PutObject, s3:PutObjectTagging, s3:PutObjectAcl: Allows rclone to upload the objects to the target bucket. See more information about IAM permissions here.

      3. Target bucket creation

      If you have not yet created the target bucket on Impossible Cloud, please follow this guide to create a bucket on Impossible Cloud.

      Step-by-Step Guide

      Step 1: Install rclone

      1. Visit the rclone download page and download rclone for your operating system (Windows, Linux, macOS).
      2. Extract the rclone executable, (e.g. rclone.exe on Windows), from the archive.
      3. Follow the installation documentation if needed.

      Step 2: Configure rclone for source and target storage provider

      1. Open a command prompt (Windows) or terminal (Linux) and browse to the directory with extracted rclone executable.

      2. Run the command to start rclone configuration:

        rclone config

      3. When prompted, choose n to create a new remote. Name the remote (e.g., AWS for the third-party S3 bucket) and hit Enter .

      4. Configure the source (3rd-party S3 cloud provider):

        • Find Amazon S3 Compliant Storage Providers including AWS3... and enter its number in the list, e.g. 4:

        • When asked for provider, enter the appropriate number (e.g., 1 for Amazon S3).
        • Proceed further and once asked, input the necessary Access Key ID and Secret Access Key for your third-party S3 account.
        • Input the number of the correct region (e.g., 1 for us-east-1).
        • Accept the remaining default settings or adjust as needed. The configuration of the source of the data copy source is now complete.

      5. Configure the target (Impossible Cloud).
        • Run rclone config once again.

        • When prompted, choose 'n' to create a new remote and name it, for instance "Impossible Cloud Target".

        • When prompted for storage, select Amazon S3 Compliant Storage Providers....
        • When prompted for S3 provider, select Any other S3 compatible provider   \ (Other).
        • Select Enter AWS credentials in the next step .
        • Enter the Access Key ID and Secret Access Key of  the root user or the IAM user which has access to the target bucket.
        • For the region, use v4 signatures and an empty region:
        • For the endpoint, use the proper Impossible Cloud endpoint URL according to your bucket's location. For instance:
          https://eu-central-2.storage.impossibleapi.net
        • Accept remaining defaults unless you need custom settings.
        • Configuration is now complete and you have two remotes: one for the data source ("AWS" in the example below)  and another for the target ("Impossible Cloud Target" in the example below):
        • Press q and quit rclone config.

      Step 3: Sync data from source to target

      1. Run the following command to initiate the data sync:
        rclone sync source_rclone_config_name:source-bucket-name target_rclone_config_name:target-bucket-name

        Explanation: The sync command only copies new or changed files, allowing you to resume transfers efficiently if interrupted.
        Note: if  your rclone configuration name contains spaces, enclose it in quotes like in the example below:

      2. Progress and Completion: rclone will display progress in the terminal. When complete, verify that all data has been copied successfully (see verification steps below).

      Only the latest version of objects will be copied if the source bucket has Versioning enabled. Other versions are ignored.

      Resuming transfers if interrupted

      If the transfer is interrupted for any reason (e.g., network disconnection or system shutdown), please rerun exactly the same rclone sync command above. rclone will skip files that have already been transferred, resuming from the last incomplete point.

      Verify transfer

      To confirm the transfer was successful:

      1. Re-run the rclone sync command and add -v by the end to view the verbose output. If you see the message "There was nothing to transfer", then all your data has been synced to the target bucket:

      2. (Optional) Log in to your Impossible Cloud account.
        1. Navigate to the target bucket and check that the number of files matches the source bucket.
        2. Spot-check a few files to ensure they were transferred correctly.