Select the right steps for copying data between 3rd-party S3 and Impossible Cloud buckets, or between different Impossible Cloud buckets.
Introduction
This guide provides an overview of using rclone to copy data between different buckets, specifically when transferring data between S3-compatible services like AWS and Impossible Cloud, or between two Impossible Cloud buckets. Since Impossible Cloud does not support the CopyObject S3 operation across different buckets, rclone can be used as a workaround for these transfers.
Rclone is single executable (rclone, or rclone.exe on Windows) that you can simply download as a zip archive and extract into a location of your choosing. See the install documentation for more details.
Impossible Cloud does not charge for egress traffic; however, transfers should comply with our Fair Egress Policy, which encourages responsible data usage to ensure service quality for all users.
System Requirements
To ensure a smooth data transfer process, please ensure the following:
- Network: A reliable and stable network connection is recommended, especially for large data volumes.
- Storage: While rclone transfers data “on the fly” without requiring extensive local storage, ensure there’s sufficient space to handle temporary files needed for ongoing operations during the transfer.
- Operating System Compatibility: rclone supports multiple operating systems, including Windows, MacOS and Linux.
Selecting Your Transfer Scenario
If the source bucket has Versioning or Object Lock enabled, only the latest version of each object will be copied during the transfer. Previous versions will not be included. This applies to all scenarios listed below. If the current version is a DeleteMarker, that object will not be copied during the transfer.
The process of copying data between buckets can vary based on the source and target bucket configuration. Please select the scenario that applies to you:
- Scenario A: Copy data from any 3rd-party S3 bucket (e.g., AWS S3) to an Impossible Cloud bucket.
- Scenario B: Copy data between two Impossible Cloud buckets.
- Alternative Scenario: For very large data migrations (e.g., over 10 TB), please consider Flexify, our partner that specializes in cloud-to-cloud migrations.
Each scenario includes step-by-step instructions tailored to each setup, covering installation, configuration, and transfer commands.
Error Handling
Should you encounter any issues, visit our Impossible Cloud Help Center for troubleshooting guidance, access to additional documentation, or to submit a support ticket.