Skip to content
More support
Contact us
  • There are no suggestions because the search field is empty.

How to configure Impossible Cloud Storage as a datastore in lakeFS

Configure your lakeFS server with Impossible Cloud Storage using the s3 blockstore type.

lakeFS adds version control to object storage (i.e. branch, commit, merge, and revert) with zero copy. Together with Impossible Cloud's sovereign and geo-fenced infrastructure, this article guides you to setup a control plane and governance layer at little to no additional storage volume from your source data.

Prerequisites

  • A bucket in Impossible Cloud Storage containing your existing source data. This article refers to this as the source bucket.
  • A running lakeFS server. 

It is recommended that the source bucket do not have bucket versioning and/or object lock enabled.

This guide was tested with lakeFS Enterprise 1.97.0, but the blockstore configuration is identical in open-source lakeFS.

Step 1: Create a bucket for lakeFS repository

Access your Impossible Cloud storage account and create a new bucket in the same region as your source bucket. The new bucket should have the bucket versioning and object lock disabled.

This bucket will be used to store lakeFS repository data that allows versioning of your source bucket. From this point on, the article refer to this bucket as repository bucket.

Follow the steps in the following documentation for a detailed step-by-step guide on how to create an s3 bucket in Impossible Cloud.

Optionally, you can create a folder inside your existing source bucket and use it as your lakeFS repository.

Step 2: Set up s3 as your lakeFS blockstore type

In your lakeFS deployment, add the following as environment variables so it can connect to Impossible Cloud storage. This example uses a bucket region in eu-north-1:

LAKEFS_BLOCKSTORE_TYPE=s3
LAKEFS_BLOCKSTORE_S3_REGION=eu-north-1
LAKEFS_BLOCKSTORE_S3_ENDPOINT=https://eu-north-1.storage.impossibleapi.net
LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID=<your-ic-access-key>
LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY=<your-ic-secret-key>

Or, if your deployment uses a configuration file, the equivalent YAML:

blockstore:
  type: s3
  s3:
    region: eu-north-1
    endpoint: https://eu-north-1.storage.impossibleapi.net
    credentials:
      access_key_id: <your-ic-access-key>
      secret_access_key: <your-ic-secret-key>

If you have an existing lakeFS instance, a restart of the lakeFS server may be required after changing its configuration.

If your existing lakeFS instance already hosts repositories on another backend, see lakeFS's Multiple Storage Backends to run several lakeFS backends allowing multi-cloud blockstores.

Step 3: Create a lakeFS repository

In the lakeFS UI (e.g. http://localhost:8000), click Create Repository and fill in:

  • Repository ID: The name of your repo (e.g. ic-repo)

  • Default Branch Name (default): main

  • Storage Namespace: S3 URI to your repository bucket (e.g. s3://repository-bucket/ or if you are using the same bucket s3://source-bucket/repo/ic-repo)

Step 4: Import data and version it

Go to your new repository created in Step 3 in lakeFS UI and click Import.

Fill in the details:

  • Import from: S3 URI to your source bucket (e.g. s3://source-bucket/datasets/

  • Destination: Where you want to store it in the repository (e.g. lakefs://ic-repo/main/datasets )

  • Commit message: A short description of this change (e.g. "first commit")

Import is zero-copy: lakeFS records pointers to the existing objects instead of duplicating them, so importing a large dataset costs no additional storage.

Summary

You have successfully integrated Impossible Cloud to lakeFS and setup your "main" branch in the lakeFS repository based on your source bucket.

As lakeFS manages the data versioning in a seperate metadata layer. Any action and version of your data involves zero-copy of your actual data. You can start versioning your data, branch, edit, diff, and merge data without actually copying your data from the source bucket.

Note on Immutability: lakeFS does not support S3 bucket versioning or Object Lock because it maintains its own version history above the object store. For operational immutability, organizations should route writes through lakeFS, restrict direct write and delete access to the backing bucket, protect production branches, and reference data by commit ID.

Resources