Using Civo object stores

Introduction

The Civo object store is a managed service that allows you to upload unstructured data in an S3-compatible system. If you have used S3-style object storage systems before the way you interact with Civo object stores (the API) will be familiar to you.

This guide will walk you through creating an object store in your Civo account, uploading files to it through the web interface and the command line using the API, and handling credentials for object stores. It will then walk you through deleting your object store at the end.

Civo object stores are region-specific. This means that the endpoint URL for an object store will depend on the region where you create a store, and credentials to manage and access object stores are tied to the region where they were created.

Prerequisites

You will need to have an active Civo account to follow along.

Additionally, you will need some command-line tools which the guide will assume you have installed on your machine. These are:

• Civo CLI. If you have previously installed Civo CLI, make sure to run civo update to ensure you have the latest version.
• s3cmd for interacting with the object stores, or another S3 client of your choice. This tutorial will assume you are using s3cmd though.

Object store creation

Start by navigating to the object store management section in your Civo account. You can find the "Object Stores" menu item in the sidebar, which will take you to the store management page. You will be able to create multiple object stores here as long as they fit in your quota.

Creating your first object store credential

To access object store contents through the S3 API, you will need to create a credential that can be associated with the store. All Civo object stores are created as private and require an access key ID and secret credentials.

Click on "Credentials" to be taken to the credentials management page:

Next, click the "Create a new Credential" button to be given the option to give a name for your credential.

Once you click "Create", the system will prepare your new credential to be assigned to an object store of your choice.

Return back to the object store management page to proceed with the next step.

Note: If you create an object store through the Civo CLI and do not specify a credential, one will be automatically generated for you, but this guide is intended to show you all the stops you can take.

Creating your first object store

Start by clicking "Create an Object Store", which should show you the following screen:

For the Name field, use a descriptive name for the object store.

As you created a credential in the previous step, you will be able to assign it to your object store as an authorized accessor by dropping down to the Credential menu to select the one you created. Pick the credential name you just created from the drop-down menu.

You can size an object store in 500GB increments up to your object store quota limit. The price (displayed monthly but charged on an hourly basis) will change depending on the size of the object store you create.

For this demo, leave the store size as 500GB.

Next, click "Create store" to proceed.

The page should show your store being created, along with the access URL and associated credentials from your list:

Once you see the URL endpoint for the store, we can proceed to add files to it.

Uploading files to object stores

Uploading files using the web

Start by clicking the name of the object store you just created. You will be taken to the file management page for the store:

Next, click "Upload file" and select a file to add to the store from your local machine. Keep the Permissions private for now. In S3-type storage, objects can be private (accessible only by providing the credential details) or public (accessible by anyone with the URI to the object). By default, objects are created as Private.

When you have chosen a file to upload, click "Upload file".

When the file is added, you should see a message informing you that it was successfully created on the object store, along with its details in the store.

You can also create folders and upload files to particular folders to create a directory structure. The folder name will be part of the URL used to retrieve any files you upload.

Uploading files using the command line

You now have everything you need to upload files using the command line - the S3 client from the Prerequisites section, and details of the created object store and credentials to manage it.

Setting up credentials for s3cmd

In order to interact with the Civo object store you created, you'll need s3cmd to be made aware of the credentials and location. You can either use environment variables like this tutorial, or input credential values directly into the s3cmd configuration which you can create by running s3cmd --configure and following the prompts.

In order to get the credential details from your account, you can use Civo CLI to get these.

First, ensure you are working in the region where the object store is set up, in my case FRA1:

civo region use FRA1

Next, confirm the CLI can see your object store:

$ civo objectstore show demo-store

                   ID : 5f4c95b4-7f11-4ebd-8191-5e9dc8881bb3
                 Name : demo-store
                 Size : 500
Object Store Endpoint : objectstore.fra1.civo.com
               Region : FRA1
           Access Key : OIPUMAEH30PMWHYQU2IX
               Status : ready
To access the secret key run: civo objectstore credential secret --access-key=OIPUMAEH30PMWHYQU2IX

The Access Key of the store's credential in the command above can be used with the following CLI command to get the environment variable commands needed for S3cmd to work:

$ civo objectstore credential export --access-key=OIPUMAEH30PMWHYQU2IX
# Tip: You can redirect output with (>> ~/.zshrc) to add these to Zsh's startup automatically
export AWS_ACCESS_KEY_ID=OIPUMAEH30PMWHYQU2IX
export AWS_SECRET_ACCESS_KEY=kRk1RiXCrV7R7rfSDwAEZUI6HhGbP2xjgJJJ76mw
export AWS_DEFAULT_REGION=FRA1
export AWS_HOST=https://objectstore.fra1.civo.com

For the purposes of this tutorial, copy the lines beginning with export from your command's output and paste them to your terminal to create the environment variables for this session.

As S3 was developed by Amazon Web Services (AWS), tools commonly use the AWS variable names for other S3 hosts that do not run on AWS. This ensures interoperability between object store providers, where the Access Key ID/Secret Key credential mechanic is used throughout.

Using s3cmd to upload a file to an object store

Create a file to upload to the object store you have just gained access to. As an example, you can create a short text file:

$ cat <<EOF >cli_upload_demo.txt
Hello from the command line.
This file will be uploaded to a Civo object store.
EOF

The s3cmd syntax to upload a file to a store is s3cmd put FILE [FILE...] s3://STORENAME[/FOLDERNAME]. Assuming you named the store demo-store, the command would be

$ s3cmd --host=${AWS_HOST}  --host-bucket=s3://demo-store put --acl-public cli_upload_demo.txt s3://demo-store/cli/

upload: 'cli_upload_demo.txt' -> 's3://demo-store/cli/cli_upload_demo.txt'  [1 of 1]
 80 of 80   100% in    0s   572.01 B/s  done
Public URL of the object is: http://objectstore.fra1.civo.com/demo-store/cli/cli_upload_demo.txt

In this command, the structure is as follows:

• The --host argument specifies the object store host, that is the Civo region where the store was created.
• The --host-bucket argument names the object store in the host region as an S3 URI, in this case "demo-store".
• The put command tells s3cmd to upload file to be specified.
• The --aci-public argument makes the permissions on this uploaded file public,
• cli_upload_demo.txt is the file to be uploaded
• s3://demo-store/cli is the path in the object store to upload the file to.

Now, when you look on the web interface, you can see that a folder called cli/ has been created and it contains the file you uploaded:

Downloading files from object stores

Crucially, because the file above was uploaded with public permissions, copying the file URL by clicking the file name, and opening it up in a browser tab, will let you access the file directly.

Aside from being able to access public files in object stores through a browser or a tool such as curl by providing the URI, the S3 API allows for downloading files on the command line.

You uploaded a file to a directory called cli/ in the above example. If you were to want to download that entire directory's contents to another machine, you could run:

$ s3cmd --host=${AWS_HOST} --host-bucket=s3://test-store get --recursive s3://test-store/cli/

download: 's3://test-store/cli/cli_upload_demo.txt' -> './cli_upload_demo.txt'  [1 of 1]
 80 of 80   100% in    0s   596.15 B/s  done

In this command, the crucial parts that differ from previous examples are the get command to download the path specified, and the --recursive switch which allows you to get the entire contents of a directory. If you do not specify a path, the contents of the directory location will be downloaded to the current working directory on the machine where you run the command.

If you only needed to download a single file, you would specify its path and filename without needing the --recursive option.

If you want to download a file that was added to the store as Private you would need to perform the download on the command line as this passes the credentials to authorise you to perform the action.

Deleting object stores

As mentioned at the start of this tutorial, object stores are charged on your account based on their size. You can delete object stores to stop them being charged from that point onwards.

Note: Object store deletion destroys all objects (files) within the store. There is no way to recover these files if the object store is deleted.

You can delete an object store from your account by navigating to the listing of object stores and selecting "Delete store" from the Action column:

This will prompt you to confirm the name of the store to be deleted as a security measure:

All files and the object store itself will be deleted as soon as you confirm this step, so make sure you only do this if you have recovered all files you may need.

You will then be able to delete any credentials that were assigned to the object store. Note that you cannot delete credentials that are assigned to active object stores - the store itself must be deleted before the credential can be removed.

Conclusion

Civo object stores are a great way to store files and manage them with the widely-adopted S3 standard API. The S3 compatibility ensures that the behavior of the object store API will feel familiar if you have used other S3 services before. In fact, the command-line examples used throughout in the tutorial make use of s3cmd, which is commonly used with all S3-type storage. The API is compatible with other S3 tools as well.

With storage availability in increments of 500GB, you can resize your Civo object stores up when needed without having to worry about running out of space.