HomeGuidesChangelog
GuidesDiscussionChangelogLog In

Object Storage

Team Server uses object storage (e.g. Minio, S3) to store certain files. This includes Dataset files, files managed by Git LFS (any file in the input and output section of a Project), some Gitlab artifacts, and depending on your configuration, backup data.

Local Object Storage

By default, a local instance of Minio is simply deployed and configured.

This option requires no further input from the user making it easy and quick. Local Object Storage is a good choice for smaller teams or teams that do not work with large datasets.

There are a few drawbacks to using local object storage. The first is that all data is stored on the local file system, so this file system must be big enough to handle all of the data. The second, is that all data will flow through the Team Server instance. For larger teams with larger datasets, there are performance benefits that come from offloading large objects to an external object store.

To protect against data loss while using the local object store, the gigactl tool's backup function will make copies of all the object store data to the backup directory specified in the settings file. It is recommended that this backup directory be either mounted from another location or located in an external storage bucket to ensure its safety if the server goes down.

External Object Storage

This is a Team Server Pro feature that adds support for integration with external object stores.

If using an external object store, the user must manage the components themselves. This includes creating the buckets, ensuring they are clear of any pre-existing files before a clean install, backup the buckets, and restoring the contents of the buckets before a restore from backup install.

Currently the only external object store service that can be used is S3, but more services will be officially supported soon.

To enable external object storage:

  1. Create the 6 buckets as shown in the file below
  2. Create a connection file with credentials that grant access to the buckets. In the case of AWS S3, you should ideally create a policy allowing read/write access to just the required buckets and an IAM user with that policy attached.
# Example for AWS S3
provider: AWS
# Specify the region
region: us-east-1
# Specify access/secret keys
aws_access_key_id: BOGUS_ACCESS_KEY
aws_secret_access_key: BOGUS_SECRET_KEY
  1. Add the following fields to the Settings File prior to installation, as shown in this example:
admin_email: [email protected]
server:
  name: My Test Server
email:
  enabled: false
external_object_store:
  enabled: true
  service: s3
  connection: /home/myuser/connection.yaml
  buckets:
    backup: my-team-server-gitlab-backups
    tmp: my-team-server-tmp
    lfs: my-team-server-git-lfs
    packages: my-team-server-gitlab-packages
    artifacts: my-team-server-gitlab-artifacts
    uploads: my-team-server-gitlab-uploads
    dataset: my-team-server-dataset-objects

Note, no explicit backup section is needed when using external object storage.

It's strongly recommended to run frequent, regular backups of the object store buckets to protect against data loss. When using an external object store, the gigactl tool's backup function does not create copies of the contents of any external buckets, but only creates a backup of git data (excluding GitLab LFS files) and stores this archive in the backup bucket. Options for backing up external object store data can be found here:

🚧

gigactl Does Not Backup External Object Storage

When running with External Object Storage, you are responsible for backing up and maintaining all buckets.

Users must also provide a connection file formatted according to GitLab guidelines to allow GitLab and the gigactl tool to connect to the external object store data. Example files demonstrating the format can be found here: