Trove

Trove is an S3-compatible object store with a web console. Create buckets, share them with other users, and mint S3 access keys for tools and Coder workspaces.

The console is public behind Cloudflare Access. The S3 endpoint is internal only: https://trove.k8s.internal.galaxygridlabs.com

User Guide

How to Connect - Console

  1. Request access to the labusers@hul.to group from an admin
  2. Navigate to https://trove.astral-labs.work/
  3. Login with Vault
  4. Create a bucket (lowercase S3 naming rules), then upload and download from the browser.

How to Get S3 Keys

  1. Console > Access keys > Create
  2. Copy the secret. It’s shown once and can’t be recovered, revoke and reissue instead.

How to connect - S3 cli

Needs the lab network (Coder or Wireguard) and the GGL CA, see Trusting Vault root CA.

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
aws --endpoint-url https://trove.k8s.internal.galaxygridlabs.com s3 ls

Note πŸ“ - Path-style addressing only. No presigned URLs or versioning.

How to connect - rclone mount

  1. Install rclone and fuse3
  2. Create ~/.config/rclone/rclone.conf (chmod 0600)
[trove]
type = s3
provider = Other
env_auth = false
access_key_id = ...
secret_access_key = ...
endpoint = https://trove.k8s.internal.galaxygridlabs.com
region = us-east-1
list_version = 2
force_path_style = true
no_check_bucket = true
  1. Mount your bucket
mkdir -p ~/trove
rclone mount trove:<bucket> ~/trove \
  --vfs-cache-mode writes \
  --disable-http2 \
  --dir-perms 0755 --file-perms 0644 --umask 022 \
  --ca-cert /usr/local/share/ca-certificates/ggl.crt

Note πŸ“ - --disable-http2 is required, and --vfs-cache-mode writes is used over s3fs because s3fs fails per-file chmod with ENOSYS. Unmount with fusermount3 -uz ~/trove.

How to Share a Bucket

Bucket > Grants > pick a user and read, write or admin. Users only see buckets they own or were granted.

How to Access the Palo Image Bucket

The palo-disks bucket holds the palo disk images. It is owned by another user, so you must be granted access before it shows up for you.

  1. Install rclone and fuse3:
sudo apt install -y fuse3
curl https://rclone.org/install.sh | sudo bash
  1. Sign in to the Trove console and create your own S3 keys, see How to Get S3 Keys.
  2. Send your Trove user id (a UUID string) to a lab admin and ask them to share palo-disks with you (read is enough, see How to Share a Bucket).
  3. Add a [trove] remote with your keys to ~/.config/rclone/rclone.conf, see How to connect - rclone mount.
  4. Allow other users in FUSE (one-time), required by --allow-other:
echo user_allow_other | sudo tee -a /etc/fuse.conf
  1. Mount the bucket read-only:
mkdir -p ~/palo-disks
rclone mount trove:palo-disks ~/palo-disks \
  --vfs-cache-mode writes \
  --disable-http2 \
  --dir-perms 0755 \
  --file-perms 0644 \
  --umask 022 \
  --ca-cert /usr/local/share/ca-certificates/ggl.crt \
  --links --allow-other --read-only --daemon

Note πŸ“ - --read-only blocks writes so nobody can modify the images by accident. Drop it only if you were granted write. --links makes symlinks resolve (e.g. grub.conf β†’ maint_grub.conf). Without a grant you will get AccessDenied errors. Unmount with fusermount3 -uz ~/palo-disks.

How to Integrate Coder

Put your keys in the Vault workspaces/<username>@hul.to/env secret, see How to Integrate Trove. Supported templates mount your bucket at ~/trove.

Admin Guide

Runs on smoll-harvester from src/fleet_smoll_harvester/trove with Postgres. Pulumi pushes the secrets and config in src/2_app/services/trove/trove.py.

Auth

Console login is Vault OIDC. labusers@hul.to can sign in, labadmins@hul.to are lab admins. Lab admins can recover a bucket from a departed owner, revoke any key, and read the audit log.

Note: The app key encrypts stored S3 secrets. Replacing it makes every existing access key unusable. Replacing the session secret logs everyone out.

Limits

  • Objects up to 5 GiB.
  • Abandoned multipart uploads are removed after 24h.
  • Shutdown gives active uploads 25s to finish.

Benchmarks and QA

scripts/bench.sh quick for a local baseline, scripts/benchlive against a deployed endpoint, make qa-browser for the real-browser suite. See scripts/QA.md.

  • Coder
  • Vault
  • src/2_app/services/trove/scripts/QA.md