Transport via S3 (or S3-Compatible Object Storage)

Wire transfer.sh so every successful backup is uploaded to an S3 bucket. Works with AWS S3, Hetzner Object Storage, Backblaze B2 (S3 API), Wasabi, Cloudflare R2, MinIO, or any other S3-compatible target. The setup is the same; only the endpoint URL changes.

This pattern uses one cron job (the backup, which calls the transport hook at the end) and lets the object storage handle retention via a lifecycle policy.

Prerequisites

• A bucket dedicated to Operaide backups. Do not share it with unrelated content.
• IAM credentials with s3:PutObject on the bucket. Add s3:DeleteObject only if transfer.sh will clean up old bundles itself instead of relying on a lifecycle policy.
• The AWS CLI v2 installed on the Operaide host (aws --version). For non-AWS targets, the same CLI works with a custom endpoint.
• Backup setup already enabled on the Operaide host. See Enable Backups first.

Step 1: install the AWS CLI

On Debian or Ubuntu hosts:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscli.zip
unzip awscli.zip
sudo ./aws/install
aws --version   # should print aws-cli/2.x.x
rm -rf awscli.zip ./aws

Skip this step if aws --version already responds.

Step 2: configure credentials

The cron job runs as root, so credentials live in root's AWS config:

sudo aws configure --profile operaide-backup

Enter the access key, secret key, default region, and output format. The profile name operaide-backup keeps these credentials separate from any other AWS use on the host.

For S3-compatible providers other than AWS, set the endpoint URL too. AWS CLI v2 reads it from the environment or a profile setting:

sudo aws configure --profile operaide-backup set endpoint_url https://fsn1.your-objectstorage.com

(Example shown is Hetzner Object Storage in the FSN1 region. Replace with your provider's endpoint.)

Test the credentials by listing the bucket:

sudo aws --profile operaide-backup s3 ls s3://my-operaide-backups/

Step 3: edit transfer.sh

Open <project>/backup/transfer.sh and replace the placeholder body with the upload command.

#!/usr/bin/env bash
# transfer.sh — upload to S3-compatible object storage.
#
# Called by operaide-outer-backup.sh with $1 = path to the new tar.
# Touches a .transferred sidecar after a successful upload so the
# backup-status UI marks the bundle as transferred. The local tar
# stays on disk; remote retention is handled by a bucket lifecycle
# policy.
set -euo pipefail

BUCKET="my-operaide-backups"
PREFIX="operaide/$(hostname -s)"
PROFILE="operaide-backup"

aws --profile "$PROFILE" s3 cp "$1"        "s3://$BUCKET/$PREFIX/$(basename "$1")"
aws --profile "$PROFILE" s3 cp "$1.sha256" "s3://$BUCKET/$PREFIX/$(basename "$1").sha256"

touch "$1.transferred"

Replace BUCKET, PREFIX, and PROFILE to match your setup. The $(hostname -s) segment in the prefix keeps multi-instance deployments separate inside one bucket.

If you prefer to drop the local tar after a successful upload (saves disk on the Operaide host), replace the touch line with rm -f "$1" "$1.sha256". The deletion is also recognized by the UI as "transferred".

Make the script executable and check syntax:

chmod +x <project>/backup/transfer.sh
bash -n <project>/backup/transfer.sh

Step 4: dry-run against an existing tar

Run the transfer once by hand against the most recent backup tar:

cd <project>
bash backup/transfer.sh backup/backups/full-backup-YYYYMMDD-HHMMSS.tar

Expected: the AWS CLI prints upload progress, the .transferred sidecar appears next to the tar, and the file is in the bucket.

Verify on the remote side:

sudo aws --profile operaide-backup s3 ls s3://my-operaide-backups/operaide/$(hostname -s)/

Step 5: confirm the transferred badge

Refresh System Admin -> Backups. The Outer-block card should now show Transferred (green check) thanks to the .transferred sidecar.

Step 6: retention via lifecycle policy

S3-compatible providers expose lifecycle rules that delete or archive objects after N days. This is the right place to enforce backup retention. Example AWS lifecycle JSON that deletes anything older than 90 days under the prefix:

{
  "Rules": [
    {
      "ID": "expire-operaide-backups",
      "Status": "Enabled",
      "Filter": { "Prefix": "operaide/" },
      "Expiration": { "Days": 90 }
    }
  ]
}

Apply with:

sudo aws --profile operaide-backup s3api put-bucket-lifecycle-configuration \
  --bucket my-operaide-backups \
  --lifecycle-configuration file://lifecycle.json

Match the rule's prefix to the path your transfer.sh writes to. Verify the rule:

sudo aws --profile operaide-backup s3api get-bucket-lifecycle-configuration \
  --bucket my-operaide-backups

Lifecycle policies run asynchronously; the deletion happens within 24 hours of the threshold being crossed.

Step 7: encryption at rest

S3 server-side encryption protects the bundle on disk in the provider's storage. AWS S3 encrypts buckets by default since 2023. For S3-compatible providers, check the documentation; some require an explicit --sse aws:kms flag on s3 cp.

Encryption in transit is automatic when the endpoint URL uses https://.

For end-to-end encryption (the provider cannot read the bundle), encrypt the tar locally before upload:

gpg --symmetric --cipher-algo AES256 --batch --passphrase-file /root/.operaide-backup-key "$1"
aws --profile "$PROFILE" s3 cp "$1.gpg" "s3://$BUCKET/$PREFIX/$(basename "$1").gpg"
rm -f "$1.gpg"
touch "$1.transferred"

This trades restore-time complexity (you have to decrypt the tar before running outer-restore.sh) for stronger off-host secrecy. Pick deliberately.

Common mistakes

• Wrong IAM scope. Granting s3:* on * from the Operaide host means a compromised host can read or delete every bucket in the account. Scope the policy to one bucket and one or two actions.
• Bucket without lifecycle. Storage costs grow forever. Decide on retention before turning the cron on.
• Profile not visible to root. aws configure writes to the invoking user's home. The cron job runs as root, so credentials must be in /root/.aws/. Use sudo aws configure (or set AWS_SHARED_CREDENTIALS_FILE).
• Endpoint URL drift. S3-compatible providers add or rename regions occasionally. Check the provider's status page after unexplained transport failures.
• Putting credentials inline in transfer.sh. Keep them in /root/.aws/credentials. Inline keys end up in the backup tar (since transfer.sh is bundled), increasing the blast radius of a leaked bundle.
• No drill. See Recovery Drill. A bundle that uploads successfully but cannot be downloaded later is worse than no backup.