Skip to main content
Version: 3.1

Backup and Restore

Operaide ships a backup system that captures a full instance into one tar file: MongoDB, the data volume, the SQLite agent databases, the operator-uploaded logos and documents, and the container image itself. Restore on a fresh host runs from one command and brings the instance back online.

This page explains what the system covers, when to use it instead of a hypervisor snapshot, and how the pieces fit together. The other pages in this section are step-by-step recipes.

When to use this versus a hypervisor snapshot​

Use a hypervisor or filesystem-level snapshot (ZFS, LVM, EBS) when your host environment supports it and you trust your operations team to manage retention and off-site copies. A snapshot captures the entire VM in one atomic moment and restores faster than re-importing a tar.

Use the Operaide backup system when one of the following holds:

  • Host without snapshot tooling. No ZFS, no LVM, no managed snapshots.
  • Portable bundle. You need to copy the instance to an unrelated host (different cloud, different hypervisor).
  • Application-aware snapshots. You want consistent MongoDB and SQLite snapshots that survive partial filesystem corruption.

The two approaches compose. Run the Operaide backup as a portable recovery option, and let the hypervisor snapshot cover fast same-host rollback.

What the bundle contains​

A successful backup produces one file plus a sha256 sidecar:

backup/backups/full-backup-YYYYMMDD-HHMMSS.tar
backup/backups/full-backup-YYYYMMDD-HHMMSS.tar.sha256

Top-level entries inside the tar:

  • operaide-image.tar: the exact Docker image the instance was running, captured via docker save. Restore on a fresh host does not need network access to a registry.
  • compose.yml plus .env and .env.openid: the deployment configuration files from the project directory.
  • inner-backup-YYYYMMDD-HHMMSS.tar.gz plus its sha256: the container-internal payload (see below).
  • operaide-outer-restore.sh plus transfer.sh: the host-side scripts. Restore reads them out of the bundle on a fresh host so you do not need a running instance to start recovery.
  • meta-outer.json: bundle metadata.

Inside the inner-backup tar:

  • data/: the /data volume (uploaded files, agent databases, Studio workspaces) without the live SQLite files.
  • app-assets/: operator-uploaded logos and documents from the legacy /app/bundle/programs/server/assets/app/ volume.
  • sqlite-vacuum/: consistent VACUUM INTO copies of every agent SQLite database, mirrored at the original relative path.
  • mongodb-dump-pre/ and mongodb-dump-post/: two MongoDB dumps, one taken at the start of the backup and one at the end. The restore picks one (see below).
  • meta.json: schema version and payload flags.

Two MongoDB dumps​

Backups take minutes for a typical instance, and MongoDB keeps accepting writes during the run. A single dump leaves a gap between mongo state and the captured /data snapshot.

Inner-backup runs two dumps to give the restore a choice:

  • pre: taken before /data is staged. Consistent with the filesystem snapshot, but does not see writes that arrived during the rsync window.
  • post: taken after /data is staged. Has the freshest mongo state, but may reference files the bundle does not contain.

Restore defaults to post and prompts the operator to override.

Two transport patterns​

The backup system writes the bundle to local disk. You decide how the bundle leaves the host. There are two patterns.

Pattern A: transport script invoked by outer-backup​

Configure transfer.sh next to operaide-outer-backup.sh in the project's backup/ directory. The backup script invokes transfer.sh automatically at the end of every run, after releasing its own lock. You configure transport in one file, and schedule one cron job for the backup. A long-running transfer never blocks the next backup, because the lock is released before the transport starts.

This is the default. The container ships a stub transfer.sh and seeds it into backup/ on first start. Edit it to point at your transport target.

Pattern B: separate cron for transport​

Leave transfer.sh as the no-op stub and run your own cron job that watches backup/backups/ and ships completed tars off-site. The Operaide backup script never invokes transport, and the two schedules run independently. Use this when your transport tool needs its own runtime (Borg, restic, a managed agent).

Either pattern uses the same off-site signal: when a tar is shipped, either delete it or touch <tar>.transferred. The "Transferred" badge on the system admin "Backups" page and the "transport stuck" warning both watch for these two signals. Without one, the bundle counts as not transferred and starts showing as stuck after 7 days.

Encryption is your responsibility​

The backup tar is sensitive. It contains .env (database credentials, OIDC client secrets, audit tokens), .env.openid if present, every uploaded document and logo, the full MongoDB contents, and transfer.sh (which may carry inline transport credentials). Treat the tar exactly like you treat your .env file.

warning

Operaide does NOT encrypt the bundle. The tar leaves the container in plaintext. If the tar leaves the host in plaintext as well, anyone with read access to the off-site location can recreate your full instance.

Encrypt the bundle before, during, or at rest at the off-site target. The mechanism depends on your transport pattern.

Pick at least one of these layers:

  • TLS in transit. Most transport tools already encrypt the wire (rsync over SSH, aws s3 cp to HTTPS endpoints, scp). Verify by reading your transport command. A plain ftp or http:// upload is not enough.
  • Server-side encryption at rest. S3-compatible buckets encrypt object data on the provider's disk. Hetzner Storage Box stores files on encrypted volumes. This protects against disk-level theft on the provider, not against a leaked credential.
  • Client-side encryption before upload. Encrypt the tar locally with GPG, age, or a tool like restic that encrypts as part of its pipeline. Only this layer protects against a compromised provider.

For a sensitive instance pick all three. For a hobby instance pick at least one. For an empty test instance you may skip; production is never that.

The transport pages (rsync, S3, SCP, BYO) cover the mechanism for each pattern. The S3 page also walks through GPG-based client-side encryption as an example.

What is in scope​

  • Single MongoDB instance, single container, daily backup.
  • Application-aware snapshots (MongoDB dumps, SQLite VACUUM).
  • Self-contained bundle that restores on an unrelated host.
  • Off-site transport hook, with optional separate transport cron.

What is not in scope​

  • Cross-source consistency between MongoDB and /data. The two payloads sit at slightly different points in time. Operations that reference uploaded files from MongoDB rows survive both ways with small inconsistencies you would also see on a power loss.
  • Encryption. See the section above. Operators bring their own.
  • Two-phase restore on a running instance. Restore brings the instance down.
  • Multi-tenant restore of a single organization. Restore works at the whole-instance level.

Pages in this section​

Common mistakes​

  • No off-site copy. A bundle that lives only on the same host disappears with the host. Pick a transport pattern.
  • Only one full backup retained. The disk-space safety in inner-backup refuses to evict the only remaining tar; it then fails the run. Keep enough room for at least two.
  • No recovery drill. Backups that have never been restored are theory. Schedule a drill at least once per quarter.
  • Backup tar treated as non-secret. The bundle contains .env, .env.openid, and transfer.sh. Treat the tar as you would treat your secrets file.