Recovery Drill

Verify your backups can actually be restored. A drill is a full end-to-end test of the backup-restore cycle on a separate target host, without touching production. Run one at least once per quarter, and immediately after any change to your transport or compose configuration.

A backup that has never been restored is a hypothesis. The drill turns it into a fact.

Prerequisites

• A spare host (VM, cloud instance, dedicated machine) with Docker and docker compose installed and nothing else.
• Network access from the spare host to wherever your transport drops bundles (storage box, S3 bucket, bastion).
• A recent full-backup tar produced by your production instance.
• Optional: a bookmark of your production instance's data so you can compare after restore.

The spare host does not need to be reachable from the public internet. The drill verifies the bundle is restorable, not that the restored instance is publicly accessible.

Step 1: collect a known-good backup tar

Pick a recent backup from the off-site location your transport writes to. The drill is meaningless if you copy from the same disk that produced the backup, since you would be testing the local filesystem, not the off-site copy.

If your transport pattern is rsync or scp, pull the tar onto the spare host:

rsync -avP storagebox:/srv/operaide/full-backup-YYYYMMDD-HHMMSS.tar /tmp/
rsync -avP storagebox:/srv/operaide/full-backup-YYYYMMDD-HHMMSS.tar.sha256 /tmp/

If the transport pattern is S3 or compatible:

aws s3 cp s3://my-bucket/operaide/full-backup-YYYYMMDD-HHMMSS.tar /tmp/
aws s3 cp s3://my-bucket/operaide/full-backup-YYYYMMDD-HHMMSS.tar.sha256 /tmp/

If the transport pattern is "deletes the source after copying" (rsync --remove-source-files), pulling back to the original host means the bundle is gone from the source. That is the intended behaviour. Skip the sha256 sidecar in that case if it was not transferred separately.

Step 2: restore on the spare host

Use the standard restore path with a target directory that is clearly marked as a drill:

bash operaide-outer-restore.sh /tmp/full-backup-YYYYMMDD-HHMMSS.tar

Answer y at the prerequisites prompt (it defaults to no). When the script asks for the target project directory, type a drill-specific path:

~/operaide-drill

Accept the defaults for the remaining prompts. The restore takes the same time it would on a real recovery.

If you do not have operaide-outer-restore.sh on the spare host yet, extract it from the bundle first:

tar -xf /tmp/full-backup-YYYYMMDD-HHMMSS.tar -C /tmp/drill-extract operaide-outer-restore.sh
bash /tmp/drill-extract/operaide-outer-restore.sh /tmp/full-backup-YYYYMMDD-HHMMSS.tar

Step 3: verify the restored instance

Open the configured ROOT_URL in a browser. It will likely point at the production hostname, which the spare host cannot serve. Override the local DNS or /etc/hosts to make the URL resolve to the spare host, or temporarily edit the .env in the drill project directory to point at a local hostname:

cd ~/operaide-drill
# edit .env: ROOT_URL=http://localhost:7077  (or similar)
docker compose up -d

Once the URL responds:

1. Sign in with an operator account that existed at backup time.
2. Open one app that had real data. Verify reaktors, deployments, and conversation history are present.
3. Click through to a known document. It should download and match what you saw in production.
4. Open System Admin -> Backups. The status snapshot should show the restore time as the new "started at" baseline. The "Transferred" badge will be Pending transfer because no transport ran on the drill host.
5. Pick one operator-uploaded logo. Verify it renders.
6. Verify the agent-database contents. Open one of the agent databases through the app and confirm content survived via its size.

If anything is missing or broken, the bundle has a problem. Investigate before tearing down the drill.

Step 4: tear down

Drills accumulate. After the verification, remove the drill project directory and its volumes so the spare host stays empty:

cd ~/operaide-drill
docker compose down -v
cd ~
rm -rf ~/operaide-drill

Optionally remove the loaded image:

docker image rm ghcr.io/objective-partner/op-no-gravity-operaide:<tag>

The drill leaves no trace.

Step 5: log the result

Track drill outcomes somewhere your team can find them. A simple text file in your operations notes is enough.

A failed drill is the most valuable kind. It catches the problem now, when production is still healthy, instead of during a real outage.

What to drill in addition to the standard restore

Run these scenarios at least once. They expose issues the standard drill does not:

• SHA-256 corruption. Flip one byte in the tar with dd and confirm the restore script aborts with SHA-256 mismatch. The restore must refuse, not silently proceed.
• Old backup compatibility. Restore a bundle that is at least one release older than your current image. Schema or path changes in inner-backup output should still be readable by current inner-restore.
• pre mongo dump. Pick pre at the dump-selection prompt instead of the default post. Verify the app still works and data is consistent.
• Overwrite a populated install. Set up a fresh instance, then restore on top and pick o (overwrite) at the existing-data prompt. The result should match the bundle's contents exactly, not a mix of both.

Common mistakes

• Drill on the same host as production. The drill's volumes, project directory, and ports collide with production. Use a separate host.
• Drill from the same disk that produced the backup. Tests the local filesystem, not the off-site copy. Pull from the off-site location first.
• No drill schedule. A "we will do it eventually" drill never happens. Calendar it. Quarterly is the floor.
• No log of past drills. Without a record, you cannot tell whether the system has ever produced a restorable bundle. The first failed drill on an unrecorded history is too late.
• Skipping the verification step. "The restore script exited 0" is not the same as "the data is intact". Click through actual content.