C8 — Recovery is part of security

From cognokratos/arktos-wallet · docs/capability/08-recovery-is-part-of-security.md · pinned revision 92650a034799

Security includes the ability of the legitimate owner to recover.

Question: the server's disk dies. What exactly do you need to bring Arktos back on a clean machine with every wallet intact? And what happens when one of those things is missing?

Prerequisite (optional): durable state, crash and restart semantics in general are covered in the sophos-agent runtime path. This lesson is about the cryptographic side: with encryption, a missing key is not a degraded mode. Whatever that key protected is unrecoverable from the encrypted data.

Mental model

Every encryption layer adds something you must keep in order to read your own data. Arktos has two layers, so a restore depends on three things:

database files          arktos.db (+ arktos.db-wal, arktos.db-shm while running)
+ DATABASE_KEY          opens the SQLCipher file
+ MASTER_KEY            decrypts recovery phrases, and verifies client API keys

There are also two non-secret requirements:

• A compatible binary. Migrations only move forward and run at startup. A binary older than the database's schema refuses to open it (newer_schema_version_is_rejected in tests/persistence_tests.rs).
• The same chain configuration (BITCOIN_NETWORK). Nothing is lost if it differs: accounts are stored per network. But the server will serve and derive a different set of addresses.

ADMIN_API_KEY is not a recovery dependency. It is compared against the environment and is not bound to any stored data, so a new value works immediately.

What each loss means

Every consequence below describes what can be recovered from Arktos's own data. "Lost from Arktos storage" is not the same as "provably lost everywhere": an operator may hold an independent backup of a recovery phrase, for example one exported deliberately with the operator tooling (make decrypt). The lesson is that Arktos's data alone no longer suffices.

The dangerous partial failure

The MASTER_KEY row has a trap in it. Suppose an operator "recovers" by starting the server with a new MASTER_KEY and the old database:

1. The server starts normally. MASTER_KEY is valid, and DATABASE_KEY opens the file.
2. Every agent gets 401, because the old API-key hashes no longer match.
3. The admin re-issues keys with POST /admin/api-keys/{id}/rotate. Ownership is by key id, so the owners get their wallets back.
4. get_*_address for an already-derived index succeeds: it is served from the public accounts row without any decryption.
5. get_*_address for a new index fails with internal error.

Step 4 is the trap. The service keeps handing out deposit addresses for wallets whose private keys it can no longer recover. An agent will pass those addresses to payers. Unless an independent backup of the phrase exists somewhere, payments to them are effectively lost. (Arktos implements no signing today, so even a healthy instance cannot spend; the point is that a correct restore would let the owner recover the phrase and spend elsewhere, and this one cannot.) This exact sequence is pinned in a test:

cargo test --test secret_storage_tests losing_the_master_key_leaves_only_already_public_data_usable

What should an operator, or a future version of Arktos, do to make this failure loud? Arktos has no "key-check value" that would detect a wrong MASTER_KEY at startup. Consider what adding one would cost and what it would leak.

Recovery combinations

Fill in this table before reading the answer below it.

Backups with SQLCipher and WAL

Arktos runs SQLite in WAL mode (src/database.rs). While it runs, committed transactions can live in arktos.db-wal and not yet be in arktos.db, so copying only arktos.db from a running server can lose committed wallets. The supported approaches, from Architecture — Files, Permissions and Backups:

A backup is still encrypted with DATABASE_KEY, and the phrases inside it still need MASTER_KEY. A backup without its keys is not a backup.

Lab

With the lab server and a wallet with at least one derived address:

Observe: an online backup

labsql "VACUUM INTO '$LAB/backup.db';"
ls -l "$LAB"

Note the backup's permissions. VACUUM INTO creates the file with your umask, often 0644, until Arktos opens it and tightens it to 0600. The contents are ciphertext, but treat the file as sensitive from the moment it exists.

Restore on a "clean machine"

DATABASE_PATH="$LAB/backup.db" cargo run --quiet --bin arktos-wallet -- db-info

db-info prints diagnostics only: the SQLCipher version, the schema version and the pragmas. It prints no data and no secrets. Then simulate each loss:

DATABASE_PATH="$LAB/backup.db" DATABASE_KEY=wrong cargo run --quiet --bin arktos-wallet -- db-info

To observe the MASTER_KEY trap end to end, restart the lab server with MASTER_KEY=$(make secret) against the same database. Watch mcp "$A" ping '{}' fail with 401, re-issue A's key through /admin/api-keys/{id}/rotate, and then request an index you derived earlier and an index you did not.

Exercise: the recovery inventory

Write the document an on-call engineer would need at 3 a.m. to answer: "What must I have to restore Arktos on a clean machine?" For each item, record:

• what it is, and where the authoritative copy lives;
• who can access it, and who can access it alone;
• how you would detect that it is wrong before serving traffic (compare the trap);
• how old it may be. The database backup has a recovery point objective (RPO). The keys must never be "old", because there is exactly one valid value of each.

Explain: should the keys live with the database backup?

Every extra separation reduces the number of people who can steal everything, and it increases the number of ways everything can be lost. Pick deliberately, write the choice down, and rehearse the restore.

Rotation and migration today

Failure mode

• Strong encryption with keys stored "somewhere safe" that nobody has ever restored from.
• Backing up arktos.db from a running server without its WAL.
• One bundle containing the database and both keys, which turns two layers into zero.
• "Recovering" with a fresh key and a server that looks healthy.

Takeaway

Encryption without a recovery plan can turn a security control into permanent data loss.

Related reference

• Architecture — Files, Permissions and Backups
• Deployment Guide — Generating Secrets, Persistence
• Development Guide — Database Operations
• Challenge 4 — HSM/KMS-backed keys

Previous: C7 · Next: Secret lifecycle walkthrough