Availability, Backups and Disaster Recovery

This page covers what keeps Parascope running, what is copied where when it stops, and how much of the recovery path has been exercised. The recovery figures further down are design targets taken from the platform's resilience design. They are the numbers the system is built and operated against, and the sections below are explicit about which parts of the path are tested on a schedule and which are not.

The availability target

Parascope is operated against a target of 99.9% monthly availability, measured per tenant. That allows roughly 43 minutes of unavailability in a month before the target is missed.

The target carries a definition, because a number without one cannot be checked. A tenant counts as available when that tenant's API answers a health request with HTTP 200 inside five seconds, and the health check queries the tenant's database on the way through, so a healthy web process sitting in front of an unreachable database reads as unavailable. The accounting is per tenant: one tenant's outage counts against that tenant and not against the others, which follows from each tenant running its own API and processor.

Planned maintenance sits outside the target. Deploys roll one pod at a time behind the load balancer, and tenant namespaces are updated in sequence with each one verified before the next starts, so a routine release is not expected to register as downtime.

Coverage is a separate question from the target, and it is worth stating plainly: Parascope does not run a 24/7 on-call rotation. Alerting is automated and reaches the operator by email and push notification, and response happens in business hours. Incident Response is where the response commitments are written down. The 99.9% figure describes how the platform is designed and operated; anything you need as a contractual commitment is a conversation for [email protected].

The status page

Platform status and incident history are published at parascope.instatus.com. The page is hosted by a third party outside the Parascope cluster, so an outage that takes the platform down does not take its own status reporting with it. Closed incidents stay on the page, so the history is readable without asking anyone for it.

What is backed up

Three layers sit under the database, each recovering a different kind of loss.

The first two land in object storage at Hetzner, in a different failure domain from the cluster that produced them. The third exists because the first two leave a single provider account holding both the running platform and its backups, and an account-level incident should not be able to take both.

Backup objects are encrypted before upload, so the bytes at rest in either store are ciphertext. Encryption names the ciphers and states what the key handling looks like. Neither encryption nor off-provider replication is optional in production: the wrapper that applies a production configuration change refuses to run unless both are switched on in the live configuration, and an operator overriding that has to say so explicitly.

Two things about retention are worth being exact about. On the primary store, retention is the work of scheduled jobs: four weekly base backups are kept along with the write-ahead log needed to roll forward from the oldest of them, and tenant dumps age out on the seven-daily, four-weekly, three-monthly schedule above. The off-provider mirror is additive: the hourly sync copies and never deletes, so a scheduled prune is what ages it out. The prune removes mirrored objects past an age threshold of its own, and the sync re-copies anything the primary store still holds, so a mirrored copy survives until the primary has aged the original out and that threshold has passed. Mirrored write-ahead-log copies are bounded at sixty days on that path, and a deleted tenant's rows persist in the encrypted backup set for roughly ninety days at the outside, which is the figure Data Handling and Retention relies on. Those windows are the firm answer for an erasure request, not a first offer. Backups are not edited to remove individual records, and the mirrored copies sit under a bucket lock that refuses deletion to every credential the running platform holds, for twenty-five days on tenant dumps and fifty-five days on write-ahead-log archives. So a request sent to [email protected] is actioned against the live systems straight away, and the backup copies age out on the schedules just described: mirrored write-ahead-log copies at sixty days, and the set as a whole inside the ninety-day bound above. Note which number is the honest one. The prune threshold on the dump prefixes is thirty days, but the hourly sync re-copies any dump the primary store still holds, so those mirrored copies track the primary ladder rather than the threshold. Ninety days is the bound to hold us to. We do not lift the lock to beat it.

What is verified, and how often

A backup nobody has restored is a hypothesis. A verification job runs weekly, on Wednesdays, and does two separate things.

It checks the point-in-time recovery chain: that at least one base backup exists, and that every write-ahead log segment between the oldest base backup and the cluster's current position is present in storage. A gap there is what would make a point-in-time restore impossible. That step reads the production primary read-only.

It then restores tenant dumps. The job starts a throwaway PostgreSQL server inside its own pod, restores the selected tenants' latest dumps into it, checks each restored database's schema against the current migration head and its row counts against the live source, and discards the scratch server when the pod exits. Production is read, and nothing is written to it. Tenants are picked by a rotating window over the sorted tenant list, so coverage is bounded and predictable. A run that finds no backups fails. A separate weekly job fails when the newest base backup is older than its freshness threshold. Both jobs publish a result that goes stale and alerts if a run stops happening.

Here is the limit, and it matters for reading the table below. The weekly job does not perform a restore with write-ahead log replay to a chosen timestamp. That is a manual drill, run against a throwaway cluster and not against production.

The off-provider side is checked on two cadences. A scheduled job runs daily against the mirror itself, reading the marker the sync writes on success and the one the disaster-recovery secret bundle writes on upload, and publishing both as freshness metrics that the alerting stack watches. A sync that stops feeding the mirror trips a staleness alert once its marker ages past the threshold. The deeper test stays manual and quarterly: the encrypted secret bundle is decrypted and listed, and an offsite tenant dump is restored into a local container and row-counted. A catalogue of failure scenarios sits alongside those tests, each scenario carrying simulation steps and the recovery behaviour to expect, on a quarterly rotation.

Recovery targets

The figures below come from the platform's resilience design. They are targets, and the measured facts are the ones in the section above.

The first row is the one that resolves without anybody being woken up. The database runs as a three-member cluster under a high-availability manager with asynchronous streaming replication, and a failed primary is replaced automatically by the standby that has replayed the most of its history. Asynchronous means a standby can trail the primary, so the loss bound is stated rather than waved at: a standby is only eligible for promotion while it trails by less than 4 MiB of write-ahead log, which makes 4 MiB the most an unplanned failover can lose. In steady state the replicas track the primary at close to zero lag, and alerting fires at half the bound, so the ordinary distance to that worst case is monitored rather than assumed.

The second row is the ordinary restore path, and the weekly job exercises its closest analogue. Its data-loss figure is set by the artefact it restores from: one tenant's database comes back from that tenant's most recent daily dump, which can be up to a day old. A recovery point tighter than that means replaying the write-ahead log at cluster level, which is the third row.

The third row assumes provisioning a new cluster and restoring into it. Its one-hour figure follows from the archive and mirror cadences, not from a stopwatch held over a rehearsal.

What this does not cover

Single region, no standby site. The platform runs in one region, in Helsinki. There is no second region on warm standby and no automatic failover between providers. The off-provider mirror holds backup data, and turning it back into a running service means provisioning a cluster and restoring into it, which is the four-hour row above.

The recovery targets are design targets. What runs on a schedule is the chain-integrity check and the dump restores. A full timestamp-targeted replay of the write-ahead log has not been exercised against production, so treat the cluster-level figures as the design intent they are and ask for the drill record if your review needs more than that.

Response is not around the clock. Detection is automated and continuous. The human on the other end of it works business hours, and Incident Response is where that is written down in the form of commitments you can hold us to.

Verify it yourself

• The status page is independent of the platform. Open parascope.instatus.com and read the incident history. It is hosted outside the cluster it reports on, which is what makes it readable during an outage.
• Ask for the current evidence. [email protected] will give you the date of the most recent successful backup verification run and of the most recent restore drill. Both answers are dates, so you can hold them against the cadences this page claims.
• Backup encryption is stated precisely elsewhere. Encryption covers which cipher protects which artefact, and where the keys live, including what at-rest encryption does not defend against.
• The deletion side is documented. Data Handling and Retention covers what deprovisioning removes and what stays in backups afterwards. Erasure questions go to [email protected].
• Controls with their own verify pointers are listed in the control matrix.