MongoDB on Civo Kubernetes: Building for backups and point-in-time recovery

Running databases on Kubernetes still carries a reputation for being unreliable—and that skepticism is notoriously stronger for non-relational systems. Stateful workloads require durable storage, careful failover, and a recovery story that goes far beyond simply restarting a pod.

For MongoDB in particular, many teams stop at a basic StatefulSet and a scheduled mongodump. Unfortunately, this covers neither high availability (HA) nor point-in-time recovery (PITR).

Popular approaches often avoid this problem rather than solve it by relying on managed services like Atlas or keeping MongoDB outside the cluster entirely. But when your requirements dictate a self-managed MongoDB on Kubernetes—with durability, ACID semantics, and true recoverability—the operational gap becomes painfully clear.

Looking for a stack that treats backup and PITR as first-class concerns led us to the Percona Operator for MongoDB and Percona Backup for MongoDB (PBM). We deploy that stack on Civo Kubernetes. Here is our engineering decision record on why we chose this stack, the technical mechanics of how it works, and the topology we deployed.

The production-readiness scorecard

Major database platforms (Atlas, Aurora, Cloud SQL) describe production readiness using a shared vocabulary. They distinctly separate guarantees that teams often mistakenly treat as one:

• Durability: A committed write survives a crash (journal + durable storage). In our setup, PVCs use Civo’s default civo-volume class—NVMe over 25GbE, 99.95% volume SLA, no per-IOPS surcharge.
• Availability: The system stays up when a node or zone fails (replicas + automatic failover via Raft-like elections).
• Recoverability: You can restore after corruption, accidental deletes, or cluster loss (base backup + continuous oplog archival for PITR).

Atlas is explicit: high availability is not disaster recovery. Failover handles infrastructure failure; backups and PITR handle data integrity failure. Multi-document ACID consistency also requires replica-set semantics, not a bare standalone mongod.

Why not a plain StatefulSet?

Many teams begin with a mongod image inside a StatefulSet—one pod, one PVC, and maybe a CronJob wrapping mongodump.

With a plain StatefulSet, you own the hard parts yourself. This model usually breaks down in three specific areas:

• Replica Set Initialization: StatefulSets do not run MongoDB commands. You are responsible for generating keyfiles, executing rs.initiate(), and dynamically adding/removing members via rs.reconfig() when scaling.
• Split-Brain & Networking: If a pod restarts with a new IP and you haven't perfectly configured headless services and DNS, MongoDB nodes lose track of each other, leading to election loops or split-brain scenarios.
• Backups & True PITR: A CronJob doing a mongodump only provides logical snapshots at specific intervals. True PITR requires archiving the MongoDB oplog.rs collection continuously, which requires custom scripts and tracking upload cursors to object storage.

The solution: Percona operator + PBM architecture

The Percona Operator runs Percona Server for MongoDB (PSMDB) via Kubernetes Custom Resources (CRs). The operator is a Go-based controller loop that constantly watches the K8s API. If a node fails, the operator reconciles the desired state, orchestrating the safe replacement of the pod and ensuring the new member joins the replica set.

Crucially, Percona Backup for MongoDB (PBM) runs as a sidecar (pbm-agent) next to every mongod container. The agents coordinate via a control collection inside MongoDB itself to elect a backup leader (preferring secondary nodes to avoid hitting the primary with backup I/O).

Live data stays on civo-volume NVMe; PBM pushes bases and oplog slices to Civo Object Store over the S3 API.

How point-in-time recovery (PITR) actually works

PITR is easier to understand if you stop thinking of a "backup" as a single file. With PBM, you have two distinct layers:

• A Base Backup (Physical): Instead of a logical mongodump (which reads data into BSON files via the query engine), PBM performs a physical backup. It utilizes the $backupCursor command to safely copy the underlying WiredTiger data files directly to object storage. This is significantly faster for large datasets.
• Oplog Slices: Small, continuous uploads of everything that changed after that snapshot. The MongoDB oplog is an idempotent changelog of every write.

Restore means landing on the nearest good base backup, then having the PBM agent fetch and apply oplog slices forward until the exact moment you care about.

RPO (how much data you might lose) is bounded by your oplogSpanMin setting (typically 5–10 minutes). RTO (how long a restore takes) is dominated by copying the base backup back into the cluster and replaying the oplog. This is why frequent full snapshots still matter—compute-heavy oplog replay takes time, so keeping the replay window short is essential.

On Civo, continuous uploads to Object Store do not add egress fees, which makes a tight oplogSpanMin easier to justify; NVMe on the restore target helps the base land quickly.

Topologies & trade-offs: 1-node vs. 3-node

Our workload features important data but currently has low traffic. We had a hard requirement to stay self-managed on Kubernetes (Civo’s regions across the US, UK, EU, and India helped with latency). Against our scorecard, this meant buying durability and recoverability immediately, but deferring live multi-node availability until uptime becomes the constraint.

We opted for a 1-member replica set. It is critical to use a 1-member replica set and not a standalone mongod. Standalone nodes do not have a local database with an oplog.rs collection, meaning PBM cannot function, and they do not support multi-document transactions.

When our uptime requirements tighten, the migration is just an incremental config change: increase replsets[].size from 1 to 3, and change application connection strings to demand w: majority. The operator handles the node provisioning, network routing, and replica set reconfiguration automatically—scale the Civo node pool beside it so the new secondaries have CPU, RAM, and NVMe ready.

What we deployed (the declarative YAML)

Here is a simplified look at the declarative state that achieves full physical backups and 5-minute PITR on a single node:

1apiVersion: psmdb.percona.com/v1
2kind: PerconaServerMongoDB
3
4metadata:
5  name: mongo-prod
6
7spec:
8  crVersion: 1.16.0
9
10  replsets:
11    - name: rs0
12      size: 1
13
14      volumeSpec:
15        persistentVolumeClaim:
16          storageClassName: civo-volume
17          resources:
18            requests:
19              storage: 50Gi
20
21  backup:
22    enabled: true
23
24    pitr:
25      enabled: true
26      oplogSpanMin: 5
27
28    storages:
29      s3-main:
30        type: s3
31        s3:
32          bucket: mongo-prod-backups
33          region: LON1
34          credentialsSecret: mongo-backup-s3
35          endpointUrl: https://objectstore.lon1.civo.com
36
37    tasks:
38      - name: daily-physical
39        enabled: true
40        schedule: "0 2 * * *"
41        keep: 14
42        storageName: s3-main
43        type: physical

Day-2 rules for survival

A workable topology is not enough on its own. A few habits keep the recovery story honest:

• Monitor the metrics: The Percona Operator deploys an integrated Prometheus exporter. You must set up alerts on specific metrics: alert on pbm_backup_status if a base backup fails, and monitor pbm_pitr_status to ensure oplog uploads are not falling behind. A silent backup pipeline is vastly more dangerous than no pipeline at all.
• Treat object storage as part of the database: Use immutable or versioned buckets, set lifecycle retention policies you can explain, and utilize cross-region copies if region loss matters. We point PBM at Civo Object Store (S3-compatible)—live WiredTiger on NVMe, bases and oplog off-cluster.
• Drill your restores: Restore on a schedule into a scratch cluster (on Civo that is cheap and fast to stand up). Keep a short runbook outlining who performs the restore, how apps reconnect, and how PITR is re-enabled afterward.
• Define your scaling triggers: Decide in advance what triggers the move to three nodes (uptime, compliance, or traffic) so the upgrade is a planned size change, not a panicked incident response.

FAQs

Conclusion

Running important data on MongoDB in Kubernetes is less about picking a container image and more about deciding which guarantees you need—and proving you can restore when something goes wrong.

For us, the Percona Operator for MongoDB + PBM closed the operational gap. It offers declarative lifecycle management, physical backups, continuous oplog PITR to object storage, and a restore path we could actually execute—on Civo, with NVMe for the hot path and Object Store for PITR, without egress fees chewing into continuous oplog traffic.

Start small or go multi-node, tighten RPO with denser oplog archival, or add geographic DR when the business asks for it—all without throwing away your backup model.

Backups are inventory. Restores are proof. PITR is the difference between losing an hour and losing the wrong hour entirely.

Read more

If you're working with Kubernetes, Civo, and infrastructure storage, you may also like:

• Civo Kubernetes: Where Does Your Pod's Traffic Actually Go? — Understand Civo Kubernetes networking and pod traffic flow.
• AWS Charges Extra for Storage Performance That Civo Ships by Default — Compare Civo NVMe storage performance with AWS EBS.
• AWS RDS vs Civo Postgres: Performance, Latency, and Cost Compared — Compare managed database performance and cost between AWS and Civo.