Backup to S3 – Configure Zalando Postgres Operator Backup with WAL-G

Doing basebackups of Postgres databases enables you to do a Point-in-time recovery (PITR) of your database. Doing backups for your databases that have been deployed using the Zalando Postgres Operator, is a different beast. Here I show you how it’s done with a MinIO (self)hosted S3 compatible Object Storage.

The so called Spilo images that are deployed when using the Zalando Postgres Operator, can do backups and WAL archiving to S3 (compatible) storage using WAL-E or it’s successor WAL-G. For me the problem is, that the documentation on WAL-G integration on Zalando side is not very good. You have to put quite some puzzle pieces together in order to get it running. Because I went through this process, I thought, it might come handy for you too.

I assume, that you’ve the following prerequisites setup:

• Zalando Postgres Operator is deployed on your Kubernetes cluster. If you search for a tutorial on that topic, you can find it here.
• You have a working S3 Object Storage up and running. If you want to setup your own, self-hosted Object Storage using MinIO, you can find the instructions for it here.

Attention: I use my own MinIO Object Storage in this example. There might be some different configurations for you, if you use a different provider. I will try to mention MinIO specific parameters inline. But your milage may vary.

If you want to quickstart the whole process, you can directly head to my Github repository and apply the kustomization overlay under overlays/enabled-backup.

What the overlay does is first add (patch) the Zalando Operator central configmap with the parameter pod_environment_configmap. This is the reference to a Pod specific Configmap which holds the environment variables that configures WAL-G to use our S3 Object Storage.

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-operator
data:
  pod_environment_configmap: "postgres-operator/pod-config"

If your Pod specific Configmap resides in a Namespace other than default, you need to specify the Namespace before the name of the Configmap (postgres-operator in my case).

The Pod specific Configmap holds these configuration in my case.

apiVersion: v1
kind: ConfigMap
metadata:
  name: pod-config
data:
  WAL_S3_BUCKET: postgresql
  WAL_BUCKET_SCOPE_PREFIX: ""
  WAL_BUCKET_SCOPE_SUFFIX: ""
  USE_WALG_BACKUP: "true"
  USE_WALG_RESTORE: "true"
  BACKUP_SCHEDULE: '00 10 * * *'
  AWS_ACCESS_KEY_ID: postgresql
  AWS_SECRET_ACCESS_KEY: supersecret
  AWS_S3_FORCE_PATH_STYLE: "true" # needed for MinIO
  AWS_ENDPOINT: http://minio.home.lab:9000 # Endpoint URL to your S3 Endpoint; MinIO in this example
  AWS_REGION: de01
  WALG_DISABLE_S3_SSE: "true"
  BACKUP_NUM_TO_RETAIN: "5"
  CLONE_USE_WALG_RESTORE: "true"

Let’s have a look on the parameters.

The kustomization overlay looks like this (see the Github repository mentioned above):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: postgres-operator

resources:
  - pod-config.yaml
  - ../../base
  - ../../ui

patchesStrategicMerge:
  - configmap.yaml

You may now apply the kustomization overlay like this:

kubectl apply -k overlays/enabled-backup/

This will deploy the Zalando Operator with all needed adjustments regarding backup. If you have already had deployed the Operator, it will patch the postgres-operator Configmap. You will need to restart the Operator Pod in order to get the Pod specific Configmap applied.

Now let’s quickly build a Postgres cluster with the Operator. You can find some examples in my Github repository under the manifests folder.

apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  name: postgres-demo-cluster
  namespace: postgres
spec:
  teamId: "postgres"
  volume:
    size: 2Gi
  numberOfInstances: 2
  users:
    demouser:  # database owner
    - superuser
    - createdb
  databases:
    demo: demouser  # dbname: owner
  preparedDatabases:
    demo: {}
  postgresql:
    version: "14"

Let’s deploy it:

kubectl apply -f demo-cluster.yaml

After a short time, your cluster should get started and directly after the cluster has been created, a backup will be made (regardless of the schedule you’ve configured). Execute into the container, if your configuration works, you should see all environment variables set as an envdir under /run/etc/wal-e.d/env.

> /run/etc/wal-e.d/env# ls -ltr
total 68
-rw-r--r-- 1 postgres root  1 Mar 26 15:59 WALG_UPLOAD_CONCURRENCY
-rw-r--r-- 1 postgres root 50 Mar 26 15:59 WALG_S3_PREFIX
-rw-r--r-- 1 postgres root  1 Mar 26 15:59 WALG_DOWNLOAD_CONCURRENCY
-rw-r--r-- 1 postgres root  4 Mar 26 15:59 WALG_DISABLE_S3_SSE
-rw-r--r-- 1 postgres root 50 Mar 26 15:59 WALE_S3_PREFIX
-rw-r--r-- 1 postgres root 31 Mar 26 15:59 WALE_S3_ENDPOINT
-rw-r--r-- 1 postgres root  4 Mar 26 15:59 USE_WALG_RESTORE
-rw-r--r-- 1 postgres root  4 Mar 26 15:59 USE_WALG_BACKUP
-rw-r--r-- 1 postgres root  8 Mar 26 15:59 AWS_SECRET_ACCESS_KEY
-rw-r--r-- 1 postgres root  4 Mar 26 15:59 AWS_S3_FORCE_PATH_STYLE
-rw-r--r-- 1 postgres root  4 Mar 26 15:59 AWS_REGION
-rw-r--r-- 1 postgres root 26 Mar 26 15:59 AWS_ENDPOINT
-rw-r--r-- 1 postgres root 10 Mar 26 15:59 AWS_ACCESS_KEY_ID
-rw-r--r-- 1 postgres root  6 Mar 26 15:59 WALE_LOG_DESTINATION
-rw-r--r-- 1 root     root 25 Mar 26 15:59 TMPDIR
-rw-r--r-- 1 postgres root  4 Mar 26 15:59 PGPORT
-rw-r--r-- 1 postgres root  1 Mar 26 15:59 BACKUP_NUM_TO_RETAIN

You can check for created backups using the WAL-G client. Issue the following command as root user from within your Postgres container.

> envdir "/run/etc/wal-e.d/env" wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000004 2022-03-26T16:00:13Z 000000010000000000000004

If you are not able to see any backups or you get error messages when using the backup-list command, check for more information in the Pod logs of your Postgres cluster Pods.

As in my case, I can use the MinIO CLI client to view the objects stored on my S3 storage.

> mc ls minio/postgresql/spilo/postgres-demo-cluster/wal/14/basebackups_005/
[2022-03-26 17:00:13 CET] 174KiB STANDARD base_000000010000000000000004_backup_stop_sentinel.json
[2022-03-26 17:19:06 CET]     0B base_000000010000000000000004/

But WAL-G does more than only basebackups, because we setup WAL-G, the Zalando Operator configured our PostgreSQL database to use WAL-G for WAL archiving too. You can find WAL archives under the following path on your S3:

> mc ls minio/postgresql/spilo/postgres-demo-cluster/wal/14/wal_005/
[2022-03-26 16:59:19 CET] 4.2MiB STANDARD 000000010000000000000001.lz4
[2022-03-26 16:59:27 CET]   255B STANDARD 000000010000000000000002.00000028.backup.lz4
[2022-03-26 16:59:27 CET]  65KiB STANDARD 000000010000000000000002.lz4
[2022-03-26 17:00:11 CET] 184KiB STANDARD 000000010000000000000003.lz4
[2022-03-26 17:00:12 CET]   266B STANDARD 000000010000000000000004.00000028.backup.lz4
[2022-03-26 17:00:12 CET]  65KiB STANDARD 000000010000000000000004.lz4

So our backup is working. As mentioned above, restore and cloning is a differnt topic and I promise to write about it soon.

Philip

Ähnliche Beiträge:

Restore and Clone from S3 – Configure Zalando Postgres Operator Restore with WAL-G Keep the Elephants in line – Deploy Zalando Postgres Operator on your Kubernetes cluster Kubernetes cert-manager mit Hetzner DNS nutzen SHORTY – How to deploy Uptime-Kuma on Kubernetes