Setting up and managing PostgreSQL operator with MinIO on Kubernetes

In this article, we will walk through the process of setting up the PostgreSQL Operator, MinIO for backups, and HostPath CSI driver on a Kubernetes cluster. This setup is essential for managing PostgreSQL clusters and ensuring data backups using MinIO.

To begin, let's review the environment details used in this setup:

• Client Version: v1.30.2
• Kustomize Version: v5.0.4-0
• Server Version: v1.30.2
• Database: PostgreSQL 16.3
• Operator: cloud-native-postgresql 1.23.2

First, we need to deploy the PostgreSQL Operator, which simplifies the management of PostgreSQL clusters on Kubernetes.

kubectl apply --server-side -f https://get.enterprisedb.io/cnp/postgresql-operator-1.23.2.yaml

This command deploys the PostgreSQL Operator and related Custom Resource Definitions (CRDs), service accounts, roles, and deployments required to manage PostgreSQL clusters.

Verify the deployment:

kubectl get deployment -n postgresql-operator-system postgresql-operator-controller-manager

Expected output:

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
postgresql-operator-controller-manager   1/1     1            1           58s

MinIO is a high-performance, S3-compatible object storage, which we will use for storing PostgreSQL backups.

The values provided in the below YAML are Base64 encoded strings.

• ACCESS_KEY_ID: bWluaW8= Decoded value: minio
• ACCESS_SECRET_KEY: bWluaW8xMjM= Decoded value: minio123

Create a minio.yaml file with the following content:

apiVersion: v1
kind: Secret
metadata:
  name: minio-creds
data:
  ACCESS_KEY_ID: bWluaW8=
  ACCESS_SECRET_KEY: bWluaW8xMjM=

---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
spec:
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: standard

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: minio-pv-claim
      containers:
      - name: minio
        image: minio/minio:RELEASE.2024-07-10T18-41-49Z
        args:
        - server
        - /data
        env:
        - name: MINIO_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: minio-creds
              key: ACCESS_KEY_ID
        - name: MINIO_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: minio-creds
              key: ACCESS_SECRET_KEY
        ports:
        - containerPort: 9000
        readinessProbe:
          httpGet:
            path: /minio/health/ready
            port: 9000
          initialDelaySeconds: 30
        livenessProbe:
          httpGet:
            path: /minio/health/live
            port: 9000
          initialDelaySeconds: 30

Apply the MinIO configuration:

kubectl apply -f minio.yaml

Verify the MinIO deployment:

kubectl get deployment minio

Expected output:

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
minio   1/1     1            1           70s

To enable volume snapshots for backups, we need to deploy the HostPath CSI driver.

Create a deploy-hostpath-csi.sh script with the following content:

#!/bin/env bash

CSI_BASE_URL=https://raw.githubusercontent.com/kubernetes-csi
CSI_DRIVER_HOST_PATH_VERSION=v1.11.0
SNAPSHOTTER_VERSION="v6.3.1"
PROVISIONER_VERSION="v3.6.1"
RESIZER_VERSION="v1.9.1"
ATTACHER_VERSION="v4.4.1"

## Install external snapshotter CRD
kubectl apply -f "${CSI_BASE_URL}"/external-snapshotter/"${SNAPSHOTTER_VERSION}"/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f "${CSI_BASE_URL}"/external-snapshotter/"${SNAPSHOTTER_VERSION}"/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f "${CSI_BASE_URL}"/external-snapshotter/"${SNAPSHOTTER_VERSION}"/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
kubectl apply -f "${CSI_BASE_URL}"/external-snapshotter/"${SNAPSHOTTER_VERSION}"/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
kubectl apply -f "${CSI_BASE_URL}"/external-snapshotter/"${SNAPSHOTTER_VERSION}"/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml
kubectl apply -f "${CSI_BASE_URL}"/external-snapshotter/"${SNAPSHOTTER_VERSION}"/deploy/kubernetes/csi-snapshotter/rbac-csi-snapshotter.yaml

## Install external provisioner
kubectl apply -f "${CSI_BASE_URL}"/external-provisioner/"${PROVISIONER_VERSION}"/deploy/kubernetes/rbac.yaml

## Install external attacher
kubectl apply -f "${CSI_BASE_URL}"/external-attacher/"${ATTACHER_VERSION}"/deploy/kubernetes/rbac.yaml

## Install external resizer
kubectl apply -f "${CSI_BASE_URL}"/external-resizer/"${RESIZER_VERSION}"/deploy/kubernetes/rbac.yaml

## Install driver and plugin
kubectl apply -f "${CSI_BASE_URL}"/csi-driver-host-path/"${CSI_DRIVER_HOST_PATH_VERSION}"/deploy/kubernetes-1.24/hostpath/csi-hostpath-driverinfo.yaml
kubectl apply -f "${CSI_BASE_URL}"/csi-driver-host-path/"${CSI_DRIVER_HOST_PATH_VERSION}"/deploy/kubernetes-1.24/hostpath/csi-hostpath-plugin.yaml

## create volumesnapshotclass
kubectl apply -f "${CSI_BASE_URL}"/csi-driver-host-path/"${CSI_DRIVER_HOST_PATH_VERSION}"/deploy/kubernetes-1.24/hostpath/csi-hostpath-snapshotclass.yaml

## create storage class
kubectl apply -f "${CSI_BASE_URL}"/csi-driver-host-path/"${CSI_DRIVER_HOST_PATH_VERSION}"/examples/csi-storageclass.yaml

Run the script:

bash deploy-hostpath-csi.sh

% kubectl get deployment -A                                               
NAMESPACE                    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
default                      minio                                    1/1     1            1           2d19h
kube-system                  coredns                                  2/2     2            2           2d20h
kube-system                  snapshot-controller                      2/2     2            2           2d19h
local-path-storage           local-path-provisioner                   1/1     1            1           2d20h
postgresql-operator-system   postgresql-operator-controller-manager   1/1     1            1           2d19h

Command to restart the deployment:

% kubectl rollout restart deployment -n postgresql-operator-system postgresql-operator-controller-manager

Expected output:

deployment.apps/postgresql-operator-controller-manager restarted

Verify the CSI driver deployment:

kubectl get deployment -A

Expected output includes:

NAMESPACE                    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system                  snapshot-controller                      2/2     2            2           53s
default                      minio                                    1/1     1            1           70s
postgresql-operator-system   postgresql-operator-controller-manager   1/1     1            1           2m15s

Now, let's create a PostgreSQL cluster and configure it to use MinIO for backups.

Create a cluster-example-with-backup.yaml file with the following content:

apiVersion: postgresql.k8s.enterprisedb.io/v1
kind: Cluster
metadata:
  name: cluster-example-backup
spec:
  instances: 2

  storage:
    storageClass: csi-hostpath-sc
    size: 1Gi

  # Backup properties
  # This assumes a local minio setup
  backup:
    volumeSnapshot:
      className: csi-hostpath-snapclass
    barmanObjectStore:
      destinationPath: s3://cluster-backups/
      endpointURL: http://minio-service:9000
      s3Credentials:
        accessKeyId:
          name: minio-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: minio-creds
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
      data:
        immediateCheckpoint: true
    retentionPolicy: "30d"

Apply the configuration:

kubectl apply -f cluster-example-with-backup.yaml

Verify the cluster status:

kubectl cnp status cluster-example-backup

Expected output:

Cluster Summary
Name:                cluster-example-backup
Namespace:           default
System ID:           7389252359945887770
PostgreSQL Image:    quay.io/enterprisedb/postgresql:16.3
Primary instance:    cluster-example-backup-1
Primary start time:  2024-07-08 13:13:19 +0000 UTC (uptime 2m39s)
Status:              Cluster in healthy state 
Instances:           2
Ready instances:     2

To create a volume snapshot backup, create a volume-snapshot-backup.yaml file:

apiVersion: postgresql.k8s.enterprisedb.io/v1
kind: Backup
metadata:
  name: volume-snapshot-backup
spec:
  cluster: cluster-example-backup
  method: volumesnapshot
  volumesnapshot:
    className: csi-hostpath-snapclass

Apply the backup configuration:

kubectl apply -f volume-snapshot-backup.yaml

Verify the backup status:

kubectl cnp status backup volume-snapshot-backup

Expected output:

Backup Summary
Name:             volume-snapshot-backup
Cluster:          cluster-example-backup
Type:             volumeSnapshot
Status:           Completed
Start time:       2024-07-08 14:30:47 +0000 UTC (started 4s ago)
End time:         2024-07-08 14:30:51 +0000 UTC (took 4s)
Volume Snapshots: 2

To create a Barman backup for your PostgreSQL cluster, follow these steps:

1. Create a backup.yaml file with the following content:

   apiVersion: postgresql.k8s.enterprisedb.io/v1
   kind: Backup
   metadata:
     name: backup-example
   spec:
     cluster:
       name: cluster-example-backup
   

2. Apply the backup configuration:

   kubectl apply -f backup.yaml
   

3. Verify the backup status:

   kubectl get backup
   

   Expected output:

   NAME                     AGE     CLUSTER                  METHOD              PHASE       ERROR
   backup-example           18s     cluster-example-backup   barmanObjectStore   completed   
   snapshot-backup-1.23.2   2d19h   cluster-example-backup   volumeSnapshot      completed   
   

4. Get the detailed information about the backup:

   kubectl describe backup backup-example
   

   Example output:

   Name:         backup-example
   Namespace:    default
   Labels:       <none>
   Annotations:  <none>
   API Version:  postgresql.k8s.enterprisedb.io/v1
   Kind:         Backup
   Metadata:
     Creation Timestamp:  2024-07-11T08:21:30Z
     Generation:          1
     Resource Version:    81003
     UID:                 edb065b4-41d9-433c-82d0-2f0deceee4c1
   Spec:
     Cluster:
       Name:  cluster-example-backup
     Method:  barmanObjectStore
   Status:
     Backup Id:         20240711T082133
     Backup Name:       backup-20240711082132
     Begin LSN:         0/6008870
     Begin Wal:         000000010000000000000006
     Destination Path:  s3://cluster-backups/
     End LSN:           0/8000000
     End Wal:           000000010000000000000008
     Endpoint URL:      http://minio-service:9000
     Instance ID:
       Container ID:  containerd://ddcb4c8e5ddac49991b1a363d4619ec9b0e15bebbdab4e0f831bd57ac4057f55
       Pod Name:      cluster-example-backup-2
     Method:          barmanObjectStore
     Phase:           completed
     s3Credentials:
       Access Key Id:
         Key:   ACCESS_KEY_ID
         Name:  minio-creds
       Secret Access Key:
         Key:      ACCESS_SECRET_KEY
         Name:     minio-creds
     Server Name:  cluster-example-backup
     Started At:   2024-07-11T08:21:33Z
     Stopped At:   2024-07-11T08:21:34Z
   Events:
     Type    Reason     Age   From                            Message
     ----    ------     ----  ----                            -------
     Normal  Starting   89s   cloud-native-postgresql-backup  Starting backup for cluster cluster-example-backup
     Normal  Starting   88s   instance-manager                Backup started
     Normal  Completed  84s   instance-manager                Backup completed
   

To verify the backup on the MinIO web interface, follow these steps:

1. Port-forward the MinIO service:

   kubectl port-forward svc/minio-service 9000:9000
   

   You should see the output indicating the port-forwarding is active:

   Forwarding from 127.0.0.1:9000 -> 9000
   Forwarding from [::1]:9000 -> 9000
   Handling connection for 9000
   Handling connection for 9000
   Handling connection for 9000
   

2. Open your web browser and navigate to:

   http://localhost:9000
   

Once you access the MinIO web interface:

1. Log in using the credentials (ACCESS_KEY_ID and ACCESS_SECRET_KEY) defined in your minio-creds secret.
2. Navigate to the cluster-backups bucket.
3. You should see a folder named cluster-example-backup/ containing base/ and wals/ directories, which store the base backup and WAL segments respectively.

This completes the setup, backup creation, and verification process using PostgreSQL Operator with MinIO and HostPath CSI driver on Kubernetes.

• MinIO Gateway
• Common object stores
• Backup on volume snapshots
• Backup