This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Pepr Tutorials

1: Tutorial - Create a Pepr Module
2: Tutorial - Create a Pepr Dashboard
3: Building a Kubernetes Operator with Pepr

In this section, we provide tutorials for using Pepr. These tutorials are:

1 - Tutorial - Create a Pepr Module

Introduction

This tutorial will walk you through the process of creating a Pepr module.

Each Pepr Module is it’s own Typescript project, produced by npx pepr init. Typically a module is maintained by a unique group or system. For example, a module for internal Zarf mutations would be different from a module for Big Bang. An important idea with modules is that they are wholly independent of one another. This means that 2 different modules can be on completely different versions of Pepr and any other dependencies; their only interaction is through the standard K8s interfaces like any other webhook or controller.

Prerequisites

Steps

Create the module:
Use npx pepr init to generate a new module.

Quickly validate system setup:

Every new module includes a sample Pepr Capability called HelloPepr. By default, this capability is deployed and monitoring the pepr-demo namespace. There is a sample yaml also included you can use to see Pepr in your cluster. Here’s the quick steps to do that after npx pepr init:

# cd to the newly-created Pepr module folder
cd my-module-name

# If you don't already have a local K8s cluster, you can set one up with k3d
npm run k3d-setup

# Launch pepr dev mode
# If using another local K8s distro instead of k3d, use `npx pepr dev --host host.docker.internal`
npx pepr dev

# From another terminal, apply the sample yaml
kubectl apply -f capabilities/hello-pepr.samples.yaml

# Verify the configmaps were transformed using kubectl, k9s or another tool

Create your custom Pepr Capabilities
Now that you have confirmed Pepr is working, you can now create new capabilities. You’ll also want to disable the HelloPepr capability in your module (pepr.ts) before pushing to production. You can disable by commenting out or deleting the HelloPepr variable below:
```
new PeprModule(cfg, [
  // Remove or comment the line below to disable the HelloPepr capability
  HelloPepr,

  // Your additional capabilities go here
]);
```
Note: if you also delete the capabilities/hello-pepr.ts file, it will be added again on the next npx pepr update so you have the latest examples usages from the Pepr SDK. Therefore, it is sufficient to remove the entry from your pepr.ts module config.
Build and deploy the Pepr Module
Most of the time, you’ll likely be iterating on a module with npx pepr dev for real-time feedback and validation Once you are ready to move beyond the local dev environment, Pepr provides deployment and build tools you can use.
npx pepr deploy - you can use this command to build your module and deploy it into any K8s cluster your current kubecontext has access to. This setup is ideal for CI systems during testing, but is not recommended for production use. See npx pepr deploy for more info.

Additional Information

By default, when you run npx pepr init, the module is not configured with any additional options. Currently, there are 3 options you can configure:

deferStart - if set to true, the module will not start automatically. You will need to call start() manually. This is useful if you want to do some additional setup before the module controller starts. You can also use this to change the default port that the controller listens on.
beforeHook - an optional callback that will be called before every request is processed. This is useful if you want to do some additional logging or validation before the request is processed.
afterHook - an optional callback that will be called after every request is processed. This is useful if you want to do some additional logging or validation after the request is processed.

You can configure each of these by modifying the pepr.ts file in your module. Here’s an example of how you would configure each of these options:

const module = new PeprModule(
  cfg,
  [
    // Your capabilities go here
  ],
  {
    deferStart: true,

    beforeHook: req => {
      // Any actions you want to perform before the request is processed, including modifying the request.
    },

    afterHook: res => {
      // Any actions you want to perform after the request is processed, including modifying the response.
    },
  }
);

// Do any additional setup before starting the controller
module.start();

Summary

Checkout some examples of Pepr modules in the excellent examples repo. If you have questions after that, please reach out to us on Slack or GitHub Issues

2 - Tutorial - Create a Pepr Dashboard

Introduction

This tutorial will walk you through the process of creating a dashboard to display your Pepr metrics. This dashboard will present data such as the number of validation requests processed, the number of mutation requests that were allowed, the number of errors that were processed, the number of alerts that were processed, the status of the Pepr pods, and the scrape duration of the Pepr pods. This dashboard will be created using Grafana. The dashboard will display data from Prometheus, which is a monitoring system that Pepr uses to collect metrics.

This tutorial is not intended for production, but instead is intended to show how to quickly scrape Pepr metrics. The Kube Prometheus Stack provides a starting point for a more production suitable way of deploying Prometheus in prod.

An example of what the dashboard will look like is shown below:

Pepr Dashboard

Note: The dashboard shown above is an example of what the dashboard will look like. The dashboard will be populated with data from your Pepr instance.

Steps

Step 1. Get Cluster Running With Your Pepr Module Deployed

You can learn more about how to create a Pepr module and deploy it in the Create a Pepr Module tutorial. The short version is:

#Create your cluster
k3d cluster create

#Create your module
npx pepr init

#Change directory to your module that was created using `npx pepr init`
npx pepr  dev
kubectl apply -f capabilities/hello-pepr.samples.yaml

#Deploy your module to the cluster
npx pepr deploy

Step 2: Create and Apply Our Pepr Dashboard to the Cluster

Create a new file called grafana-dashboard.yaml and add the following content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: pepr-dashboard
  namespace: default
data:
  pepr-dashboard.json: |
    {
    "__inputs": [
      {
        "name": "DS_PROMETHEUS",
        "label": "Prometheus",
        "description": "",
        "type": "datasource",
        "pluginId": "prometheus",
        "pluginName": "Prometheus"
      }
    ],
    "__elements": {},
    "__requires": [
      {
        "type": "grafana",
        "id": "grafana",
        "name": "Grafana",
        "version": "9.1.6"
      },
      {
        "type": "datasource",
        "id": "prometheus",
        "name": "Prometheus",
        "version": "1.0.0"
      },
      {
        "type": "panel",
        "id": "stat",
        "name": "Stat",
        "version": ""
      },
      {
        "type": "panel",
        "id": "timeseries",
        "name": "Time series",
        "version": ""
      }
    ],
    "annotations": {
      "list": [
        {
          "builtIn": 1,
          "datasource": {
            "type": "grafana",
            "uid": "-- Grafana --"
          },
          "enable": true,
          "hide": true,
          "iconColor": "rgba(0, 211, 255, 1)",
          "name": "Annotations & Alerts",
          "target": {
            "limit": 100,
            "matchAny": false,
            "tags": [],
            "type": "dashboard"
          },
          "type": "dashboard"
        }
      ]
    },
    "editable": true,
    "fiscalYearStartMonth": 0,
    "graphTooltip": 0,
    "id": null,
    "links": [],
    "liveNow": false,
    "panels": [
      {
        "collapsed": false,
        "gridPos": {
          "h": 1,
          "w": 24,
          "x": 0,
          "y": 0
        },
        "id": 18,
        "panels": [],
        "title": "Pepr Status",
        "type": "row"
      },
      {
        "datasource": {
          "type": "prometheus",
          "uid": "prometheus"
        },
        "description": "Pepr pod status by pod",
        "fieldConfig": {
          "defaults": {
            "color": {
              "mode": "thresholds"
            },
            "mappings": [
              {
                "options": {
                  "0": {
                    "color": "red",
                    "index": 1,
                    "text": "Down"
                  },
                  "1": {
                    "color": "green",
                    "index": 0,
                    "text": "Up"
                  }
                },
                "type": "value"
              },
              {
                "options": {
                  "match": "empty",
                  "result": {
                    "color": "blue",
                    "index": 2,
                    "text": "?"
                  }
                },
                "type": "special"
              }
            ],
            "min": 0,
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 0,
          "y": 1
        },
        "id": 14,
        "options": {
          "colorMode": "value",
          "graphMode": "none",
          "justifyMode": "auto",
          "orientation": "auto",
          "reduceOptions": {
            "calcs": [
              "last"
            ],
            "fields": "",
            "values": false
          },
          "text": {
            "titleSize": 16,
            "valueSize": 70
          },
          "textMode": "auto"
        },
        "pluginVersion": "9.1.6",
        "targets": [
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "up{container=\"server\"}",
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "A"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "up{container=\"watcher\"}",
            "hide": false,
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "B"
          }
        ],
        "title": "Pepr Status",
        "type": "stat"
      },
      {
        "datasource": {
          "type": "prometheus",
          "uid": "prometheus"
        },
        "fieldConfig": {
          "defaults": {
            "color": {
              "mode": "palette-classic"
            },
            "custom": {
              "axisCenteredZero": false,
              "axisColorMode": "text",
              "axisLabel": "",
              "axisPlacement": "auto",
              "barAlignment": 0,
              "drawStyle": "line",
              "fillOpacity": 0,
              "gradientMode": "none",
              "hideFrom": {
                "legend": false,
                "tooltip": false,
                "viz": false
              },
              "lineInterpolation": "linear",
              "lineWidth": 1,
              "pointSize": 5,
              "scaleDistribution": {
                "type": "linear"
              },
              "showPoints": "auto",
              "spanNulls": false,
              "stacking": {
                "group": "A",
                "mode": "none"
              },
              "thresholdsStyle": {
                "mode": "off"
              }
            },
            "mappings": [],
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 12,
          "y": 1
        },
        "id": 12,
        "options": {
          "legend": {
            "calcs": [],
            "displayMode": "list",
            "placement": "bottom",
            "showLegend": true
          },
          "tooltip": {
            "mode": "single",
            "sort": "none"
          }
        },
        "targets": [
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "scrape_duration_seconds{container=\"server\"}",
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "A"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "scrape_duration_seconds{container=\"watcher\"}",
            "hide": false,
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "B"
          }
        ],
        "title": "Scrape Duration Seconds",
        "type": "timeseries"
      },
      {
        "collapsed": false,
        "gridPos": {
          "h": 1,
          "w": 24,
          "x": 0,
          "y": 9
        },
        "id": 6,
        "panels": [],
        "title": "Error, Alert, Validate and Mutate Counts",
        "type": "row"
      },
      {
        "datasource": {
          "type": "prometheus",
          "uid": "prometheus"
        },
        "fieldConfig": {
          "defaults": {
            "color": {
              "fixedColor": "dark-red",
              "mode": "fixed"
            },
            "mappings": [],
            "min": 0,
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 6,
          "x": 0,
          "y": 10
        },
        "id": 16,
        "options": {
          "colorMode": "value",
          "graphMode": "area",
          "justifyMode": "auto",
          "orientation": "auto",
          "reduceOptions": {
            "calcs": [
              "last"
            ],
            "fields": "",
            "values": false
          },
          "text": {
            "titleSize": 16,
            "valueSize": 70
          },
          "textMode": "auto"
        },
        "pluginVersion": "9.1.6",
        "targets": [
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "count by(instance) (rate(pepr_errors{container=\"server\"}[$__rate_interval]))",
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "A"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "count by(instance) (rate(pepr_errors{container=\"watcher\"}[$__rate_interval]))",
            "hide": false,
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "B"
          }
        ],
        "title": "Pepr: Error Count",
        "type": "stat"
      },
      {
        "datasource": {
          "type": "prometheus",
          "uid": "prometheus"
        },
        "description": "Count of Pepr Alerts by pod",
        "fieldConfig": {
          "defaults": {
            "color": {
              "fixedColor": "dark-yellow",
              "mode": "fixed"
            },
            "mappings": [],
            "min": 0,
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 6,
          "x": 6,
          "y": 10
        },
        "id": 10,
        "options": {
          "colorMode": "value",
          "graphMode": "area",
          "justifyMode": "auto",
          "orientation": "auto",
          "reduceOptions": {
            "calcs": [
              "last"
            ],
            "fields": "",
            "values": false
          },
          "text": {
            "titleSize": 16,
            "valueSize": 70
          },
          "textMode": "auto"
        },
        "pluginVersion": "9.1.6",
        "targets": [
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "pepr_alerts{container=\"server\"}",
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "A"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "pepr_alerts{container=\"watcher\"}",
            "hide": false,
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "B"
          }
        ],
        "title": "Pepr: Alert Count",
        "type": "stat"
      },
      {
        "datasource": {
          "type": "prometheus",
          "uid": "prometheus"
        },
        "description": "Count of Pepr Validate actions by pod",
        "fieldConfig": {
          "defaults": {
            "color": {
              "fixedColor": "dark-purple",
              "mode": "fixed"
            },
            "mappings": [],
            "min": 0,
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 6,
          "x": 12,
          "y": 10
        },
        "id": 4,
        "options": {
          "colorMode": "value",
          "graphMode": "area",
          "justifyMode": "auto",
          "orientation": "auto",
          "reduceOptions": {
            "calcs": [
              "last"
            ],
            "fields": "",
            "values": false
          },
          "text": {
            "titleSize": 16,
            "valueSize": 66
          },
          "textMode": "auto"
        },
        "pluginVersion": "9.1.6",
        "targets": [
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "exemplar": false,
            "expr": "pepr_validate_count{container=\"server\"}",
            "instant": false,
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "A"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "pepr_validate_sum{container=\"watcher\"}",
            "hide": false,
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "B"
          }
        ],
        "title": "Pepr: Validate Count",
        "type": "stat"
      },
      {
        "datasource": {
          "type": "prometheus",
          "uid": "prometheus"
        },
        "description": "Count of Pepr mutate actions applied by pod.",
        "fieldConfig": {
          "defaults": {
            "color": {
              "fixedColor": "dark-blue",
              "mode": "fixed"
            },
            "mappings": [],
            "min": 0,
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 6,
          "x": 18,
          "y": 10
        },
        "id": 2,
        "options": {
          "colorMode": "value",
          "graphMode": "area",
          "justifyMode": "auto",
          "orientation": "auto",
          "reduceOptions": {
            "calcs": [
              "last"
            ],
            "fields": "",
            "values": false
          },
          "text": {
            "titleSize": 16,
            "valueSize": 70
          },
          "textMode": "value_and_name"
        },
        "pluginVersion": "9.1.6",
        "targets": [
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "pepr_mutate_count{container=\"server\"}",
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "A"
          },
          {
            "datasource": {
              "type": "prometheus",
              "uid": "prometheus"
            },
            "editorMode": "builder",
            "expr": "rate(pepr_mutate_count{container=\"watcher\"}[24h])",
            "hide": false,
            "legendFormat": "{{instance}}",
            "range": true,
            "refId": "B"
          }
        ],
        "title": "Pepr: Mutate Count",
        "type": "stat"
      }
    ],
    "schemaVersion": 37,
    "style": "dark",
    "tags": [],
    "templating": {
      "list": []
    },
    "time": {
      "from": "now-24h",
      "to": "now"
    },
    "timepicker": {},
    "timezone": "",
    "title": "Pepr Dashboard",
    "uid": "j7BjgMpIk",
    "version": 17,
    "weekStart": ""
    }

Now, apply the grafana-dashboard.yaml file to the cluster:

kubectl apply -f grafana-dashboard.yaml

Step 3: Install Prometheus and Grafana using the kube-prometheus-stack Helm Chart

First, create a values.yaml file to add our endpoints to Prometheus and allow us to see our dashboard.

prometheus:
  enabled: true
  additionalServiceMonitors:
    - name: admission
      selector:
        matchLabels:
          pepr.dev/controller: admission
      namespaceSelector:
        matchNames:
          - pepr-system
      endpoints:
        - targetPort: 3000
          scheme: https
          tlsConfig:
            insecureSkipVerify: true
    - name: watcher
      selector:
        matchLabels:
          pepr.dev/controller: watcher
      namespaceSelector:
        matchNames:
          - pepr-system
      endpoints:
        - targetPort: 3000
          scheme: https
          tlsConfig:
            insecureSkipVerify: true
  additionalClusterRoleBindings:
    - name: scrape-binding
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: scrape-resources
      subjects:
      - kind: ServiceAccount
        name: prometheus-operator
        namespace: default
grafana:
  enabled: true
  adminUser: admin
  adminPassword: secret
  defaultDashboardsTimezone: browser
  extraVolumeMounts:
    - mountPath: /var/lib/grafana/dashboards
      name: pepr-dashboard
  extraVolumes:
    - name: pepr-dashboard
      configMap:
        name: pepr-dashboard
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - name: 'default'
        isDefault: true
        orgId: 1
        folder: ''
        type: file
        disableDeletion: false
        editable: true
        options:
          path: /var/lib/grafana/dashboards/default
  dashboardsConfigMaps:
    default: pepr-dashboard

Now, install the kube-prometheus-stack Helm Chart using the values.yaml file we created.

helm install -f values.yaml monitoring prometheus-community/kube-prometheus-stack

Step 4: Check on Services

kubectl get svc

You should see something similar to the following services:

$ kubectl get svc
NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kubernetes                                ClusterIP   10.43.0.1       <none>        443/TCP                      99m
monitoring-kube-prometheus-prometheus     ClusterIP   10.43.116.49    <none>        9090/TCP,8080/TCP            81s
monitoring-kube-state-metrics             ClusterIP   10.43.232.84    <none>        8080/TCP                     81s
monitoring-grafana                        ClusterIP   10.43.82.67     <none>        80/TCP                       81s
monitoring-kube-prometheus-operator       ClusterIP   10.43.197.97    <none>        443/TCP                      81s
monitoring-kube-prometheus-alertmanager   ClusterIP   10.43.40.24     <none>        9093/TCP,8080/TCP            81s
monitoring-prometheus-node-exporter       ClusterIP   10.43.152.179   <none>        9100/TCP                     81s
alertmanager-operated                     ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   81s
prometheus-operated                       ClusterIP   None            <none>        9090/TCP                     81s

Step 5: Port Forward Prometheus and Grafana Services

kubectl port-forward service/prometheus-operated 9090

kubectl port-forward service/monitoring-grafana 3000:80

Step 6: View Prometheus Metrics Targets

You should be able to see the Pepr targets in the Prometheus UI by visiting the following URL:

http://localhost:9090/targets

The targets should look something like this:

Admission Endpoints Watcher Endpoint

Step 7: Test the Prometheus Connection in Grafana

You should be able to test the Prometheus connection in the Grafana UI by visiting the following URL:

http://localhost:3000/connections/datasources

The login information for Grafana was set in the values.yaml file:

username: admin password: secret

By clicking on the Prometheus data source, you should be able to test the connection to Prometheus by clicking the “Test” button at the bottom of the screen.

NOTE: The Prometheus server URL should be something like:

http://monitoring-kube-prometh-prometheus.default:9090/

You should now be able to select the Pepr Dashboard from the Grafana UI in the “Dashboards” section.

Note: The dashboard may take a few minutes to populate with data.

Summary

This tutorial demonstrated how to use Prometheus and Grafana to display metrics from your Pepr instance. If you have questions about Pepr metrics or dashboards, please reach out to us on Slack or GitHub Issues

3 - Building a Kubernetes Operator with Pepr

Introduction

This tutorial guides you through building a Kubernetes Operator using Pepr. You’ll create a WebApp Operator that manages custom WebApp resources in your Kubernetes cluster.

If you get stuck at any point, you can reference the complete example code in the Pepr Excellent Examples repository.

What You’ll Build

The WebApp Operator will:

Deploy a custom WebApp resource definition (CRD)
Watch for WebApp instances and reconcile them with the actual cluster state
For each WebApp instance, manage:
- A Deployment with configurable replicas
- A Service to expose the application
- A ConfigMap containing configurable HTML with language and theme options

All resources will include ownerReferences, triggering cascading deletion when a WebApp is removed. The operator will also automatically restore any managed resources that are deleted externally.

Prerequisites

A Kubernetes cluster (local or remote)
Access to the curl command
Basic understanding of Kubernetes concepts
Familiarity with TypeScript
Node.js ≥ 18.0

Create a new Pepr Module

[🟢⚪⚪⚪⚪⚪] Step 1 of 6

First, create a new Pepr module for your operator:

npx pepr init \
  --name operator \
  --uuid my-operator-uuid \
  --description "Kubernetes Controller for WebApp Resources" \
  --errorBehavior reject \
  --confirm &&
cd operator # set working directory as the new pepr module

To track your progress in this tutorial, let’s treat it as a git repository:

git init && git add --all && git commit -m "npx pepr init"

Create CRD

[🟢🟢⚪⚪⚪⚪] Step 2 of 6

The WebApp Custom Resource Definition (CRD) specifies the structure and validation for your custom resource. Create the necessary directory structure:

mkdir -p capabilities/crd/generated capabilities/crd/source

Generate a class based on the WebApp CRD using kubernetes-fluent-client. This allows us to react to the CRD fields in a type-safe manner. Create a CRD named crd.yaml for the WebApp that includes:

Theme selection (dark/light)
Language selection (en/es)
Configurable replica count
Status tracking

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/source/crd.yaml \
  -o capabilities/crd/source/crd.yaml

Examine the contents of capabilities/crd/source/crd.yaml. Note that the status should be listed under subresources to make it writable. We provide descriptions for each property to clarify their purpose. Enums are used to restrict the values that can be assigned to a property.

Create an interface for the CRD spec with the following command:

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/generated/webapp-v1alpha1.ts \
  -o capabilities/crd/generated/webapp-v1alpha1.ts

Examine the contents of capabilities/crd/generated/webapp-v1alpha1.ts.

Create a TypeScript file that contains the WebApp CRD named webapp.crd.ts. This will enable the controller to automatically create the CRD on startup. Use the command:

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/source/webapp.crd.ts \
  -o capabilities/crd/source/webapp.crd.ts

Take a moment to commit your changes for CRD creation:

git add capabilities/crd/ && git commit -m "Create WebApp CRD"

The webapp.crd.ts file defines the structure of our CRD, including the validation rules and schema that Kubernetes will use when WebApp resources are created or modified.

Create a file that will automatically register the CRD on startup named capabilities/crd/register.ts:

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/register.ts \
  -o capabilities/crd/register.ts

The register.ts file contains logic to ensure our CRD is created in the cluster when the operator starts up, avoiding the need for manual CRD installation.

Create a file to validate that WebApp instances are in valid namespaces and have a maximum of 7 replicas. Create a validator.ts file with the command:

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/validator.ts \
  -o capabilities/crd/validator.ts

The validator.ts file implements validation logic for our WebApp resources, ensuring they meet our requirements before they’re accepted by the cluster.

In this section, we’ve generated the CRD class for WebApp, created a function to automatically register the CRD, and added a validator to ensure WebApp instances are in valid namespaces and don’t exceed 7 replicas.

Commit your changes for CRD registration & validation:

git add capabilities/crd/ && git commit -m "Create CRD handling logic"

Create Helpers

[🟢🟢🟢⚪⚪⚪] Step 3 of 6

Now, let’s create helper functions that will generate the Kubernetes resources managed by our operator. These helpers will simplify the creation of Deployments, Services, and ConfigMaps for each WebApp instance.

Create a controller folder in the capabilities folder and create a generators.ts file. This file will contain functions that generate Kubernetes objects for the operator to deploy (with the ownerReferences automatically included). Since these resources are owned by the WebApp resource, they will be deleted when the WebApp resource is deleted.

mkdir -p capabilities/controller

Create generators.ts with the following command:

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/controller/generators.ts \
  -o capabilities/controller/generators.ts

The generators.ts file contains functions to create all the necessary Kubernetes resources for our WebApp, including properly configured Deployments, Services, and ConfigMaps with appropriate labels and selectors.

Our goal is to simplify WebApp deployment. Instead of requiring users to manage multiple Kubernetes objects, track versions, and handle revisions manually, they can focus solely on the WebApp instance. The controller reconciles WebApp instances against the actual cluster state to achieve the desired configuration.

The controller deploys a ConfigMap based on the language and theme specified in the WebApp resource and sets the number of replicas according to the WebApp specification.

Commit your changes for deployment, service, and configmap generation:

git add capabilities/controller/ && git commit -m "Add generators for WebApp deployments, services, and configmaps"

Create Reconciler

[🟢🟢🟢🟢⚪⚪] Step 4 of 6

Now, create the function that reacts to changes in WebApp instances. This function will be called and placed into a queue, guaranteeing ordered and synchronous processing of events, even when the system is under heavy load.

In the base of the capabilities folder, create a reconciler.ts file and add the following:

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/reconciler.ts \
  -o capabilities/reconciler.ts

The reconciler.ts file contains the core logic of our operator, handling the creation, updating, and deletion of WebApp resources and ensuring the cluster state matches the desired state.

Finally, create the index.ts file in the capabilities folder and add the following:

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/index.ts \
  -o capabilities/index.ts

The index.ts file contains the WebAppController capability and the functions that are used to watch for changes to the WebApp resource and corresponding Kubernetes resources.

When a WebApp is created or updated, validate it, store the name of the instance and enqueue it for processing.
If an “owned” resource (ConfigMap, Service, or Deployment) is deleted, redeploy it.
Always redeploy the WebApp CRD if it was deleted as the controller depends on it

In this section we created a reconciler.ts file that contains the function responsible for reconciling the state of WebApp instances with the cluster and updating their status. The index.ts file contains the WebAppController capability and functions that watch for changes to WebApp resources and their corresponding Kubernetes objects. The Reconcile action processes callbacks in a queue, guaranteeing ordered and synchronous processing of events.

Commit your changes with:

git add capabilities/ && git commit -m "Create reconciler for webapps"

Add capability to Pepr Module

Ensure that the PeprModule in pepr.ts uses WebAppController. The implementation should look something like this:

new PeprModule(cfg, [WebAppController]);

Using sed, replace the contents of the file to use our new WebAppController:

# Use the WebAppController Module
sed -i '' -e '/new PeprModule(cfg, \[/,/\]);/c\
new PeprModule(cfg, [WebAppController]);' ./pepr.ts

# Update Imports
sed -i '' 's|import { HelloPepr } from "./capabilities/hello-pepr";|import { WebAppController } from "./capabilities";|' ./pepr.ts

Commit your changes now that the WebAppController is part of the Pepr Module:

git add pepr.ts && git commit -m "Register WebAppController with pepr module"

Build and Deploy Your Operator

[🟢🟢🟢🟢🟢⚪] Step 5 of 6

Preparing Your Environment

Create an ephemeral cluster with k3d.

k3d cluster delete pepr-dev &&
k3d cluster create pepr-dev --k3s-arg '--debug@server:0' --wait &&
kubectl rollout status deployment -n kube-system

What is an ephemeral cluster?
An ephemeral cluster is a temporary Kubernetes cluster that exists only for testing purposes. Tools like Kind (Kubernetes in Docker) and k3d let you quickly create and destroy clusters without affecting your production environments.

Update and Prepare Pepr

Make sure Pepr is updated to the latest version:

npx pepr update --skip-template-update

⚠️ Important Note: Be cautious when updating Pepr in an existing project as it could potentially override custom configurations. The --skip-template-update flag helps prevent this.

Building the Operator

Build the Pepr module by running:

npx pepr format && 
npx pepr build

Commit your changes after the build completes:

git add capabilities/ package*.json && git commit -m "Build pepr module"

The build process explained

The pepr build command performs three critical steps:

Compile TypeScript: Converts your TypeScript code to JavaScript using the settings in tsconfig.json
Bundle the Operator: Packages everything into a deployable format using esbuild
Generate Kubernetes Manifests: Creates all necessary YAML files in the dist directory, including:
- Custom Resource Definitions (CRDs)
- The controller deployment
- ServiceAccounts and RBAC permissions
- Any other resources needed for your operator

This process creates a self-contained deployment unit that includes everything needed to run your operator in a Kubernetes cluster.

┌─────────────────────┐          ┌──────────────────┐          ┌───────────────────┐
│                     │          │                  │          │                   │
│  Your Pepr Code     │────────▶ │  pepr build      │────────▶ │ dist/             │
│  (TypeScript)       │          │  (Build Process) │          │(Deployment Files) │
│                     │          │                  │          │                   │
└─────────────────────┘          └──────────────────┘          └───────────────────┘
                                                                        │
                                                                        │
                                                                        ▼
                                                               ┌───────────────────┐
                                                               │                   │
                                                               │  Kubernetes       │
                                                               │  Cluster          │
                                                               │                   │
                                                               └───────────────────┘

Deploy to Kubernetes

To deploy your operator to a Kubernetes cluster:

kubectl apply -f dist/pepr-module-my-operator-uuid.yaml &&
kubectl wait --for=condition=Ready pods -l app -n pepr-system --timeout=120s

What’s happening during deployment?

The first command applies all the Kubernetes resources defined in the YAML file, including:
- The WebApp CRD (Custom Resource Definition)
- A Deployment that runs your operator code
- The necessary RBAC permissions for your operator to function
The second command waits for the operator pod to be ready before proceeding. This ensures your operator is running before you attempt to create WebApp resources.

Troubleshooting Deployment Issues

Troubleshooting operator startup problems

If your operator doesn’t start properly, check these common issues:

Check pod logs:

kubectl logs -n pepr-system -l app --tail=100

Verify permissions:
```
kubectl describe deployment -n pepr-system
```
Look for permission-related errors in the events section.

Verify the deployment was successful by checking if the CRD has been properly registered:

kubectl get crd | grep webapp

You should see webapps.pepr.io in the output, which confirms your Custom Resource Definition was created successfully.

Understanding the WebApp Resource

You can use kubectl explain to see the structure of your custom resource. It may take a moment for the cluster to recognize this resource before the following command will work:

kubectl explain wa.spec

Expected Output:

GROUP:      pepr.io
KIND:       WebApp
VERSION:    v1alpha1

FIELD: spec <Object>

DESCRIPTION:
    <empty>
FIELDS:
  language      <string> -required-
    Language defines the language of the web application, either English (en) or
    Spanish (es).

  replicas      <integer> -required-
    Replicas is the number of desired replicas.

  theme <string> -required-
    Theme defines the theme of the web application, either dark or light.

💡 Note: wa is the short form of webapp that kubectl recognizes. This resource structure directly matches the TypeScript interface we defined earlier in our code.

Test Your Operator

[🟢🟢🟢🟢🟢🟢] Step 6 of 6

Understanding reconciliation

Reconciliation is the core concept behind Kubernetes operators. It’s the process of:

Observing the current state of resources in the cluster
Comparing it to the desired state (defined in your custom resource)
Taking actions to align the actual state with the desired state

This continuous loop ensures your application maintains its expected configuration even when disruptions occur.

┌───────────────────┐
│                   │
│  Custom Resource  │◄────────────┐
│  (WebApp)         │             │
│                   │             │
└───────┬───────────┘             │
        │                         │
        │ Observe                 │
        ▼                         │
┌───────────────────┐    ┌────────────────┐
│                   │    │                │
│  Pepr Operator    │───►│  Reconcile     │
│  Controller       │    │  (Take Action) │
│                   │    │                │
└───────┬───────────┘    └────────────────┘
        │                         ▲
        │ Create/Update           │
        ▼                         │
┌───────────────────┐             │
│  Owned Resources  │─────────────┘
│  • ConfigMap      │ If deleted or
│  • Service        │ changed, trigger
│  • Deployment     │ reconciliation
└───────────────────┘

Creating a WebApp Instance

Let’s create an instance of our custom WebApp resource in English with a light theme and 1 replicas:

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/webapp-light-en.yaml \
  -o webapp-light-en.yaml

Examine the contents of webapp-light-en.yaml. It defines a WebApp with English language, light theme, and 1 replica.

Next, apply it to the cluster:

kubectl create namespace webapps &&
kubectl apply -f webapp-light-en.yaml

How resource creation works

Kubernetes API server receives the WebApp resource
Our operator’s controller (in index.ts) detects the new resource via its reconcile function
The controller validates the WebApp using our validator
The reconcile function creates three “owned” resources:
- A ConfigMap with HTML content based on the theme and language
- A Service to expose the web application
- A Deployment to run the web server pods with the specified number of replicas
The status is updated to track progress

All this logic is in the code we wrote earlier in the tutorial.

Verifying Resource Creation

Now, verify that the WebApp and its owned resources were created properly:

kubectl get cm,svc,deploy,webapp -n webapps

Expected Output:

NAME                                    DATA   AGE
configmap/kube-root-ca.crt              1      6s
configmap/web-content-webapp-light-en   1      5s

NAME                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/webapp-light-en   ClusterIP   10.43.85.1   <none>        80/TCP    5s

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/webapp-light-en   1/1     1            1           5s

💡 Tip: Our operator created three resources based on our single WebApp definition - this is the power of operators in action!

Checking WebApp Status

The status field is how our operator communicates the current state of the WebApp:

kubectl get wa webapp-light-en -n webapps -ojsonpath="{.status}" | jq

Expected Output:

{
  "observedGeneration": 1,
  "phase": "Ready"
}

Understanding status fields

observedGeneration: A counter that increments each time the resource spec is changed
phase: The current lifecycle state of the WebApp (“Pending” during creation, “Ready” when all components are operational)

This status information comes from our reconciler code, which updates these fields during each reconciliation cycle.

You can also see events related to your WebApp that provide a timeline of actions taken by the operator:

kubectl describe wa webapp-light-en -n webapps

Expected Output:

Name:         webapp-light-en
Namespace:    webapps
API Version:  pepr.io/v1alpha1
Kind:         WebApp
Metadata: ...
Spec:
  Language:  en
  Replicas:  1
  Theme:     light
Status:
  Observed Generation:  1
  Phase:                Ready
Events:
  Type    Reason                    Age   From             Message
  ----    ------                    ----  ----             -------
  Normal  InstanceCreatedOrUpdated  36s   webapp-light-en  Pending
  Normal  InstanceCreatedOrUpdated  36s   webapp-light-en  Ready

Viewing Your WebApp

To access your WebApp in a browser, use port-forwarding to connect to the service. The following command runs the portforward in the background. Be sure to make note of the Port-forward PID for later when we tear down the test environment.

kubectl port-forward svc/webapp-light-en -n webapps 3000:80 &
PID=$!
echo "Port-forward PID: $PID"

About port-forwarding

Port-forwarding creates a secure tunnel from your local machine to a pod or service in your Kubernetes cluster. In this case, we’re forwarding your local port 3000 to port 80 of the WebApp service, allowing you to access the application at http://localhost:3000 in your browser.

Now open http://localhost:3000 in your browser or run curl http://localhost:3000 to see the response in a terminal. The browser should display a light theme web application:

WebApp Light Theme

Testing the Reconciliation Loop

A key feature of operators is their ability to automatically repair resources when they’re deleted or changed. Let’s test this by deleting the ConfigMap:

kubectl delete cm -n webapps --all &&
sleep 10 && 
kubectl get cm -n webapps

Expected output:

configmap "kube-root-ca.crt" deleted
configmap "web-content-webapp-light-en" deleted
NAME                          DATA   AGE
kube-root-ca.crt              1      0s
web-content-webapp-light-en   1      0s

Now that we’ve successfully deployed a WebApp, commit your changes:

git add webapp-light-en.yaml && git commit -m "Add WebApp resource for light mode in english"

Behind the scenes of reconciliation

When you deleted the ConfigMap, Kubernetes sent a DELETE event
Our operator (in index.ts) was watching for these events via the onDeleteOwnedResource handler
This triggered the reconciliation loop, which detected that the ConfigMap was missing
The reconciler recreated the ConfigMap based on the WebApp definition
This all happened automatically without manual intervention - the core benefit of using an operator!

🛠️ Try it yourself: Try deleting the Service or Deployment. What happens? The operator should recreate those too!

Updating the WebApp

Now let’s test changing the WebApp’s specification. Copy down the next WebApp resource with the following command:

curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/webapp-dark-es.yaml \
  -o webapp-dark-es.yaml

Compare the contents of webapp-light-en.yaml and webapp-dark-es.yaml with the command:

diff --side-by-side \
  webapp-light-en.yaml \
  webapp-dark-es.yaml

kubectl apply -f webapp-dark-es.yaml

💡 Note: We’ve changed the theme from light to dark and the language from English (en) to Spanish (es).

Your port-forward should still be active, so you can refresh your browser to see the changes. If your porf-forward is no longer active for some reason, create a new one:

# Only needed if previous port-forward closed
kubectl port-forward svc/webapp-light-en -n webapps 3000:80 &
PID=$!
echo "Port-forward PID: $PID"

Now open http://localhost:3000 in your browser or run curl http://localhost:3000 to see the response in a terminal. The browser should display a dark theme web application:

WebApp Dark Theme

Now that we’ve successfully updated a WebApp, commit your changes:

git add webapp-dark-es.yaml && git commit -m "Update WebApp resource for dark mode in spanish"

How updating works

When you apply the changed WebApp, Kubernetes sends an UPDATE event
Our operator’s controller (in index.ts) detects this via the onUpdate handler
The updated spec is validated and then queued for reconciliation
The reconciler compares the current resources with what’s needed for the new spec
It updates the ConfigMap with the new theme and language content
The Deployment automatically detects the ConfigMap change and restarts the pod with the new content

Cleanup

When you’re done testing, you can delete your WebApp and verify that all owned resources are removed:

kill $PID && # Close port-forward
kubectl delete wa -n webapps --all && 
sleep 5 &&
kubectl get cm,deploy,svc -n webapps

You can also delete the entire test cluster when you’re finished:

k3d cluster delete pepr-dev

Congratulations!

You’ve successfully built a Kubernetes operator using Pepr. Through this tutorial, you:

Created a custom resource definition (CRD) for WebApps
Implemented a controller with reconciliation logic
Added validation for your custom resources
Deployed your operator to a Kubernetes cluster
Verified that your operator correctly manages the lifecycle of WebApp resources

This pattern is powerful for creating self-managing applications in Kubernetes. Your operator now handles the complex task of maintaining your application’s state according to your specifications, reducing the need for manual intervention.

What You’ve Learned

By completing this tutorial, you’ve gained experience with several important concepts:

Custom Resource Definitions (CRDs): You defined a structured, validated schema for your WebApp resources
Reconciliation: You implemented the core operator pattern that maintains desired state
Owner References: You used Kubernetes ownership to manage resource lifecycles
Status Reporting: Your operator provides feedback about resource state through status fields
Watch Patterns: Your operator reacts to changes in both custom and standard Kubernetes resources

These concepts form the foundation of the operator pattern and can be applied to manage any application or service on Kubernetes.

Next Steps

Now that you understand the basics of building an operator with Pepr, you might want to:

Add more sophisticated validation logic
Implement status conditions that provide detailed health information
Add support for upgrading between versions of your application
Explore more complex reconciliation patterns for multi-component applications
Add metrics and monitoring to your operator

For more information, check out: