Extending the Kubernetes API: A comprehensive guide to Custom Resource Definitions (CRDs)

Kubernetes ships with a fixed vocabulary of objects: Pods, Services, Deployments, and a few dozen more. That vocabulary is excellent until the day you need to model something it does not know about, a managed database, a certificate, a feature flag, or some domain object specific to your platform. You could bolt that concept on with config maps and naming conventions, but then you lose the things that make Kubernetes objects pleasant to work with: a typed schema, server-side validation, kubectl get, RBAC, and a stored history.

The better answer is to teach the API server a new noun. Custom Resource Definitions (CRDs) let you define new types of resources that your cluster manages just like the built-in ones. Once you have defined a CRD, you can create, get, list, watch, update, patch, and delete instances of your custom resource exactly as you would a native one.

In this guide we'll define a real, validated custom resource and drive it entirely with kubectl, which is all most people ever need. Then, for the readers who want their resources to actually do something, we'll cover controllers using the modern approach and link out to the tooling that scaffolds them for you.

The Kubernetes resource life cycle

Before any YAML, it is worth understanding what goes into creating a custom resource and how it ties into the wider resource life cycle.

1. API extension is the idea: teach the Kubernetes API server about a new noun.
2. CustomResourceDefinition (CRD) is the schema: it registers that noun and says what fields are valid. This is the part most people need.
3. Custom Resource (CR) is an instance: one actual object of your new kind, created with kubectl apply.
4. Controller is the behaviour: a program that watches your custom resources and makes the cluster match what they ask for.
5. Operator is the package: a controller plus its CRDs, shipped together so others can install it in one step.

Prerequisites

You will need:

• A Civo account and a running Kubernetes cluster. A single-node K3s cluster is plenty, since everything here is control-plane work with no real workloads.
• kubectl installed and pointed at your cluster.
• Cluster admin rights (registering a CRD is a cluster-scoped operation).
• A basic understanding of Kubernetes objects like Pods and Deployments.

To spin up a cluster on Civo:

1civo k3s create --create-firewall --nodes 1 -m --save --switch --wait crd-demo
2kubectl get nodes

Define the Fruit CRD

We'll use a single running example throughout: a Fruit resource with a color and a sweetness score. Apply the CRD with a heredoc so you can copy and paste it directly:

1kubectl apply -f - <<'EOF'
2apiVersion: apiextensions.k8s.io/v1
3kind: CustomResourceDefinition
4metadata:
5  name: fruits.examples.com
6spec:
7  group: examples.com
8  scope: Namespaced
9  names:
10    plural: fruits
11    singular: fruit
12    kind: Fruit
13    shortNames:
14      - fr
15  versions:
16    - name: v1
17      served: true
18      storage: true
19      schema:
20        openAPIV3Schema:
21          type: object
22          properties:
23            spec:
24              type: object
25              required: ["color"]
26              properties:
27                color:
28                  type: string
29                sweetness:
30                  type: integer
31                  minimum: 0
32                  maximum: 10
33                  default: 5
34      additionalPrinterColumns:
35        - name: Color
36          type: string
37          jsonPath: .spec.color
38        - name: Sweetness
39          type: integer
40          jsonPath: .spec.sweetness
41EOF

A few things are doing important work here:

• type: object at the root is what makes the schema structural: Under apiextensions.k8s.io/v1, a structural schema.openAPIV3Schema is mandatory for every served version. A CRD without one is rejected at apply time.
• required: ["color"] sits on the spec object: OpenAPI required is a list of property names scoped to the object that declares it, not a dotted path. Writing required: ["spec.color"] at the top would silently enforce nothing.
• default: 5 on sweetness means a resource that omits the field gets 5 written back into the stored object.
• additionalPrinterColumns make kubectl get fruit show Color and Sweetness instead of just NAME and AGE.

Confirm the CRD registered and reached the Established condition:

1kubectl get crd fruits.examples.com
2kubectl wait --for=condition=Established crd/fruits.examples.com --timeout=30s

The wait should return condition met.

Create and read custom resources

Now create an instance. This is the happy path, our running apple:

1kubectl apply -f - <<'EOF'
2apiVersion: examples.com/v1
3kind: Fruit
4metadata:
5  name: apple
6spec:
7  color: red
8  sweetness: 10
9EOF

Create a second one that omits sweetness so we can watch the default apply:

1kubectl apply -f - <<'EOF'
2apiVersion: examples.com/v1
3kind: Fruit
4metadata:
5  name: pear
6spec:
7  color: green
8EOF

Now read them back:

1kubectl get fruit

The printer columns render, and pear comes back with sweetness: 5 even though we never set it:

1NAME    COLOR   SWEETNESS
2apple   red     10
3pear    green   5

kubectl get fruit: apple at 10, pear defaulted to 5, with the Color and Sweetness printer columns.

Inspect a single object in full, and describe it:

1kubectl get fruit apple -o yaml
2kubectl describe fruit apple

Everything you can do with a native object works here: kubectl get, -o yaml, describe, edit, patch, delete, label selectors, and the short name fr (kubectl get fr).

Watch validation

The API server enforces it on the way in. This sweetness: 99 violates maximum: 10:

1kubectl apply -f - <<'EOF'
2apiVersion: examples.com/v1
3kind: Fruit
4metadata:
5  name: toosweet
6spec:
7  color: green
8  sweetness: 99
9EOF

The apply is rejected before anything is stored:

1The Fruit "toosweet" is invalid: spec.sweetness: Invalid value: 99: spec.sweetness in body should be less than or equal to 10

Omitting the required color fails the same way:

1kubectl apply -f - <<'EOF'
2apiVersion: examples.com/v1
3kind: Fruit
4metadata:
5  name: nocolor
6spec:
7  sweetness: 3
8EOF
9The Fruit "nocolor" is invalid: spec.color: Required value

Schema validation rejecting sweetness 99 (over the maximum) and a resource missing the required color.

Multiple versions and conversion

Over time, CRDs evolve. CRDs support serving more than one version of the same kind at once. The CRD below adds a v2 alongside v1, with v1 remaining the storage version:

1kubectl apply -f - <<'EOF'
2apiVersion: apiextensions.k8s.io/v1
3kind: CustomResourceDefinition
4metadata:
5  name: fruits.examples.com
6spec:
7  group: examples.com
8  scope: Namespaced
9  names:
10    plural: fruits
11    singular: fruit
12    kind: Fruit
13    shortNames:
14      - fr
15  conversion:
16    strategy: None
17  versions:
18    - name: v1
19      served: true
20      storage: true
21      schema: &fruitSchema
22        openAPIV3Schema:
23          type: object
24          properties:
25            spec:
26              type: object
27              required: ["color"]
28              properties:
29                color:
30                  type: string
31                sweetness:
32                  type: integer
33                  minimum: 0
34                  maximum: 10
35                  default: 5
36    - name: v2
37      served: true
38      storage: false
39      schema: *fruitSchema
40EOF

Three rules to remember here:

• Exactly one version is the storage version (storage: true): That is the form the object is actually persisted as. Here it is v1.
• Each version needs its own schema.openAPIV3Schema: The YAML anchors (&fruitSchema and *fruitSchema) just let us reuse one schema block without retyping it. If you prefer, inline the second schema.
• strategy: None is the default and only changes the apiVersion stamped on the object: It is safe only while every served version shares an identical schema. The moment v2 adds or renames a field, None would hand back malformed data, so you must switch to strategy: Webhook and run a conversion webhook.

You can prove conversion is working by reading the same stored apple through both versions:

1kubectl get fruit apple -o yaml | grep apiVersion
2kubectl get fruits.v2.examples.com apple -o yaml | grep apiVersion

The object is stored as v1 but served as v2 on request, body unchanged:

1examples.com/v1
2examples.com/v2

Locking it down with RBAC

Custom resources participate in RBAC like any other API object. To grant full control over fruits, target the examples.com group and the fruits resource:

1kubectl apply -f - <<'EOF'
2apiVersion: rbac.authorization.k8s.io/v1
3kind: ClusterRole
4metadata:
5  name: fruit-admin
6rules:
7  - apiGroups: ["examples.com"]
8    resources: ["fruits"]
9    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
10EOF

Bind it to a user or service account with a ClusterRoleBinding (or a namespaced RoleBinding) the same way you would for any built-in resource.

Clean up

1kubectl delete fruit --all
2kubectl delete crd fruits.examples.com

If you just wanted a typed, validated, kubectl-managed object, you are done: you now have a custom kind with schema validation, defaulting, printer columns, multiple versions, and RBAC, all without writing a line of code. Read on only if you want your resources to trigger real behaviour.

Controllers and operators

A CRD by itself doesn't do much. It stores data and validates it, but nothing happens when you create a Fruit. To make creating a resource cause something, for example provisioning a Deployment, calling an external API, or updating status, you add a controller.

A controller watches your custom resources and continuously drives the cluster toward the state they describe. The key idea is that controllers are level-triggered, not edge-triggered. The controller does not react to "a Fruit was created" or "a Fruit was edited" as distinct events. Instead, whenever anything changes, it looks at the current desired state of the object and makes the world match it. That makes the loop naturally robust: a missed event does not matter, because the next reconcile sees the true current state anyway.

The modern way: controller-runtime

You almost never write the informer and work-queue plumbing by hand anymore. You scaffold the project with Kubebuilder or the Operator SDK, and both generate the manager, the CRD, the RBAC, and all the wiring. Your job is to fill in a single method:

1// Reconcile is called whenever a Fruit changes. controller-runtime handles the
2// informers, caching, work queue, and retries for you. You write the desired-state
3// logic and return.
4func (r *FruitReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
5    var fruit examplesv1.Fruit
6    if err := r.Get(ctx, req.NamespacedName, &fruit); err != nil {
7        // The object was deleted. Nothing to do, stop requeuing.
8        return ctrl.Result{}, client.IgnoreNotFound(err)
9    }
10
11
12    // Desired state lives in fruit.Spec. Make the world match it here, for example
13    // create or update a Deployment, then report status back on fruit.Status.
14
15
16    // Returning an empty Result tells controller-runtime we are reconciled.
17    // Return ctrl.Result{RequeueAfter: time.Minute} to be called again later.
18    return ctrl.Result{}, nil
19}

That is the entire shape of a modern controller. The framework calls Reconcile whenever a Fruit changes, hands you the request, and you make the cluster match the spec.

Error handling and retries, the Kubernetes-native way

Because the loop is level-triggered, retries are built in. You do not reach for a general-purpose retry library. You have two levers:

• Return the error: controller-runtime requeues the request with exponential backoff automatically. Use this for transient failures (an API call timed out, a dependency is not ready yet).
• Return ctrl.Result{RequeueAfter: d}: This schedules another reconcile after a fixed delay, useful when you are polling something that takes time to settle.

Both keep all retry logic inside the controller loop where the framework can manage it.

Packaging as an operator

When you bundle a controller together with the CRDs it manages and ship them as one installable unit, you have an operator. Both Kubebuilder and the Operator SDK produce the manifests and container image to do this, so installing your custom behaviour on a new cluster becomes a single kubectl apply or Helm install.

A controller from scratch with client-go

You will not normally write this much code, controller-runtime does it for you, but it is worth seeing once to understand what the framework is doing on your behalf. The listing below builds the same machinery by hand with client-go: a shared informer to watch Fruit objects, a rate-limited work queue, event handlers that enqueue keys, and a worker loop that drains the queue and reconciles each item.

1package main
2
3
4import (
5    "fmt"
6    "time"
7
8
9    "k8s.io/apimachinery/pkg/util/runtime"
10    "k8s.io/apimachinery/pkg/util/wait"
11    "k8s.io/client-go/tools/cache"
12    "k8s.io/client-go/util/workqueue"
13    "k8s.io/klog/v2"
14)
15
16
17// FruitController watches Fruit objects and reconciles them. controller-runtime
18// generates all of this for you; this is the underlying machinery.
19type FruitController struct {
20    informer cache.SharedIndexInformer
21    queue    workqueue.RateLimitingInterface
22}
23
24
25func NewFruitController(informer cache.SharedIndexInformer) *FruitController {
26    queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter())
27
28
29    informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
30        AddFunc: func(obj interface{}) {
31            if key, err := cache.MetaNamespaceKeyFunc(obj); err == nil {
32                queue.Add(key)
33            }
34        },
35        UpdateFunc: func(old, new interface{}) {
36            if key, err := cache.MetaNamespaceKeyFunc(new); err == nil {
37                queue.Add(key)
38            }
39        },
40        DeleteFunc: func(obj interface{}) {
41            if key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(obj); err == nil {
42                queue.Add(key)
43            }
44        },
45    })
46
47
48    return &FruitController{informer: informer, queue: queue}
49}
50
51
52func (c *FruitController) Run(workers int, stopCh <-chan struct{}) {
53    defer runtime.HandleCrash()
54    defer c.queue.ShutDown()
55
56
57    go c.informer.Run(stopCh)
58    if !cache.WaitForCacheSync(stopCh, c.informer.HasSynced) {
59        runtime.HandleError(fmt.Errorf("timed out waiting for caches to sync"))
60        return
61    }
62
63
64    for i := 0; i < workers; i++ {
65        go wait.Until(c.runWorker, time.Second, stopCh)
66    }
67    <-stopCh
68}
69
70
71func (c *FruitController) runWorker() {
72    for c.processNextItem() {
73    }
74}
75
76
77func (c *FruitController) processNextItem() bool {
78    key, quit := c.queue.Get()
79    if quit {
80        return false
81    }
82    defer c.queue.Done(key)
83
84
85    if err := c.reconcile(key.(string)); err != nil {
86        // Requeue with backoff on error, the same behaviour controller-runtime
87        // gives you for free when you return an error from Reconcile.
88        c.queue.AddRateLimited(key)
89    } else {
90        c.queue.Forget(key)
91    }
92    return true
93}
94
95
96func (c *FruitController) reconcile(key string) error {
97    obj, exists, err := c.informer.GetIndexer().GetByKey(key)
98    if err != nil {
99        return err
100    }
101    if !exists {
102        klog.Infof("Fruit %s deleted", key)
103        return nil
104    }
105
106
107    // Desired state lives on the object's spec. Make the world match it here.
108    klog.Infof("reconciling Fruit %s: %v", key, obj)
109    return nil
110}

Compare this to the eight-line Reconcile method above and the value of controller-runtime is obvious: every concern except your actual business logic, the informer, the cache sync, the queue, the backoff, the worker pool, is boilerplate the framework owns. Write it once by hand to understand it, then let Kubebuilder generate it forever after.

Summary

Kubernetes extensibility provides endless opportunities for building on top of it. We defined a Fruit CRD with a real structural schema, validation, defaults, printer columns, multiple served versions with conversion, and RBAC, all driven by kubectl. That covers what most teams need.

For the cases that need behaviour, we saw the modern controller-runtime Reconcile loop and the scaffolding tools that generate everything around it, with the from-scratch client-go machinery in the appendix for the curious.

Additional resources

If you want to know more about Custom Resource Definitions and extending the Kubernetes API, take a look at these resources:

• Exposing your application to HTTPS with Kubernetes, Traefik and cert-manager
• What is container orchestration and Kubernetes?
• Kubernetes Monitoring: Best Practices, Alerting, and Solutions