Introduction

Kubernetes configuration files and object definitions are most commonly written in YAML files. As these files define how things run on our clusters, improving YAML file security and flow is important. Among the most crucial things, our files need to be audited and validated before being applied to our Kubernetes clusters. Misconfigurations such as escalated container privileges and missing keys have to be avoided at all costs, as these can open access to unauthorized users.

Luckily, there are many Kubernetes security tools that have been emerging such as Kubescape and ValidKube. In order to take advantage of these tools, they should be implemented in your Kubernetes workflow - especially in the case of ValidKube, as will be described.

Kubescape

Kubescape is a Kubernetes command line tool that audits and scans YAML files and clusters using the NSA-CISA and MITRE ATT&CK frameworks. Kubescape is an especially attractive tool, as it will give you recommendations on how you can fix detected vulnerabilities.

ValidKube

On the other side, we have ValidKube from Komodor which is a free tool used to validate and enforce Kubernetes best practices on your manifests and configuration files, folding in multiple open-source tools.

ValidKube tool is used to:

  • Validate configuration files
  • Clean manifest files
  • Scan and secure your YAML files by ensuring best practice

You don't need special skills or cram commands just to use ValidKube, as it is an online web tool. All you need to do is paste your YAML contents and choose if you want to clean, audit, secure, or validate the YAML contents.

ValidKube interface showing the input box on the left and the results window on the right

In this tutorial, you will learn how to clean and secure your manifests using ValidKube and Kubescape.

This tutorial will cover the following points

  1. Scanning and securing your manifests using Validkube
  2. How to audit and validate your YAML files using Kubescape, and scanning clusters against security frameworks
  3. Conclusion

Prerequisites

You need a running Kubernetes cluster, as well as the KUBECONFIG for it.

Scanning and securing your manifests using Validkube

ValidKube is great at catching common Kubernetes manifest misconfigurations, such as missing keys and null values. You can validate any type of Kubernetes resource or object. In this tutorial, I will show you how to validate and secure a deployment object. Here are the contents of a sample deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: boemo-app
  namespace: default
  labels:
    app: boemo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: boemo-app
  template:
    metadata:
      labels:
        app: boemo-app
    spec:
      containers:
      - name: server
        image: nginx:1.17
        volumeMounts:
          - name: boemo-app
            mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
          protocol: TCP
        resources:
          requests:
            cpu: 100m
            memory: "128M"
          limits:
            cpu: 100m
            memory: "256M"
        env:
        - name: LOG_LEVEL
          value: "DEBUG"
      volumes:
      - name: boemo-app
        configMap:
          name: boemo-app
          items:
          - key: body
            path: index.html

Copy the above YAML and paste it on ValidKube, click on the validate text button to validate the YAML file:

Pasting the demonstration deployment into ValidKube

When you click on the validate button ValidKube immediately outputs the results, which should appear on the right hand side showing the following:

- errors: []
  filename: /tmp/yaml/target_yaml.yaml
  kind: Deployment
  status: valid

But, If I enter an invalid value on the replica field, changing it from 1 to a string, ValidKube will give out the expected value type which is the integer type:

- errors:
  - 'spec.replicas: Invalid type. Expected: [integer,null], given: string'
  filename: /tmp/yaml/target_yaml.yaml
  kind: Deployment
  status: invalid

Validkube uses Kubeval to validate the deployment object. Kubeval is a command line that validates YAML files by comparing them to Kubernetes generated API schemas.

Cleaning a YAML file using ValidKube

Next, click on the clean text button on ValidKube to clean up the YAML file. Under the hood, ValidKube uses Kube-neat to format, reorganize, and remove clutter from the provided YAML file. Here are the results of the cleaning procedure using kube-neat.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: boemo-app
  name: boemo-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: boemo-app
  template:
    metadata:
      labels:
        app: boemo-app
    spec:
      containers:
      - env:
        - name: LOG_LEVEL
          value: DEBUG
        image: nginx:1.17
        name: server
        ports:
        - containerPort: 80
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 256M
          requests:
            cpu: 100m
            memory: 128M
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: boemo-app
      volumes:
      - configMap:
          items:
          - key: body
            path: index.html
          name: boemo-app
        name: boemo-app

ValidKube has cleaned the deployed YAML file by formatting the contents and added the volumeMount which was missing. This process of linting, or removing clutter and formatting files the right way makes it easy for code reviewers because they won't be reviewing unnecessary code lines.

Carrying out security analysis using ValidKube

ValidKube uses Trivy from AquaSecurity to check security vulnerabilities. Trivy is a tool that scans images and clusters looking for security vulnerabilities. Here are the security scan results after clicking on the secure text button on our sample deployment file from earlier:

ArtifactName: /tmp/yaml
ArtifactType: filesystem
Metadata:
  ImageConfig:
    architecture: ""
    config: {}
    created: "0001-01-01T00:00:00Z"
    os: ""
    rootfs:
      diff_ids: null
      type: ""

The results will start by summing up the failure and specifying the detected misconfiguration together with its description:

Results:
- Class: config
  MisconfSummary:
    Exceptions: 0
    Failures: 6
    Successes: 22
  Misconfigurations:
  - Description: A program inside the container can elevate its own privileges and
      run as root, which might give the program control over the container and node.
    ID: KSV001
    IacMetadata: {}
    Layer: {}
    Message: Container 'server' of Deployment 'boemo-app' should set 'securityContext.allowPrivilegeEscalation'
      to false
    Namespace: appshield.kubernetes.KSV001
    PrimaryURL: https://avd.aquasec.com/appshield/ksv001
    Query: data.appshield.kubernetes.KSV001.deny
    References:
    - https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted
    - https://avd.aquasec.com/appshield/ksv001

Next, you will get a suggested solution to the detected vulnerabilities:

Resolution: Set 'set containers[].securityContext.allowPrivilegeEscalation' to
      'false'.
    Severity: MEDIUM
    Status: FAIL
    Title: Process can elevate its own privileges
    Type: Kubernetes Security Check
  - Description: The container should drop all default capabilities and add only those
      that are needed for its execution.
    ID: KSV003
    IacMetadata: {}
    Layer: {}
    Message: Container 'server' of Deployment 'boemo-app' should add 'ALL' to 'securityContext.capabilities.drop'
    Namespace: appshield.kubernetes.KSV003
    PrimaryURL: https://avd.aquasec.com/appshield/ksv003
    Query: data.appshield.kubernetes.KSV003.deny
    References:
    - https://kubesec.io/basics/containers-securitycontext-capabilities-drop-index-all/
    - https://avd.aquasec.com/appshield/ksv003
    Resolution: Add 'ALL' to containers[].securityContext.capabilities.drop.
    Severity: LOW
    Status: FAIL
    Title: Default capabilities not dropped
    Type: Kubernetes Security Check
  - Description: '''runAsNonRoot'' forces the running image to run as a non-root user
      to ensure least privileges.'
    ID: KSV012
    IacMetadata: {}
    Layer: {}
    Message: Container 'server' of Deployment 'boemo-app' should set 'securityContext.runAsNonRoot'
      to true
    Namespace: appshield.kubernetes.KSV012
    PrimaryURL: https://avd.aquasec.com/appshield/ksv012
    Query: data.appshield.kubernetes.KSV012.deny
    References:
    - https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted
    - https://avd.aquasec.com/appshield/ksv012
    Resolution: Set 'containers[].securityContext.runAsNonRoot' to true.
    Severity: MEDIUM
    Status: FAIL
    Title: Runs as root user
    Type: Kubernetes Security Check
  - Description: An immutable root file system prevents applications from writing
      to their local disk. This can limit intrusions, as attackers will not be able
      to tamper with the file system or write foreign executables to disk.
    ID: KSV014
    IacMetadata: {}
    Layer: {}
    Message: Container 'server' of Deployment 'boemo-app' should set 'securityContext.readOnlyRootFilesystem'
      to true
    Namespace: appshield.kubernetes.KSV014
    PrimaryURL: https://avd.aquasec.com/appshield/ksv014
    Query: data.appshield.kubernetes.KSV014.deny
    References:
    - https://kubesec.io/basics/containers-securitycontext-readonlyrootfilesystem-true/
    - https://avd.aquasec.com/appshield/ksv014
    Resolution: Change 'containers[].securityContext.readOnlyRootFilesystem' to 'true'.
    Severity: LOW
    Status: FAIL
    Title: Root file system is not read-only
    Type: Kubernetes Security Check
  - Description: Force the container to run with user ID > 10000 to avoid conflicts
      with the host's user table.
    ID: KSV020
    IacMetadata: {}
    Layer: {}
    Message: Container 'server' of Deployment 'boemo-app' should set 'securityContext.runAsUser'
      > 10000
    Namespace: appshield.kubernetes.KSV020
    PrimaryURL: https://avd.aquasec.com/appshield/ksv020
    Query: data.appshield.kubernetes.KSV020.deny
    References:
    - https://kubesec.io/basics/containers-securitycontext-runasuser/
    - https://avd.aquasec.com/appshield/ksv020

ValidKube will categorize the level of impact the vulnerability can make if exploited. The levels range is Low, Medium and High. Vulnerabilities that are classified as high should be solved immediately, but you should not forget about the lower-classified issues either. They will be marked as failures alongside your other results:

Severity: LOW
    Status: FAIL

Auditing YAML files with ValidKube

For the auditing procedure, ValidKube uses the Fairwinds Polaris tool which scans clusters looking for misconfigurations. Here are the results of the audit scan after clicking on the audit button:

AuditTime: "2022-06-03T08:28:12Z"
ClusterInfo:
  Controllers: 1
  Namespaces: 0
  Nodes: 0
  Pods: 0
  Version: unknown
DisplayName: /tmp/yaml/target_yaml.yaml
PolarisOutputVersion: "1.0"
Results:
- CreatedTime: "0001-01-01T00:00:00Z"
  Kind: Deployment
  Name: boemo-app
  Namespace: default
  PodResult:
    ContainerResults:
    - Name: server

ValidKube will categorize issues found in the audit into efficiency and security groups. If a vulnerability belongs to the efficiency category it means that it affects the cluster's performance. For example, the followin report notes efficiency issues around CPU limits and requests not being configured:

 Results:
        cpuLimitsMissing:
          Category: Efficiency
          Details: null
          ID: cpuLimitsMissing
          Message: CPU limits are set
          Severity: warning
          Success: true
        cpuRequestsMissing:
          Category: Efficiency
          Details: null
          ID: cpuRequestsMissing
          Message: CPU requests are set
          Severity: warning
          Success: true

Alongside the efficiency issues, you will get notified if your container has dangerous capabilities:

        dangerousCapabilities:
          Category: Security
          Details: null
          ID: dangerousCapabilities
          Message: Container does not have any dangerous capabilities
          Severity: danger
          Success: true

All fields that do not have values will be detected and displayed:

        hostPortSet:
          Category: Security
          Details: null
          ID: hostPortSet
          Message: Host port is not configured
          Severity: warning
          Success: true

How to audit and validate your YAML files using Kubescape

Kubescape introduction

Unlike ValidKube which is runs on the web, Kubescape is a handy command line tool that scans your cluster and detects vulnerabilities. After scanning your cluster it gives you a security analysis report of every object and resource in your cluster. This report is very detailed in a way that it gives you solutions and a documentation link of the detected security vulnerability. DevSecOps just got easier!

In contrast to ValidKube, which uses tools such as Polaris, Trivy, Kubeval, and Kube-neat under the hood, Kubescape uses security guidelines such as the NSA's Kubernetes hardening advice and the MITRE framework to audit your cluster. It is highly recommended that you use a tool such as Kubescape to audit your cluster before applying for any security compliance certifications. If your cluster passes the Kubescape scan without issues, then it is compliant with the NSA and MITRE security guidelines and therefore complies with the best in class advice.

Setting up Kubescape

On MacOS, Kubescape can be installed using Homebrew with the following commands:

brew tap armosec/kubescape
brew install kubescape

You can use the following command to install Kubescape, if you are a Windows user:

iwr -useb https://raw.githubusercontent.com/armosec/kubescape/master/install.ps1 | iex

Check if Kubescape has been installed successfully using the following command:

kubescape --version

If it is, you will be shown the installed version of Kubescape that shows that it has been installed successfully:

kubescape version v2.0.152

After installing Kubescape, use the kubescape -h command to get all the available commands, as there are many interesting features which we do not cover in this guide. In this tutorial, we will only use it to scan a YAML file that has a deployment. You may recognize it from earlier:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: boemo-app
  namespace: default
  labels:
    app: boemo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: boemo-app
  template:
    metadata:
      labels:
        app: boemo-app
    spec:
      containers:
      - name: server
        image: nginx:1.17
        volumeMounts:
          - name: boemo-app
            mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
          protocol: TCP
        resources:
          requests:
            cpu: 100m
            memory: "128M"
          limits:
            cpu: 100m
            memory: "256M"
        env:
        - name: LOG_LEVEL
          value: "DEBUG"
      volumes:
      - name: boemo-app
        configMap:
          name: boemo-app
          items:
          - key: body
            path: index.html

Use the following command to scan the above to scan the deployment YAML file, assuming you've saved it as deployment.yaml:

kubescape scan deployment.yaml --format pdf --output kubescape-scan.pdf

The --format flag specifies which output format should the scan results be presented using. The -output flag specifies the name of the pdf that will contain the scan results.

You can also use the following command which displays the results on the CLI:

 kubescape scan deployment.yaml

You will get more details if it is not in pdf format:

Kubescape Scan results in a terminal rather than in a PDF

Analyzing Kubescape results

The Kubescape scan results include the severity level of the vulnerability or threat detected column. The severity column has three levels which are Medium, Low, and High. All threats that are classified as high should be solved immediately. The "Control Name" column states the component of the cluster or manifest being scanned and inspected.

Sometimes, the scan results will have the "Docs" and "ASSISTANT REMEDIATION" columns which give you more information on how to solve the threat. The "Docs" column will give you the link that refers you to the threat documentation while the "ASSISTANT REMEDIATION" column will suggest ways on how you can solve the threat.

At the end of the scan results, you will see a summary that gives you the risk score based on different frameworks such as the MITRE frameworks.

Scanning a cluster using Kubescape

Since you now know how to scan a YAML file using Kubescape, let's now use it to scan a full cluster. Kubescape will give you an insight on how your cluster performs in terms of security vulnerabilities. Every object available in your cluster will be audited and included in the security report. The report is very detailed to a point where it will give you documentation of the security vulnerability and how to start going about fixing it.

This is where you need a cluster that is running, and the KUBECONFIG file for that cluster. Kubescape will use the currently-active KUBECONFIG context by default. If you do not have a KUBECONFIG configured, you will get an error if you scan a cluster with Kubescape.

Here is the Kubescape command used to scan your cluster:

kubescape scan --submit --enable-host-scan --format-version v2 --verbose

You will get similar output to this:

Cluster scan results

The above output is just an example of the many object results that Kubescape will give.

Scanning using a specific security guideline and framework

Kubescape allows you to scan your cluster or manifest using a specific framework. Use the following command to scan your cluster using the NSA security framework that we mentioned above:

kubescape scan framework nsa --submit

Use the following command to scan your cluster using the MITRE ATT&CK framework:

kubescape scan framework mitre --submit

Conclusion

ValidKube and Kubescape are tools that you should use to combat security vulnerabilities on a daily basis. Prevention is better than the cure.

ValidKube will surely improve your productivity and workflow as it audits, secures and validates files you pass to it, and bundles multiple useful scans.

Kubescape, on the other hand, can be configured to run as part of your CI/CD pipeline meaning that anything that is misconfigured or gets flagged due to not matching a particular security framework can halt your deployment and allow developers to fix the issue before it becomes a problem.