Table of Contents

what & why

The CKS (Certified Kubernetes Security Specialist) is a great resource for knowing how to secure a kubernetes cluster. It covers a lot of topics, from the cluster side (admission controller, webhooks, audit), app side (Pod Security Policies) and supply chain (image scanning). Another great resource for this is the Kubernetes Hardening Guidance by NSA & CISA

But some of the concepts defined in both these resources are very case-specific, and require a lot of time, tools & effort to setup. In some environnements, it might be infeasible to deploy each and every one of those concepts. But that doesn’t mean we should avoid some basic security-minded steps when deploying to k8s. I won’t cover things on the cluster-side (audit, tools like falco, or admission controllers), but how you can improve the security of your front-facing app by adding a few lines here and there.

how

Let say we have a python app, that exposes an API. We have a basic Dockerfile for it, and a simple deploy.yaml spec file containing our Deployment. They are what you could call ’typical’ of what can be found online when looking for a template of dockerfile or deployment:

Dockerfile:

FROM python:3-alpine
COPY requirements.txt .
RUN pip3 install -r requirements.txt

WORKDIR /app
COPY src/ ./

ENV PYTHONUNBUFFERED 1
CMD python3 app.py

deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: registry/api:latest
        ports:
        - containerPort: 5000

Let’s secure things up :

non root user

The first recommendation is to run our containers as non-root users. For that, we’ll first add a few lines to our Dockerfile:

...
RUN addgroup -S app && adduser -H -D -S app -G app -s /dev/null
USER app:app
WORKDIR /app

Note that we are using an alpine-based container, so the exact command might vary on other distros, but the goal is the same

By creating a user ‘app’, and using it to run our app, we avoid giving way too much permissions to the app. We can check that by exec’ing into the running container:

(before)

/app # id
uid=0(root) gid=0(root) groups=1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
/app # ps
PID   USER     TIME  COMMAND
    1 root      0:00 {flask-api.py} /usr/local/bin/python3 ./flask-api.py
    7 root      0:00 sh
   14 root      0:00 ps

(now)

/app $ id
uid=101(app) gid=101(app)
/app $ ps
PID   USER     TIME  COMMAND
    1 app       0:00 {flask-api.py} /usr/local/bin/python3 ./flask-api.py
    7 app       0:00 sh
   14 app       0:00 ps

Right, so now we are not running as root on the container side, but what about the host ? To which user are we mapped to on the host side ?

when running the default image:

k8s-worker > ps -aux | grep flask
root     2266529  1.8  0.1  28168 23752 ?        Ss   17:31   0:00 /usr/local/bin/python3 ./flask-api.py

We are actually mapped to the root user ! that’s not the most secure setup ! If somehow an attacker gain control of the pod and is able to escape, he will land on the host as the root user.

And if we use the new Dockerfile with the USER directive:

ps -aux | grep flask
syslog   2267104  2.8  0.1  28168 23812 ?        Ss   17:32   0:00 /usr/local/bin/python3 ./flask-api.py

What ? Why are we mapped to the syslog user ? A quick check of /etc/passwd shows us why :

cat /etc/passwd | grep 101
syslog:x:101:107::/home/syslog:/usr/sbin/nologin

This is because when we created the user app in the Dockerfile, it assigned to it the uid 101, which is the same as the syslog user on our host.

To avoid clashing with a potential user with the same uid on the host, we can use a higher uid, in the 40000-60000 range.

runAsUser

To fix that, we will tweak our deploy.yaml:

      containers:
      - name: api
        image: registry/api:latest
        securityContext:
          runAsUser: 60096
          runAsGroup: 60096

Now, from the host side, we will appear as uid 60096, which isn’t mapped to a predefined user (unless a user with the same uid exists on the host obviously).

readOnlyRootFilesystem

Another great addition to the Deployment spec, it to set the filesystem as readonly. This will block any attempt to modify the filesystem of the container, like installing binaries, modifying configuration in /etc..

      containers:
      - name: api
        image: registry/api:latest
        securityContext:
          readOnlyRootFilesystem: true
          runAsUser: 60096
          runAsGroup: 60096

Now if we try to change something, we’ll be greeted by an error message, preventing any change on the root fs:

ERROR: Unable to lock database: Read-only file system
ERROR: Failed to open apk database: Read-only file system

If our app needs to be able to write in a given folder (like a /tmp/cache folder for caching), we can create a volumeMount in the deployment, mapped to an ’emptyDir’. This will allow the app to write at this folder, while retaining the read-only root fs:

      containers:
      - name: api
        image: registry/api:latest
        volumeMounts:
        - mountPath: /tmp/cache
          name: cache-volume
      volumes:
      - name: cache-volume
        emptyDir: {}

automountServiceAccountToken

If the pod is not going to communicate with the kubernetes API, we can avoid mounting the service account’s token in the pod (which by default will be mounted in /var/run/secrets/kubernetes.io/serviceaccount/token):

    spec:
      automountServiceAccountToken: false
      containers:

resources

While we’re on the deployment, it’s also a good idea to set some resources limits on the pod. This will prevent the pods from consuming all the resources from the host, which even if it’s from a genuine mistake, can result in outages or other disurptions:

      containers:
      - name: api
        image: registry/api:latest
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 64Mi

requests will tell k8s the requirements for the pod, i.e what we can expect the pod to consume. This will aid the scheduling on an appropriate node.

limits will actually stop the pod from consuming more than what is specified, either by throttling for the cpu, or OOM for the memory.

SHA tagging

Another recommendation would be to use SHA tagging on the base image of our Dockerfile. This serves two purposes:

• making reproducible builds possible. Otherwise, as the base image can be updated, this will break our current setup even though our Dockerfile hasn’t changed.
• aleviate supply chain attacks: If the base image used is subject to a supply chain attack (a threat actor injects/modify the image), our image becomes subject as well because we’ll pull the latest version of the image.

To do so, we simply add the SHA of the image we want to fix at the end of the tag:

FROM python:3.9-alpine3.15@sha256:f2aeefbeb3846b146a8ad9b995af469d249272af804b309318e2c72c9ca035b0

We can repeat the same behavior on the deployment. Instead of using registry/api:latest, but by using a versionning scheme (like semver or calver) and combining this with the SHA of the image, we ensure that the image used in the deployment will not be updated due to upstream changes.

Results

Final versions of the Dockerfile and deploy.yaml would look like this:

Dockerfile

FROM python:3.9-alpine3.15@sha256:f2aeefbeb3846b146a8ad9b995af469d249272af804b309318e2c72c9ca035b0
COPY requirements.txt .
RUN pip3 install -r requirements.txt

RUN addgroup -S app && adduser -H -D -S app -G app -s /dev/null
USER app:app
WORKDIR /app
COPY src/ ./

ENV PYTHONUNBUFFERED 1
CMD ["python3","app.py"]

deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      automountServiceAccountToken: false
      containers:
      - name: api
        image: registry/api:v1.0.3@sha256:f469d249272af80...
        securityContext:
          readOnlyRootFilesystem: true
          runAsUser: 69096
          runAsGroup: 69096
        ports:
        - containerPort: 5000
        volumeMounts:
        - mountPath: /tmp/cache
          name: cache-volume
      volumes:
      - name: cache-volume
        emptyDir: {}

Theses are simples yet effectives ways to secure your app on k8s. If you want to go deeper, I would suggests checking more advanced topics like host-side security (using apparmor profiles for example), falco , or supply chain security (Open Policy Agent, Admission controllers..). Theses are all topics that are covered by the CKS, and with plenty of information online.

Further resources

• https://pwning.systems/posts/escaping-containers-for-fun/
• https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
• https://kubernetes.io/docs/tasks/configure-pod-container/security-context/