Fix Long Pod Startup Time (CreateContainerError)

This page describes how you can mitigate very long Pod startup times and CreateContainerError for Pods which mount persistent volumes (PVCs) with a large number of files.

Explanation

When a Pod mounts a volume which contains many files, the Pod startup time can be very long. This is caused because the container runtime updates the SELinux labels of all the files in the volume on Pod startup. Depending on the number of files, relabeling can take a long time.

When relabeling takes too long, the Pod goes into CreateContainerError status after some time. When the container is created again, the container runtime continues relabeling files where it left off. Depending on the amount of files, multiple container restarts are required before the relabeling is done. Once the relabeling is complete, the Pod will go into status Running.

Implementation

This section explains the steps required to mitigate the long Pod startup times and CreateContainerError.

Login to APPUiO Cloud

Follow these steps to login to APPUiO Cloud on your terminal:

  1. Login to the APPUiO Cloud console:

    oc login --server=https://api.${zone}.appuio.cloud:6443

    You can find the exact URL of your chosen zone in the APPUiO Cloud Portal.

    This command displays a URL on your terminal:

    You must obtain an API token by visiting
    https://oauth-openshift.apps.${zone}.appuio.cloud/oauth/token/request
  2. Click on the link above and open it in your browser.

  3. Click "Display token" and copy the login command shown as "Log in with this token"

  4. Paste the oc login command on the terminal:

    oc login --token=sha256~_xxxxxx_xxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxx-X \
        --server=https://api.${zone}.appuio.cloud:6443
  5. Switch to the correct project.

    oc project [YOUR_PROJECT_NAME]

Change the Deployment SecurityContext

In this example, a Deployment is used, but the same change can be applied to all Kubernetes resources specifying a Pod spec.

Set the following securityContext in the Deployment using oc edit:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: [YOUR_DEPLOYMENT_NAME]
  namespace: [YOUR_PROJECT_NAME]
spec:
  template:
    spec:
      securityContext:
        seLinuxOptions:
          type: spc_t
...

You can also use oc patch to change the securityContext of the Deployment:

oc patch deployment [YOUR_DEPLOYMENT_NAME] -p '{"spec":{"template":{"spec":{"securityContext":{"seLinuxOptions":{"type":"spc_t"}}}}}}'

For more information, see SELinuxOptions Spec.

All Pods managed through the Deployment will be restarted. However, with the modification in place, you should notice a much faster Pod startup time.