Fix Long Pod Startup Time (CreateContainerError)

This page describes how you can mitigate very long Pod startup times and CreateContainerError for Pods which mount persistent volumes (PVCs) with a large number of files.

Explanation

When a Pod mounts a volume which contains many files, the Pod startup time can be very long. This is caused because the container runtime updates the group ownership and SELinux labels of all the files in the volume on Pod startup. Depending on the number of files, relabeling can take a long time.

When relabeling takes too long, the Pod goes into CreateContainerError status after some time. When the container is created again, the container runtime continues updating group ownership and relabeling files where it left off. Depending on the amount of files, multiple container restarts are required before the group ownership updates and relabeling is done. Once the relabeling is complete, the Pod will go into status Running.

Implementation

This section explains the steps required to mitigate the long Pod startup times and CreateContainerError.

Login to APPUiO Cloud

Follow these steps to login to APPUiO Cloud on your terminal:

oc login --server=https://api.${zone}.appuio.cloud:6443

You can find the exact URL of your chosen zone in the APPUiO Cloud Portal.

This command displays a URL on your terminal:

You must obtain an API token by visiting
https://oauth-openshift.apps.${zone}.appuio.cloud/oauth/token/request

Click on the link above and open it in your browser.
Click "Display token" and copy the login command shown as "Log in with this token"

Paste the oc login command on the terminal:

oc login --token=sha256~_xxxxxx_xxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxx-X \
    --server=https://api.${zone}.appuio.cloud:6443

Switch to the correct project.
```
oc project [YOUR_PROJECT_NAME]
```

Change the Deployment SecurityContext

In this example, a Deployment is used, but the same change can be applied to all Kubernetes resources specifying a Pod spec.

Set the following securityContext in the Deployment using oc edit:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: [YOUR_DEPLOYMENT_NAME]
  namespace: [YOUR_PROJECT_NAME]
spec:
  template:
    spec:
      securityContext:
        fsGroupChangePolicy: OnRootMismatch (1)
        seLinuxOptions:
          type: spc_t (2)
...

1	Configure the deployment to only change permissions and ownership for files in the volume if the permission and ownership of the root directory doesn’t not match the expected permissions.
2	Use SELinux option `spc_t` to ensure the container runtime doesn’t try to relabel all files on the volume.

You can also use oc patch to change the securityContext of the Deployment:

oc patch deployment [YOUR_DEPLOYMENT_NAME] -p '{"spec":{"template":{"spec":{"securityContext":{"fsGroupChangePolicy":"OnRootMismatch","seLinuxOptions":{"type":"spc_t"}}}}}}'

For more information on fsGroupChangePolicy, see the Kubernetes documentation on configuring volume permission and ownership change. For more information on SELinux options, see the SELinuxOptions Spec.

All Pods managed through the Deployment will be restarted. However, with the modification in place, you should notice a much faster Pod startup time.