Fix Long Pod Startup Time (CreateContainerError)

This page describes how you can mitigate very long Pod startup times and CreateContainerError for Pods which mount persistent volumes (PVCs) with a large number of files.

Explanation

When a Pod mounts a volume which contains many files, the Pod startup time can be very long. This is caused because the container runtime updates the group ownership and SELinux labels of all the files in the volume on Pod startup. Depending on the number of files, relabeling can take a long time.

When relabeling takes too long, the Pod goes into CreateContainerError status after some time. When the container is created again, the container runtime continues updating group ownership and relabeling files where it left off. Depending on the amount of files, multiple container restarts are required before the group ownership updates and relabeling is done. Once the relabeling is complete, the Pod will go into status Running.

Implementation

This section explains the steps required to mitigate the long Pod startup times and CreateContainerError.

1. Login to the APPUiO Cloud console:

oc login --server=https://api.${zone}.appuio.cloud:6443 You can find the exact URL of your chosen zone in the APPUiO Cloud Portal. This command displays a URL on your terminal: You must obtain an API token by visiting https://oauth-openshift.apps.${zone}.appuio.cloud/oauth/token/request
2. Click on the link above and open it in your browser.

4. Paste the oc login command on the terminal:

--server=https://api.\${zone}.appuio.cloud:6443
5. Switch to the correct project.

oc project [YOUR_PROJECT_NAME]

Change the Deployment SecurityContext

 In this example, a Deployment is used, but the same change can be applied to all Kubernetes resources specifying a Pod spec.

Set the following securityContext in the Deployment using oc edit:

apiVersion: apps/v1
kind: Deployment
name: [YOUR_DEPLOYMENT_NAME]
namespace: [YOUR_PROJECT_NAME]
spec:
template:
spec:
securityContext:
fsGroupChangePolicy: OnRootMismatch (1)
seLinuxOptions:
type: spc_t (2)
...
 1 Configure the deployment to only change permissions and ownership for files in the volume if the permission and ownership of the root directory doesn’t not match the expected permissions. 2 Use SELinux option spc_t to ensure the container runtime doesn’t try to relabel all files on the volume.

You can also use oc patch to change the securityContext of the Deployment:

oc patch deployment [YOUR_DEPLOYMENT_NAME] -p '{"spec":{"template":{"spec":{"securityContext":{"fsGroupChangePolicy":"OnRootMismatch","seLinuxOptions":{"type":"spc_t"}}}}}}'

For more information on fsGroupChangePolicy, see the Kubernetes documentation on configuring volume permission and ownership change. For more information on SELinux options, see the SELinuxOptions Spec.

 All Pods managed through the Deployment will be restarted. However, with the modification in place, you should notice a much faster Pod startup time.