Troubleshooting the Container Storage Interface (CSI) Storage Plugin

Troubleshooting CSI Driver installation

Run the following commands to get the pods that are deployed for the CSI plugin and provisioner:

kubectl get pods -n mapr-csi

kubectl get pods -n mapr-nfscsi

The installation is considered successful if the get pods command shows the pods in the Running state. For example, your output should look similar to the following when CSI plugin is deployed on three worker nodes:

mapr-csi       csi-controller-kdf-0         5/5     Running     0          4h25m
mapr-csi       csi-nodeplugin-kdf-2kfrf     3/3     Running     0          4h25m
mapr-csi       csi-nodeplugin-kdf-lq5nw     3/3     Running     0          4h25m
mapr-csi       csi-nodeplugin-kdf-pkrzt     3/3     Running     0          4h25m

csi-controller-nfskdf-0       7/7     Running   0          22h
csi-nodeplugin-nfskdf-5rjt2   3/3     Running   0          18h
csi-nodeplugin-nfskdf-7d9cs   3/3     Running   0          22h
csi-nodeplugin-nfskdf-qw7kg   3/3     Running   0          22h

The preceding output shows the following:

Troubleshoot CSI Plugin Deployment Failures

If the pods show a failure in the deployment, run the following kubectl command to see the container logs:

kubectl logs <csi-nodeplugin-*> -n mapr-csi -c <nodeplugin-pod-container>

kubectl logs <csi-nodeplugin-*> -n mapr-nfscsi -c <nodeplugin-pod-container>

If the pods show a failure in the deployment, run the following kubectl commands to see the container logs:

kubectl logs <csi-nodeplugin-*> -n mapr-csi -c <nodeplugin-pod-container>

kubectl logs csi-controller-nfskdf-0 -n mapr-nfscsi -c <controller-pod-container>

kubectl logs csi-controller-kdf-0 -n mapr-csi -c <controller-pod-container>

Troubleshooting Volume Provisioning

Check the provisioner log and check for any provisioner errors:

tail -100f /var/log/csi-maprkdf/csi-provisioner.log

tail -100f /var/log/csi-maprkdf/csi-nfsprovisioner.log

Troubleshooting Mount Operation

Check the CSI Storage plug-in log for any mount/unmount errors:

tail -100f /var/log/csi-maprkdf/csi-plugin.log

tail -100f /var/log/csi-maprkdf/csi-nfsplugin.log

If you don’t see any errors, see the kubelet logs on the node where the pod is scheduled to run. Check the MapR CSI Storage plugin logs for specific errors.

Troubleshooting CSI Storage Plugin Discovery with kubelet

Check the kubelet path for kubernetes deployment from the kubelet process running with --root-dir. The --root-dir is a string that contains the directory path for managing kubelet files (such as volume mounts, etc.,) and defaults to /var/lib/kubelet. If the Kubernetes environment has a different kubelet path, modify the CSI driver deployment .yaml file with the new path, and redeploy the CSI Storage Plugin again.

Troubleshooting Snapshot Provisioning

tail -100f /var/log/csi-maprkdf/csi-provisioner.log

kubectl describe volumesnapshot.snapshot.storage.k8s.io <snapshot-name> -n <namespace-name>

Troubleshooting No Space on Disk Error

The devicemapper storage driver used for Docker allows only 10 GB by default resulting in "no space left on device" errors when writing to new directories for a new volume mount request. If --maxvolumepernode is configured to be greater than 20 and underlying docker is using devicemapper storagedriver, do the following to increase the storage size:

1. Change storagedriver to be other than devicemapper, which restricts container storage to 10 GB by default.
2. Increase default container storage to more than the default of 10 GB for devicemapper storagedriver for the Docker container running on Kubernetes worker node.