Troubleshooting the MapR Container Storage Interface (CSI) Storage Plugin
This section describes how to resolve common problems you might encounter when installing and using the MapR Container Storage Interface (CSI) Storage Plugin.
Troubleshooting CSI Driver installation
Run the following commands to get the pods that are deployed for the CSI plugin and provisioner:
kubectl get pods -n mapr-csi
Loopback
NFSkubectl get pods -n mapr-nfscsi
The installation is considered successful if the get pods
command shows
the pods in the Running
state. For example, your output should look similar
to the following when CSI plugin is deployed on three worker nodes:
mapr-csi csi-controller-kdf-0 5/5 Running 0 4h25m
mapr-csi csi-nodeplugin-kdf-2kfrf 3/3 Running 0 4h25m
mapr-csi csi-nodeplugin-kdf-lq5nw 3/3 Running 0 4h25m
mapr-csi csi-nodeplugin-kdf-pkrzt 3/3 Running 0 4h25m
Loopback
NFScsi-controller-nfskdf-0 7/7 Running 0 22h
csi-nodeplugin-nfskdf-5rjt2 3/3 Running 0 18h
csi-nodeplugin-nfskdf-7d9cs 3/3 Running 0 22h
csi-nodeplugin-nfskdf-qw7kg 3/3 Running 0 22h
The preceding output shows the following:
csi-nodeplugin-kdf-*
: Daemonset pods deployed on all the Kubernetes worker nodescsi-controller-kdf-0
: StatefulSet pod deployed on a single Kubernetes worker node
csi-nodeplugin-nfskdf-*
: Daemonset pods deployed on all the Kubernetes worker nodescsi-controller-nfskdf-0
: StatefulSet pod deployed on a single Kubernetes worker node
Troubleshoot MapR CSI Plugin Deployment Failures
If the pods show a failure in the deployment, run the following kubectl command to see the container logs:
kubectl logs <csi-nodeplugin-*> -n mapr-csi -c <nodeplugin-pod-container>
Loopback
NFSkubectl logs <csi-nodeplugin-*> -n mapr-nfscsi -c <nodeplugin-pod-container>
If the pods show a failure in the deployment, run the following kubectl commands to see the container logs:
kubectl logs <csi-nodeplugin-*> -n mapr-csi -c <nodeplugin-pod-container>
Loopback
NFSkubectl logs csi-controller-nfskdf-0 -n mapr-nfscsi -c <controller-pod-container>
<nodeplugin-pod-container>
with the container that is
failing. You can also run the following kubectl command to see the controller logs:
kubectl logs csi-controller-kdf-0 -n mapr-csi -c <controller-pod-container>
Here, replace <controller-pod-container>
with the container which is
failing.Troubleshooting Volume Provisioning
Check the provisioner log and check for any provisioner errors:
tail -100f /var/log/csi-maprkdf/csi-provisioner.log
Loopback
NFStail -100f /var/log/csi-maprkdf/csi-nfsprovisioner.log
Troubleshooting Mount Operation
Check the CSI Storage plug-in log for any mount/unmount errors:
tail -100f /var/log/csi-maprkdf/csi-plugin.log
Loopback
NFStail -100f /var/log/csi-maprkdf/csi-nfsplugin.log
If you don’t see any errors, see the kubelet logs on the node where the pod is scheduled to run. Check the MapR CSI Storage plugin logs for specific errors.
Troubleshooting MapR CSI Storage Plugin Discovery with kubelet
Check the kubelet path for kubernetes deployment from the kubelet process running with
--root-dir
. The --root-dir
is a string that contains the
directory path for managing kubelet files (such as volume mounts, etc.,) and defaults to
/var/lib/kubelet
. If the Kubernetes environment has a different kubelet
path, modify the CSI driver deployment .yaml
file with the new path, and
redeploy the MapR CSI Storage Plugin again.
Troubleshooting Snapshot Provisioning
tail -100f /var/log/csi-maprkdf/csi-provisioner.log
If there are no errors, run the following kubectl command to check the snapshot:
kubectl describe volumesnapshot.snapshot.storage.k8s.io <snapshot-name> -n <namespace-name>
Here:<snapshot-name>
: Name of the VolumeSnapshot Object defined in yaml<namespace-name>
: Namespace where the VolumeSnapshot object is created
Troubleshooting No Space on Disk Error
The devicemapper storage driver used for Docker allows only 10 GB by default resulting in
"no space left on device" errors when writing to new directories for a new volume mount
request. If --maxvolumepernode
is configured to be greater than 20 and
underlying docker is using devicemapper storagedriver, do the following to increase the
storage size:
- Change storagedriver to be other than devicemapper, which restricts container storage to 10 GB by default.
- Increase default container storage to more than the default of 10 GB for devicemapper storagedriver for the Docker container running on Kubernetes worker node.
- In
/etc/sysconfig/docker-storage
file, add--storage-opt dm.basesize=50G
underDOCKER_STORAGE_OPTIONS
section. - Restart Docker.
- Confirm that the setting is correctly applied by running the following command:
docker info | grep "Base Device Size"