Configuring NFSv4 Server for Kerberos

Describes how to configure and use NFSv4 on Kerberos.

About this task

You can configure MapR NFSv4 server to use Kerberos-based authentication. MapR supports configuration of NFSv4 server for Kerberos with Active Directory server and Kerberos with LDAP. You can also configure MapR NFSv4 server to work with other Kerberos installations. Before configuring MapR NFSv4 server for Kerberos, you must have performed the following:

Installed packages for Kerberos server.
Installed NFSv4 server. See Installing MapR NFS for more information.
Installed packages for Kerberos client.

NOTE The steps in this section assume a Linux-based Kerberos environment, and the specific commands for your environment may vary. Please consult with your Kerberos administrator for assistance.

By default, the NFSv4 server is configured to rely on a Kerberos infrastructure. If you don't want or don't have a Kerberos infrastructure, comment out the SecType parameter of the EXPORT section of the /opt/mapr/conf/nfs4server.conf file.

Configure NFSv4 Server for Kerberos with Active Directory Server

About this task

The following procedure describes how to configure the MapR NFSv4 server to work with the Kerberos available with Active Directory server. Before configuring the MapR NFSv4 server, ensure that Active Directory server is installed and all the nodes on the cluster have joined that Active Directory server. The following procedure requires the NFSv4 server to run under user mapr and group maprgrp.

Procedure

In an Active Directory server environment, join the cluster nodes to the Active Directory server.
Follow the sample procedure here or consult with your system administrator for assistance with installing and joining the nodes to Active Directory server.
Check if Kerberos tickets for host and NFS service principal are present, by running the following command:
```
# klist
klist: No credentials cache found (filename: /tmp/krb5cc_0)
```

Ensure host principal is available by checking to see if existing keys are present on the node.

For example, when you run the following command, the output should look similar to the following output for nfs4ad.com domain:

# klist -kt
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 04/10/2018 23:51:24 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:24 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:24 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:24 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:24 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:24 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:25 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:25 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:25 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:25 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM

Generate the host ticket by running the kinit command.

For example:

[root@atsqa4-161 ~]# kinit -k ATSQA4-161$
[root@atsqa4-161 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: ATSQA4-161$@NFS4AD.COM
Valid starting       Expires              Service principal
04/11/2018 03:04:38  04/11/2018 13:04:38  krbtgt/NFS4AD.COM@NFS4AD.COM
        renew until 04/18/2018 03:04:38

Add NFS service principal entry for the host in the AD server by running the setspn command.

For example, for nfs4ad.com domain, run the following command:

C:\Users\Administrator>setspn -A nfs/atsqa4-161.nfs4ad.com mapr
Checking domain DC=nfs4ad,DC=com
Registering ServicePrincipalNames for CN=mapr,CN=Users,DC=nfs4ad,DC=com
        nfs/atsqa4-164.nfs4ad.com
Updated object

Get the latest service ticket for the host from the AD server by running the kvno command.
For example:
```
# kvno nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM
nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM: kvno = 46
kvno nfs/qa108-43.nfs4ad.com@NFS4AD.COM
```

Add entry for NFS service principal key in the Kerberos keytab file, /etc/krb5.keytab:

# ktutil
ktutil:  addent -password -p nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM -k 46 -e RC4-HMAC
Ex: addent -password -p nfs/qa108-43.nfs4ad.com@NFS4AD.COM -k 46 -e RC4-HMAC
Password for nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM: 
( Give mapr user password i.e nfs4AD123 )
ktutil:  l
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1   46      nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM
ktutil:  wkt /etc/krb5.keytab 
ktutil:  q

Verify that NFS service principal and host principal are in the /etc/krb5.keytab file by running the klist command.

For example, for domain nfs4ad.com, run the following command and verify the entries in the file:

# klist -kt /etc/krb5.keytab 
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 ATSQA4-161$@NFS4AD.COM
   4 08/01/2018 00:29:21 ATSQA4-161$@NFS4AD.COM
   4 08/01/2018 00:29:22 ATSQA4-161$@NFS4AD.COM
   4 08/01/2018 00:29:22 ATSQA4-161$@NFS4AD.COM
   4 08/01/2018 00:29:22 ATSQA4-161$@NFS4AD.COM
  46 08/01/2018 02:58:01 nfs/atsqa4-161.nfs4ad.com@NFS4AD.COM

Ensure that /etc/krb5.keytab file is owned by user mapr and if necessary, change ownership to user mapr.

For example:

[root@qa108-41 ~]# chown mapr:root /etc/krb5.keytab 
[root@qa108-41 ~]# ls -l /etc/krb5.keytab 
-rw------- 1 mapr root 4175 Jul 22 23:53 /etc/krb5.keytab

Restart the rpcgssd service on the host to establish GSS security contexts.
```
service rpcgssd start
```
```
service gssd restart
```
Enable security variable, SecType, in the NFSv4 server configuration file at /opt/mapr/conf/nfs4server.conf.
For example:
```
  # Security type (krb5,krb5i,krb5p)
  SecType = krb5;
```
Start the NFSv4 server.
For more information, see Starting, Stopping, and Restarting MapR NFSv4.
List the shares exported on the server by running showmount -e command.
If the protocol is v4 only, the showmount command will not return the list of exported NFS shares. Instead, to view the export list, run the following command:
```
/opt/mapr/server/nfs4mgr list-exports
```
Ensure that the list-exports command runs successfully.
For example:
```
#  maprcli nfs4mgmt list-exports
Export Id     Path
30            /mapr
0             /
```

(Troubleshooting) Run the following command to restart the services if you see security-related issues.

maprcli node services -nfs4 stop -nodes `hostname` ; service rpcgssd restart; sleep 1; service rpcbind  restart ; sleep 1; service nfs restart ; service nfs stop ; sleep 2; maprcli node services -nfs4 start -nodes `hostname`

maprcli node services -nfs4 stop -nodes `hostname` ; service gssd restart; sleep 1; service rpcbind  restart ; sleep 1; service nfs-kernel-server restart ; service nfs-kernel-server stop ; sleep 2; maprcli node services -nfs4 start -nodes `hostname`

Set up VIPs for the NFSv4 servers:
1. Add entries for IPs and names of VIPs in the /etc/hosts file on the NFSv4 server host first and then on the AD server host.
  For example:
```
10.10.88.14 nfsvirtualip1
10.10.88.15 nfsvirtualip2
```
2. Add NFS service principal for the virtual IP by running the setspn command.
  For example:
```
C:\Users\Administrator>setspn -A host/nfsvirtualip1 nfsserver
C:\Users\Administrator>setspn -A nfs/nfsvirtualip1 nfsserver
                                    
C:\Users\Administrator>setspn -A host/nfsvirtualip2 nfsserver
C:\Users\Administrator>setspn -A nfs/nfsvirtualip2 nfsserver
```
3. Restart the rpcgssd service on the host to re-establish GSS security contexts.
  For example:
```
service rpcgssd restart
```

Configuring NFSv4 Server for Other Kerberos Installations

Procedure

Configure NFS server for Kerberos.
Consult with your system administrator for assistance with the commands for configuring the NFS server for Kerberos-based authentication. For example, you must do the following:
- Create a service principal with nfs as the service name.
  For example: nfs/host.domain.com@REALM
- Generate a keytab for the NFS service principal, store it in the /etc/krb5.keytab file, and set correct permissions on the file.
Enable the security variable, SecType, in the NFSv4 server configuration file at /opt/mapr/conf/nfs4server.conf.
For example:
```
  # Security type (krb5,krb5i,krb5p)
  SecType = krb5;
```
Start the NFSv4 server.
For more information, see Starting, Stopping, and Restarting MapR NFSv4.
List the shares exported on the server by running showmount -e command.
If the protocol is v4 only, the showmount command will not return the list of exported NFS shares. Instead, to view the export list, run the following command:
```
/opt/mapr/server/nfs4mgr list-exports
```
Ensure that the list-exports command runs successfully.
For example:
```
#  maprcli nfs4mgmt list-exports
Export Id     Path
30            /mapr
0             /
```

Configuring NFSv4 Client

Procedure

Ensure that NFS client has a /etc/krb5.keytab file with a valid principal similar to one of the following: nfs/<client_fqdn>@<domain>@<REALM>, host/<client_fqdn>@<domain>@<REALM>, or <HOSTNAME>$@<REALM>.
If the principal is not present, create the keytab file with the principal, which will be used to mount the share, for the OS (as mentioned in the OS vendor documentation).

Mount the cluster by running the mount command.

For example:

mount -t nfs4 -o sec=<security-type> <nfs4-server-hostname>:/<pseudo-path> <mount-point>

For example:

# mount -t nfs4 -o sec=krb5 <FQDN>:/mapr /mnt/nfs4mnt

Generate user ticket for the user to access the mount path.
For example, for user mapr on domain nfs4ad.com, run one of the following commands to generate the ticket:
- ```
kinit mapr@NFS4AD.COM
<Enter password>
```
- ```
echo usr2AD123 | kinit user2@NFS4AD.COM
```
NOTE You must renew the user ticket before it expires; otherwise, the mount path returns permissions denied error after the ticket expires.
(Troubleshooting) Restart the services and mount again to avoid security-related issues.
CentOS
```
service rpcgssd restart; sleep 1; service rpcbind  restart ; sleep 1; service nfs stop
```
Ubuntu
```
service rpcgssd restart; sleep 1; service rpcbind  restart ; sleep 1; service nfs stop
```
TROUBLE Any running IO on NFSv4 mount (with Kerberos) is stuck if the krb5 ticket expires for the current user. The mount point also hangs and becomes inaccessible.
Workaround: Restart the rpcgssd service with the new ticket to make the mount point accessible and re-trigger the IO to proceed.