Troubleshooting Tips for FIPS Installations

Answers frequently asked questions and provides troubleshooting tips for FIPS installations.

Release 7.0.0 provides support for FIPS 140-2 Level 1 compliance and many new and upgraded security features, as well as modified functionality. If you are used to configuring and running applications on the core platform before release 7.0.0, you will encounter behavioral differences. These differences can lead to incorrect or unexpected functionality, which might not be a bug. The following is a partial list of answers to frequently asked questions and troubleshooting tips.

1. Can I upgrade an existing installation to FIPS mode?

FIPS is supported only for fresh installations. However, users with pre-7.0.0 installations can take advantage of FIPS functionality while still maintaining access to critical data. You can do this by adding new FIPS nodes to an already-installed secure non-FIPS cluster and then gradually decommissioning the non-FIPS nodes. Be aware, however, that if any existing user data is encrypted using non-secure cryptographic algorithms (such as DES), the data will first have to be re-encrypted using a secure cryptographic algorithm (such as AES-256) in order to decrypt the data from a FIPS-enabled node.

2. When I enable FIPS on a node, does it mean my entire cluster is FIPS-enabled? Can I just enable FIPS for the whole cluster? After I enable FIPS, can I disable it?

The data-fabric approach to FIPS compliance is a combination of the following strategies:
  • Leveraging our partner’s FIPS certification for OpenSSL 1.1.1 for C/C++ components.
  • Bundling the Bouncy Castle Java FIPS API with the HPE Ezmeral Data Fabric distribution.
The first part means that FIPS has to be enabled at an operating-system level, on a per-node basis. This needs to be done on each cluster node before the HPE Ezmeral Data Fabric is installed. Although you can technically disable FIPS after enabling it at an operating system level, this is not supported by the HPE Ezmeral Data Fabric and will result in incorrect functionality.

The second part regarding the bundling of the Bouncy Castle JARs results in the use of BCFKS key and trust stores if the node is FIPS-enabled, and JKS key and trust stores for secure non FIPS-enabled nodes. FIPS is not just a flag, and there is no concept of a cluster-wide FIPS setting. Also, the entire cluster is FIPS-enabled if every node in the cluster is FIPS-enabled. You can use the new isFips property in the enhanced maprcli node list command to determine the FIPS setting of each server node in the cluster. See Determining if a Host Is in FIPS Mode.

3. Can a non-FIPS client communicate with a FIPS server? Can a non-FIPS server communicate with a FIPS server? How do I set this up

Yes, non-FIPS nodes can communicate with FIPS nodes, whether they are servers or clients. The TLS 1.2 communication between the nodes already uses FIPS-compliant cryptography, so no additional setup is required.

4. I want to set up a cross-cluster trust relationship. Can I still use the configure-crosscluster.sh script to configure FIPS and non-FIPS clusters?

You can use the configure-crosscluster.sh script to configure local and remote clusters if they consist of entirely FIPS nodes or entirely non-FIPS nodes. If you want to configure a cross-cluster relationship where either the local or remote clusters consist of a combination of FIPS and non-FIPS nodes, configure-crosscluster.sh provides some limited support, but some additional manual steps are required, as outlined in Configuring Cross-Cluster Security for a Mixed (FIPS and Non-FIPS) Configuration.

5. Why doesn't keytool work the same way on a FIPS node and a non-FIPS node?

The keytool command that comes as part of the JDK supports JKS key and trust stores by default since the JKS key and trust store type is packaged with the JDK distribution. JKS is also the default store type, so the -storetype parameter is optional. The JKS store type is insecure, and does not require a password for store operations such as listing the contents. For example, to list the contents of the trust store in a secure non-FIPS node:
keytool -list -keystore /opt/mapr/conf/ssl_truststore
For FIPS-enabled nodes, the HPE Ezmeral Data Fabric uses the FIPS-compliant BCFKS store type, which is part of the Bouncy Castle Java FIPS API. Since this is not packaged with the JDK distribution, you need to tell keytool where to find the Bouncy Castle provider. In addition, BCFKS stores require that you specify the password for all store operations. So, for the same example of listing the contents of the trust store in a FIPS-enabled node:
keytool -list -keystore /opt/mapr/conf/ssl_truststore.bcfks \
         -storepass B28_Xqpcu7_srB8So2T_egUiFn0q9zqZ \ 
         -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \ 
         -providerpath /opt/mapr/lib/bc-fips-1.0.2.1.jar \ 
         -providername BCFIPS -storetype bcfks

6. My custom application runs successfully on a FIPS-enabled node. Does that means my application is FIPS 140-2 Level 1 compliant?

Not necessarily. An application is considered FIPS 140-2 Level 1 compliant if all of the following are true:
  • The application runs successfully on a FIPS-enabled node such as RedHat 8.
  • All sensitive data is encrypted.
  • The application uses only cryptography from FIPS-compliant cryptographic libraries.
If, for example, your application is written in C/C++ and it statically linked a non FIPS-validated cryptographic library, such as CryptoPP, instead of using the FIPS-compliant OpenSSL 1.1.1 from RedHat, or if sensitive data in the application, such as password data, is not encrypted, the application is not considered FIPS-compliant, even if it runs on a FIPS node.

7. My custom application runs successfully on a release 7.0.0 secure non-FIPS enabled node. Why does it fail on a FIPS-enabled node?

There are multiple reasons for this. FIPS enforces the use of strong cryptographic algorithms from FIPS-approved cryptographic libraries such as RedHat’s OpenSSL 1.1.1 and the Bouncy Castle Java FIPS API. It disables weak cryptographic algorithms such as DES. Following is a partial list of reasons that your application will fail on a FIPS-enabled node:
  • The application uses weak cryptographic algorithms, such as DES and MD-5.
  • The application is written in C/C++ and uses a version of OpenSSL other than 1.1.1.
  • The application is written in Java and needs to communicate over TLS, but is unable to access the private keys and/or certificates because it does not support the BCFKS store type.
  • The application is trying to access data that was previously encrypted using nonsecure cryptographic algorithms such as DES.
  • There is insufficient entropy.

8. I have data stored in the file system or database that was previously encrypted using DES. My application on my FIPS-enabled node fails when I try to access the data. How can I retrieve the data?

FIPS-enabled nodes do not support nonsecure algorithms such as DES. If you stored encrypted data in the file system or database from a non-FIPS node, then you must migrate your data. That is, you must decrypt your DES-encrypted data and re-encrypt your data using a stronger cryptographic algorithm – such as AES – on a non-FIPS node before you can access the data on a FIPS-enabled node. To improve your security infrastructure, Hewlett Packard Enterprise recommends that you upgrade your application to support stronger cryptographic algorithms, such as AES.

9. After installing a new FIPS node, running configure.sh results in a hang (specifically, the call to “keytool” hangs for many minutes). What can be done?

The likely reason is low system entropy on the node. Low system entropy is more common on virtual machines; physical machines generate better random numbers from various hardware sources. The keytool command reads random bits from /dev/random, which blocks while trying to generate new random bits.

Having good entropy available is a pre-requirement for installing the HPE Ezmeral Data Fabric. It is not a feature that the Data Fabric can provide or test for. To see the current entropy level, use this command:
sudo cat /proc/sys/kernel/random/entropy_avail
Numbers near 3000 or higher are preferable. Some VMs show low entropy (for example, 50). You can search online for possible ways to increase the entropy. These include using the rng-tools or haveged packages. For example: see these references: