Known Issues at Release (MapR 6.0.1)

Installation and Configuration Issues

You can see generic installation issues here: MapR Installer Known Issues.

Upgrade Issues

MD-4026

Issue: After an upgrade to MapR 6.0.1 or later, Drillbits can fail to start if the cluster has Drill 1.11 or Drill 1.12 and the Drill package is not upgraded as part of the upgrade. This issue can also occur during a manual rolling upgrade. The issue occurs because the upgrade process removes symlinks to MapR-FS jar files in the /opt/mapr/lib directory.

Workaround: After upgrading MapR Core, paste the following script into a Linux shell, and execute the script on each of the upgraded nodes.

MAPR_HOME=/opt/mapr
DRILL_BASE=${MAPR_HOME}/drill
DRILL_VERSION=`cat ${DRILL_BASE}/drillversion`
DRILL_HOME=${DRILL_BASE}/drill-${DRILL_VERSION}

find ${DRILL_HOME}/jars -type l -exec rm -f {} \;
ln -sf ${MAPR_HOME}/lib/maprfs*.jar ${DRILL_HOME}/jars/3rdparty/.
ln -sf ${MAPR_HOME}/lib/maprdb*.jar ${DRILL_HOME}/jars/3rdparty/.
ln -sf ${MAPR_HOME}/lib/mapr-hbase*.jar ${DRILL_HOME}/jars/3rdparty/.
rm -f ${DRILL_HOME}/jars/3rdparty/mapr*tests.jar
rm -f ${DRILL_HOME}/jars/3rdparty/maprfs-d*
rm -f ${DRILL_HOME}/jars/3rdparty/maprdb-c*
rm -f ${DRILL_HOME}/jars/3rdparty/maprdb-s*

After invoking the script, drillbits should start successfully. To confirm that there are symlinks to jar files in opt/mapr/lib, use this command:

 ls -ltr $DRILL_HOME/jars/3rdparty/mapr* 

31038

Issue: During an upgrade to MapR 6.0.1 from any other MapR release, the following file is not updated automatically:

/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/log4j.properties

Without the properties updates, you cannot use the scheduler debug logger.

Workaround: This workaround is necessary only if you plan to use the scheduler debug logger. After the upgrade to MapR 6.0.1, add the following lines to the log4j.properties file:

yarn.debug.controller.logger = INFO, ED
yarn.debug.controller.log.file=yarn-mapr-scheduling-debug.log
yarn.debug.controller.log.maxfilesize=256MB
yarn.debug.controller.log.maxbackupindex=20

log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.scheduler.
DebugController=${yarn.debug.controller.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.scheduler.
DebugController=false
 
log4j.appender.ED=org.apache.hadoop.yarn.WorldWritableLogAppender
log4j.appender.ED.File=${hadoop.log.dir}/${yarn.debug.controller.log.file}
log4j.appender.ED.MaxFileSize=${yarn.debug.controller.log.maxfilesize}
log4j.appender.ED.layout=org.apache.log4j.PatternLayout
log4j.appender.ED.layout.ConversionPattern=%d{ISO8601} : %m%n 

29752

Issue: The Oozie process fails following a manual upgrade from MapR 5.2.x (MEP 3.0.1) to MapR 6.0 if Oozie is not upgraded to the MEP 4.0.0 version.

Workaround: Upgrading all ecosystem components to MEP 4.0.0 is required before you can enable MapR 6.0 features. If you cannot upgrade to MEP 4.0.0 immediately, use these steps to restart Oozie:

1. Stop Oozie:

   maprcli node services -name oozie -action stop -nodes <nodes>

2. Recreate the Oozie war file by using one of the following commands:
   • If SSL was enabled for Oozie, use this command:

     /opt/mapr/oozie/oozie-<version>/bin/oozie-setup.sh -hadoop <version> /opt/mapr/hadoop/hadoop-<version> -secure

   • If SSL was not enabled for Oozie, use this command:

     /opt/mapr/oozie/oozie-<version>/bin/oozie-setup.sh -hadoop <version> /opt/mapr/hadoop/hadoop-<version>

3. Start Oozie:

   maprcli node services -name oozie -action start -nodes <nodes>

Indexing and Mapr Drill

28041

Running multiple queries in sequence can cause drillbits to run out of direct memory. Limiting records in large batches to 5KB may prevent this issue from occurring. Should Drill return an out of memory exception, issue the following command to restart the drillbits:

$ maprcli node services -name drill-bits -action restart -nodes <node host names separated by a space>

28096

If you use the Google Guava library in your application and encounter the following error:

Caused by: java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch;

You must use version 18.0 of the library. If you use Maven, you can use the following dependency:

            <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>18.0</version>
            </dependency>       

MapR-DB

27057

When you issue a maprcli table changelog add command, during the initial sync when the existing data is copied to the change log, the end-to-end latency during the initial 5 minutes can be high. After the initial sync, any latency will be small, specifically 2-4 seconds.

MAPRDB-2092

In the MapR Database, adding a table index or replicating a table fails if the cluster administrator (MAPR_USER) does not have write access to the parent volume of the table.

Workaround: For more information, see this support advisory. If a patch is not available or the available patch has not been applied, another workaround is to add the MAPR_USER to the writeACE for the table parent volume.

MapR-ES

31074

MapR-ES APIs that make use of streams.rpc.timeout.ms are best effort APIs, the timeout is not guaranteed.

MapR Client

23624

The same MapR client can access both secure and nonsecure clusters; however, a MapR client that is configured to access a secure cluster can access a nonsecure cluster only if these conditions are met:

• The secure cluster must be listed first in mapr-clusters.conf.
• A user must obtain a ticket for the secure cluster even if the user wants to access only the nonsecure cluster.

MapR Control System

MON-4891

Description: The MCS displays an Oops error on the Admin > User Settings > SMTP page when the mapr.smtp.provider is not any of the 3 allowed options (Gmail, Office 365, SMTP). This impacts all upgrade from 5.x to 6.x where your SMTP provider setting is "other".

Resolution: Run the following command to resolve this error if your SMTP provider is not Gmail or Office 365:

maprcli config save -values {"mapr.smtp.provider":"smtp"}

If your SMTP provider is Office 365, run the following command:

maprcli config save -values {"mapr.smtp.provider":"office"}

MON-4793

Issue: When you use the browser to point to the same MCS URL for different MapR versions, you may see an error because the local browser storage (used to store MCS user preferences) is incongruous with the version of MCS installed.

Workaround:

1. Remove the MCS user preferences file from cluster.
   For example, to remove the preferences file, run the following commands:

   hadoop fs -rm /var/mapr/cluster/admin/.root.mcs.json
   hadoop fs -rm /var/mapr/cluster/admin/.mapr.mcs.json

2. Open an incognito browser (or delete local storage for current browser for MCS host) and then point to MCS.

   This forces the file to be recreated on the cluster.

MON-4783

Issue: When you set a custom refresh rate for the charts, the page displays an "Oops" error when the page (automatically) refreshes.

Workaround: Manually refresh the browser if/when you see the error or set the custom refresh rate to a high value so that the page does not display the error frequently.

MON-3922

Description: After upgrading to v6.0.1, editing volumes that have no volumes ACEs is not supported in MCS. Errors display when attempting to view or edit volumes that were created using MapR Converged Data Platform version <=5.1.

Workaround: To resolve the issue, do one of the following:

• Use the volume modify command to set or modify volume ACEs.
• Download an install the following patches:
  • mapr-apiserver-6.0.1.20190305130833.GA‌-1.noarch
  • mapr-webserver-6.0.1.20190305130833.GA‌-1.noarch

MON-3476

Issue:In the MapR Control System, table permissions can only be modified by the owner/creator of the table.

Workaround: User, who is not the owner but has table admin permissions, must use the CLI to modify table permissions.

MON-3452

Issue: After selecting the Include System Volumes checkbox to display system volumes in the Volumes list pane, if you set quota (using the link in the Quota column), the system volumes are no longer displayed in the list of volumes even though the system volumes checkbox is still selected.

Workaround: Deselect and then re-select the Include System Volumes checkbox to display the system volumes in the list of volumes.

MON-3438

Issue: You may experience some latency in the Admin > Cluster Settings > Alarms page when you click the name of an alarm to display the Alarm Settings window for the alarm.

MON-3425

Issue: When you filter the list of alarms in the Alarms summary page, the page does not refresh and display the correct list of alarms if there are muted alarms.

MON-3407

Issue: If you are upgrading from a pre-6.0 version, the MapR Control System might display 0 instances of webserver running in the Services page. This is because starting from v6.0, the UI static files are in mapr-webserver and the mapr-apiserver runs the server that sends the queries. For more information on webserver in v6.0, see Setting Up the MapR Control System.

MON-3404

Issue: The topic search field in the Stream (detail) > Topic tab does not display correctly in Firefox.

Workaround: Use Chrome (v58 and later) to perform a topic search.

MON-3325

Issue: The MapR Control System does not display topologies with hostname (in the topology). You cannot configure node-specific topology for a volume from MCS.

Workaround: Use the CLI to set node-specific topology for a volume.

MON-3248

Issue: The MapR Control System does not support filtering list of nodes by services running on client nodes.

Workaround: Use the node list command with the filter option to retrieve the list of nodes, including client nodes, running a service.

MON-3247

Issue: The MapR Control System does not display NFS edge nodes and POSIX client nodes that are currently connected to the cluster when you sort the list of nodes by topology.

Workaround: Use the node list command to retrieve the list of client nodes currently connected to the cluster.

MON-2409

Issue: When you try to navigate to another cluster by clicking on the cluster name from the dropdown list at the top of the page, the Overview page for the current cluster displays and it takes a few minutes to display the MCS login page for the other cluster.

MON-2219

Issue: If you add Streams license separately, you may not be able to view the Streams page.

Workaround: Contact MapR Support.

MON-2218

Issue: When you preserve snapshots, there is no notification (success message) or indication (Expires On column still does not show No Expiration) even though the operation succeeded.

Workaround: Refresh the page after preserving a snapshot to see the No Expiration setting in the Expires On column.

MON-2076

Issue: When you select topics to remove, deselect all the topics in the Remove Topic confirmation window and click Cancel, then select a topic in the page by clicking on the checkbox, the Remove Topic window opens again even if you did not click on the Remove Topic button.

Workaround: Refresh the page.

MON-2018

Issue: In the Add Change Log page, after entering field to add in the Publish Selected Field Path text field, the Add Field button is disabled.

Workaround: Click outside the text field to enable the Add Field button.

MON-2011

Issue: When adding or editing the compression setting on a replica, selecting Inherited does not have any effect. The default value is lz4.

Workaround: Do not select Inherited as the compression setting on a replica.

MON-2009

Issue: When modifying replica of a binary table with three or more column families, if you select different set of columns to replicate for different column families, the operation might result in an error.

MON-1994

Issue: If you adjust the system time, the time stamp associated with the alarm in the Active Alarms pane might show negative value.

MON-1968

Issue: In the User Disk Usage page, you cannot remove the email address after setting an email address for an entity by editing the properties.

MON-1929

Issue: The Save Changes button in the Access Control Expression builder page is disabled after entering user, group, or role name in text field.

Workaround: Click outside the field to enable the Save Changes button.

MON-1810

Issue: When editing multiple volumes, you cannot undo changes to Auditing and Accountable Entity fields by clicking .

MON-1755

Issue: When creating a mirror volume, after entering the name of the source volume on the remote cluster and selecting the remote cluster on which the volume resides, you may see an error that the source volume does not exist.

Workaround: Re-enter the name of the source volume on the remote cluster.

MON-1221

Issue: In the Alarm Settings window, if you click the cursor in the Additional email recipients text field and then press the return/enter key in the keyboard without entering an email address, the Cluster Settings page displays once again.

MapR Monitoring

ES-58

Issue: After an upgrade from MapR 5.2.x with MEP 3.0.x to MapR 6.x using the MapR Installer, Elasticsearch and Fluentd fail to start, and the Elasticsearch roles file (/opt/mapr/roles/elasticsearch) is missing. This issue has been seen on upgrades from MapR 5.2.1 with MEP 3.0.3 to MapR 6.1.0 with MEP 6.0.0 but can affect all upgrades from MEP 3.0.x.

Workaround: To work around the issue, you must re-create the roles file. On the node where Elasticsearch was upgraded and the roles file was removed:

1. Re-create the roles file by using this command:

   touch /opt/mapr/roles/elasticsearch

2. Run configure.sh with the -R option:

   /opt/mapr/server/configure.sh -R

3. Restart Fluentd using the MapR Control System (MCS) or a terminal.

OTSDB-72

Issue: The memory allocated to OpenTSDB can be insufficient, resulting in empty graphs and out-of-memory or GC overhead limit exceeded errors.

Workaround: Increase the default memory for OpenTSDB by making the following changes on all OpenTSDB nodes:

1. Edit the /opt/mapr/conf/conf.d/warden.opentsdb.conf file to change:

   service.heapsize.max=2000
   service.heapsize.min=2000

   to

   service.heapsize.max=6000
   service.heapsize.min=6000

2. Edit the /opt/mapr/opentsdb/opentsdb-*/etc/init.d/opentsdb file to change:

   $
   {JVMXMX:=-Xmx2000m -Xss1m -XX:MaxMetaspaceSize=128m}

   to

   $
   {JVMXMX:=-Xmx6000m -Xss1m -XX:MaxMetaspaceSize=128m}

3. Restart the OpenTSDB service:

   maprcli node services -name opentsdb -nodes <space-separated list of OpenTSDB nodes> -action restart

OTSDB-60

Issue: MEPs 6.0.x or earlier with MapR Installer 1.10 or earlier can generate numerous instances of the following streams error in the opentsdb.err file:

2018-06-22 21:33:55,5510 ERROR StreamsClient fs/client/marlin/cc/marlinclient.cc:402 Thread: 2958 /var/mapr/mapr.monitoring/streams/0 does not exist.

The error might be valid for users upgrading from a previous release, but is generated even when users have not upgraded.

Workaround: To stop the error from being generated, use the following workaround:

1. Create the /var/mapr/mapr.monitoring/streams directory:

   hadoop fs -mkdir /var/mapr/mapr.monitoring/streams
   

2. Use maprcli to create a stream that will prevent the error from being generated:

   maprcli stream create -path /var/mapr/mapr.monitoring/streams/0 -ttl 86400
   

ES-27

Issue: On a new installation of a secure cluster using the MapR Installer, Elasticsearch fails to start, and logs indicate that Elasticsearch key generation failed.

Workaround: Use the Incremental Install feature of the MapR Installer to retry installation of the MapR Monitoring logging components.

SPYG-916

Issue: On a non-secure cluster, the MapR Monitoring index fails to load because Kibana tried to load the template before Elasticsearch started.

Workaround: In MCS, click the Services tab, locate Kibana, and click the restart button to restart Kibana.

OPTSDB-25

Issue: Purge scripts are disabled by default because occasionally a purge process will not exit properly.

Workaround: Edit the crontab file in the /etc/crontab file in the OpenTSDB installation directory, as follows:

crontab -e mapr

In the file, remove the hashtag (#) at the beginning of the line that refers to opentsdb. You should manage and monitor the cronjob. Typically, the cronjob takes 30 - 45 minutes to run on an idle system, but on a busy system could take much longer. In most cases, it will not overlap with the next cronjob. Also, monitor the jobs to ensure that OpenTSDB is not started or stopped mid-job, which would cause a problem.

SPYG-934

Issue: On secure Ubuntu 14.04 or 16.04 clusters, Elasticsearch fails to generate a keystore password if the uuid-runtime package is not installed. uuid-runtime is one of the MapR Installer prerequisites.

Workaround: Install the uuid-runtime package from the default Ubuntu repository:

apt-get install uuid-runtime

KIB-12

Issue: Kibana versions prior to 5.6.1 have a cross-site scripting (XSS) vulnerability in Timelion (CVE‌-2017-11479). This is NOT a MapR vulnerability as MapR does not support (or recommend) Timelion.

MapR PACC

See MapR PACC Known Issues.

Oozie

29634

Issue: On a secure cluster, Oozie jobs can fail when the timeline server for the Hive-on-Tez user interface is configured with MapR-SASL.

Workaround: Restart the resource manager and timeline server. For more information, see Configuring the Timeline Server to Use the Hive-on-Tez User Interface.

29657

Issue: If TimelineServer and Oozie Bootstrap exist on your cluster, the following error message appears exactly every 10 seconds:

Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected" in at "resourcemenager.out log

This issue is caused by Jersey 1.9 minor bug.

Workaround: To suppress the message and keep the logs clean, disable logging of WadlGeneratorJAXBGrammarGenerator class in JRE_HOME/lib/logging.properties in the following way:

com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.level = OFF

YARN

MAPRYARN-185

YARN cgroups are not supported on RHEL 7.x. In RHEL 7.x, libcgroup is deprecated and systemd maintains cgroups. Moving default hierarchies that systemd manages can generate unexpected results.

PJDTest Suite Compliance

28912

Issue: When trying to open a FIFO on a FUSE mounted filesystem, permissions to perform the operation are not checked.

29234

Issue: The FUSE-based POSIX can only trace at millisecond level and the test suite utimensat in pjdtestsuite fails because the test suite sets time in nanoseconds.

29237

Issue: Any user can set time using touch -t for any file on a FUSE mounted filesystem.

Timeline Server

See Hive-on-Tez User Interface Known Issues.