Step 9: Install Log Monitoring

Installing the MapR monitoring logging components is optional. The logging components enable the collection, storage, and visualization of MapR core logs, system logs, and ecosystem component logs. MapR Monitoring components are available as part of the Ezmeral Ecosystem Pack (EEP) that you selected for the cluster.

About this task

NOTE As of release 6.0, if you install the logging components on a secure cluster, you must generate the Elasticsearch keys, certifications, and trust stores on an Elasticsearch node (designated as the primary) and then distribute them to the other Elasticsearch nodes in the cluster, as shown below in step 3.

Complete the steps to install the logging components as the root user or using sudo. Installing logging components on a client node or edge node is not supported.

Procedure

  1. For log monitoring, install the following packages:
    Component Requirements
    fluentd Install the mapr-fluentd package on each node in the cluster.
    Elasticsearch Install the mapr-elasticsearch package on at least three nodes in the cluster to allow failover of log storage if one Elasticsearch node is unavailable.
    Kibana Install the mapr-kibana package on at least one node in the cluster.
    NOTE On secure Ubuntu 14.04 or 16.04 clusters, Elasticsearch can fail to generate a keystore password if the uuid-runtime package is not installed. uuid-runtime is one of the package dependencies required for installing without the MapR Installer. See the entry for SPYG-934 and uuid-runtime in the Known Issues at Release (MapR 6.1.0).
    For example, on a three-node MapR cluster, you can run the following commands to install log packages:
    • For CentOS/RedHat:
      • Node A: yum install mapr-fluentd mapr-elasticsearch
      • Node B: yum install mapr-fluentd mapr-elasticsearch
      • Node C: yum install mapr-fluentd mapr-elasticsearch mapr-kibana
    • For Ubuntu:
      • Node A: apt-get install mapr-fluentd mapr-elasticsearch
      • Node B: apt-get install mapr-fluentd mapr-elasticsearch
      • Node C: apt-get install mapr-fluentd mapr-elasticsearch mapr-kibana
    • For SLES:
      • Node A: zypper install mapr-fluentd mapr-elasticsearch
      • Node B: zypper install mapr-fluentd mapr-elasticsearch
      • Node C: zypper install mapr-fluentd mapr-elasticsearch mapr-kibana
  2. For secure MapR clusters, run maprlogin print to verify that you have a user ticket for the MapR user and the root user. These user tickets are required for a successful installation. If you need to generate a MapR user ticket, run maprlogin password. For more information, see Generating a MapR User Ticket.
  3. For secure MapR clusters, use the following steps to generate the Elasticsearch keys, certificates, and trust stores on the master Elasticsearch node, and then distribute them to other nodes:
    1. Select one Elasticsearch node as the master, and run configure.sh -R on that node.
    2. Run the following command on the master Elasticsearch node:
      /opt/mapr/server/configure.sh -OT <comma-separated list of OpenTSDB nodes> -ES <comma-separated list of Elasticsearch nodes> -R -EPelasticsearch -genESKeys

      For descriptions of the parameters used in this command, see the table in step 6.

      IMPORTANT If DNS resolution on the nodes is not completely configured, running the configure.sh -genESKeys step will fail. For example, key creation will fail if reverse lookups do not work. Suppose that the DNS lookup for a node with hostname node1.qa.lab returns 192.100.1.1. If the DNS lookup for 192.100.1.1 returns nothing (instead of hostname node1.qa.lab), key creation will fail.
    3. For Elasticsearch, copy the specified files to the locations indicated in the following table.
      File Permissions Copy from location (on master) Copy to location (all other ES nodes)
      es-root-ca.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/ca/es-root-ca.pem /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/ca
      admin-usr-private-key.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/admin-usr-private-key.pem /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/certs
      admin-usr-signed.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/admin-usr-signed.pem /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/certs
      admin-usr-clientCombo.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/admin-usr-clientCombo.pem /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/certs
      truststore.jks 0640 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/truststore.jks /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/keystores
      admin-usr-keystore.jks 0640 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/admin-usr-keystore.jks /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/keystores/admin-usr-keystore.jks
      <node-fqdn>-srvr-keystore.jks 0640 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/<node-fqdn>-srvr-keystore.jks /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/keystores
      NOTE Only include the file that matches the nodename.
      sg2.yml 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/sg2.yml /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/
      sg_http_<node-fqdn>.yml 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/sg_http_<node-fqdn>.yml /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch
      NOTE Only include the file that matches the nodename.
      sg_ssl_<node-fqdn>.yml 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/sg_ssl_<node-fqdn>.yml /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch
      NOTE Only include the file that matches the nodename.
      .keystore_password 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/.keystore_password /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/.keystore_password
      NOTE In the preceding table and the tables that follow, <ver> in a directory path represents the Elasticsearch, Kibana, or Fluentd version. These versions depend on the currently installed EEP. This table shows the versions for various EEPs:
      EEP Elasticsearch Version Kibana Version Fluentd Version
      6.3.0 6.5.3 6.5.3 1.4.0
      6.2.0 6.5.3 6.5.3 1.4.0
      6.1.1 6.5.3 6.5.3 1.3.2.1
      6.1.0 6.5.3 6.5.3 1.3.2.0
      6.0.x 6.2.3 6.2.3 1.1.2
      4.0.0 through 5.0.2 5.4.1 5.4.1 0.14.20
    4. For Kibana, copy the specified files to the locations indicated in the following table:
      NOTE If the /ca and /certs copy-to directories do not exist for Kibana, you must create them. In addition, the directories and all files within them must be owned by mapr:mapr, or the services will not start.
      File Permissions Copy from location (on master) Copy to location (all other Kibana nodes)
      es-root-ca.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/ca/es-root-ca.pem /opt/mapr/kibana/kibana-<ver>/config/ca
      kibanaserver-usr-clientCombo.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/kibanaserver-usr-clientCombo.pem /opt/mapr/kibana/kibana-<ver>/config/certs
      kibanaserver-usr-signed.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/kibanaserver-usr-signed.pem /opt/mapr/kibana/kibana-<ver>/config/certs
      kibanaserver-usr-private-key.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/kibanaserver-usr-private-key.pem /opt/mapr/kibana/kibana-<ver>/config/certs
    5. For Fluentd, copy the specified files to the locations indicated in the following table:
      NOTE If the /ca and /certs copy-to directories do not exist for Fluentd, you must create them. In addition, the directories and all files within them must be owned by mapr:mapr, or the services will not start.
      File Permissions Copy from location (on master) Copy to location (all other Fluentd nodes)
      es-root-ca.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/ca/es-root-ca.pem /opt/mapr/fluentd/fluentd-<ver>/etc/fluentd/ca/
      fluentd-usr-signed.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/fluentd-usr-signed.pem /opt/mapr/fluentd/fluentd-<ver>/etc/fluentd/certs
      fluentd-usr-private-key.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/fluentd-usr-private-key.pem /opt/mapr/fluentd/fluentd-<ver>/etc/fluentd/certs
      fluentd-usr-clientCombo.pem 0600 /opt/mapr/elasticsearch/elasticsearch-<ver>/etc/elasticsearch/sg/fluentd-usr-clientCombo.pem /opt/mapr/fluentd/fluentd-<ver>/etc/fluentd/certs
  4. MapR 6.0.1 and later: For secure MapR clusters, copy the $ES_HOME/etc/elasticsearch/.keystore_password file from the Elasticsearch master node to $KIBANA_HOME/config/.keystore_password on the nodes where you are running Kibana. The .keystore_password file contains a system-generated password that is used to authenticate Kibanaserver-to-Elasticsearch-server communication.
  5. Release 6.0.1 and later: For secure MapR clusters, configure a password for the Elasticsearch admin user to enable authentication for the end user using Kibana to search the Elasticsearch log index. This password needs to be provided at the time of running configure.sh. If no password is specified, you will default to the pre-mep-5.0.0, default password of admin. Use one of the following methods to pass the password to Elasticsearch/Kibana:
    • On the nodes where Fluentd/Elasticsearch/Kibana is installed, export the password as an environment variable before calling configure.sh:
      export ES_ADMIN_PASSWORD="<newElasticsearchPassword>"

      Then run configure.sh as you normally would run it (go to step 6).

    • Add the following options to the configure.sh command in step 6. This method explicitly passes the password on the configure.sh command line:
      -EPelasticsearch '-password <newElasticsearchPassword>' -EPkibana '-password <newElasticsearchPassword>' -EPfluentd '-password <newElasticsearchPassword>'
      Example
      /opt/mapr/server/configure.sh -R -v -ES mfs74.qa.lab -ESDB /opt/mapr/es_db -OT mfs74.qa.lab -C mfs74.qa.lab -Z mfs74.qa.lab -EPelasticsearch '-password helloMapR' -EPkibana '-password helloMapR' -EPfluentd '-password helloMapR'
    • Add the following options to the configure.sh command in step 6. This method explicitly passes the password on the configure.sh command line by specifying a file:
      -EPelasticsearch '-password <name of local file containing new password>'  -EPkibana '-password <name of local file containing new password>' -EPfluentd '-password <name of local file containing new password>'
      Example
      /opt/mapr/server/configure.sh -R -v -ES mfs74.qa.lab -ESDB /opt/mapr/es_db -OT mfs74.qa.lab -C mfs74.qa.lab -Z mfs74.qa.lab -EPelasticsearch '-password /tmp/es_password' -EPkibana '-password /tmp/es_password' -EPfluentd '-password /tmp/es_password'
  6. Run configure.sh on each node in the MapR cluster with the -R and -ES parameters, adding parameters to configure the Fluentd/Elasticsearch/Kibana password as needed. Optionally, you can include the -ESDB parameter to specify the location for writing index data. A Warden service must be running when you use configure.sh -R. 
    /opt/mapr/server/configure.sh -R -ES <comma-separate list of Elasticsearch nodes> [-ESDB <filepath>]
    Parameter Description
    -ES Specifies a comma-separated list of host names or IP addresses that identify the Elasticsearch nodes. The Elasticsearch nodes can be part of the current MapR cluster or part of a different MapR cluster. The list is in the following format:
    • hostname/IPaddress[:port_no] [,hostname/IPaddress[:port_no]...]
    NOTE The default Elasticsearch port is 9200. If you want to use a different port, specify the port number when you list the Elasticsearch nodes.
    -ESDB Specifies a non-default location for writing index data on Elasticsearch nodes. In order to configure an index location, you only need to include this parameter on Elasticsearch nodes. By default, the Elasticsearch index is written to /opt/mapr/elasticsearch/elasticsearch-<version>/var/lib/MaprMonitoring/.
    NOTE Elasticsearch requires a lot of disk space. Therefore, a separate filesystem for the index is strongly recommended. It is not recommended to store index data under the / or the /var file system.

    Upgrading to a new version of MapR monitoring removes the /opt/mapr/elasticsearch/elasticsearch-<version>/var/lib/MaprMonitoring/ directory. If you want to retain Elasticsearch index data through an upgrade, you must use the -ESDB parameter to specify a separate filesystem or back up the default directory before upgrading. The Pre-Upgrade Steps for MapR Monitoring include this step.
    -genESKeys Generates needed keys and certificates for the master Elasticsearch node in a secure cluster.
    -OT Specifies a comma-separated list of host names or IP addresses that identify the OpenTSDB nodes. The OpenTSDB nodes can be part of the current MapR cluster or part of a different MapR cluster. Do not use this option when you configure a node for the first time. Use this option along with the -R parameter. A Warden service must be running when you use configure.sh -R -OT.
    The hostname list should use the following format: 
    hostname/IP address[:port_no][,hostname/IP address[:port_no]...]
    NOTE The default OpenTSDB port is 4242. If you want to use a different port, specify the port number when you list the OpenTSDB nodes.
    -R After initial node configuration, specifies that configure.sh should use the previously configured ZooKeeper and CLDB nodes.
    For example, to configure MapR monitoring components you can run one of the following commands:
    • In this example, a location is specified for the Elasticsearch index directory, and default ports are used for Elasticsearch nodes:
      /opt/mapr/server/configure.sh -R -ES NodeA,NodeB,NodeC -ESDB /opt/mapr/myindexlocation
    • In this example, non-default ports are specified for Elasticsearch, and the default location is used for the Elasticsearch index directory:
      /opt/mapr/server/configure.sh -R -ES NodeA:9595,NodeB:9595,NodeC:9595
    After you run configure.sh -R, if errors are displayed see Troubleshoot MapR Monitoring Installation Errors.
  7. If you installed Kibana, perform the following steps:
    1. Use one of the following methods to load the Kibana URL:
      • From the Control System, select the Kibana view. After you select the Kibana view, you may also need to select the Pop-out page into a tab option.
      • From a web browser, launch the following URL: https://<IPaddressOfKibanaNode>:5601
    2. When the Kibana page loads, it displays a Configure an index pattern screen. Provide the following values:
      NOTE The Index contains time-based events option is selected by default and should remain selected.
      Field Value
      Index name or pattern mapr_monitoring-*
      Time-field @timestamp
    3. Click Create.