Pre-Upgrade Steps for Monitoring

Complete the following steps before you upgrade Monitoring Components with or without the Installer.

About this task

During an upgrade using the Installer, a script backs up many of the configuration files. However, whether or not you are upgrading manually or by using the Installer, it is a best practice to back up the files manually. Manual backups can help in case an error occurs or the specific file you customized is not automatically backed up by the script.
Before performing the pre-upgrade steps, note these important considerations:
  • The Monitoring upgrade is an offline upgrade and not a rolling upgrade.
  • This upgrade procedure is customized for the Data Fabric implementation of the monitoring components. Because the Data Fabric implementation has a narrow focus and there are numerous components, the upgrade steps are simplified. Data Fabric upgrade documentation does not include all of the upgrade steps that are included in the vendor documentation for each component. Before starting the upgrade process, consider familiarizing yourself with the vendor-upgrade steps to determine if your environment requires extra measures to protect data and configurations.
  • This upgrade sequence does not implement security in the Monitoring components. If the cluster you are upgrading is secure and you are upgrading to a new version of Elasticsearch, the security keys will be deleted when you upgrade the monitoring packages. You must regenerate the keys and copy them to the appropriate nodes after upgrading. The Post-Upgrade Steps for MapR Monitoring provide links to the installation procedures containing this information.

Procedure

  1. Before backing up configuration files, ensure that your Elasticsearch and Kibana indexes are not affected by the upgrade:
    NOTE This step assumes that log monitoring is configured. You can skip this step if your cluster is not configured for log monitoring.
    1. If you are using Elasticsearch version 2.x, upgrade your Elasticsearch index to version 6. For upgrade information, see: https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
      You need to upgrade your Elasticsearch index if your cluster is running a EEP in the range 1.1 through 3.0.x. See the following table. EEPs 1.1 through 3.0.x use Elasticsearch version 2.3.3. If your cluster is running a EEP in the range 4.0.0 through 5.0.x, you are using Elasticsearch 5.4.1, and you do NOT need to upgrade the index. For more information about Elasticsearch / Search Guard version information, see this website.

      Core EEP Elasticsearch Version SearchGuard Version Kibana Plugin Version
      6.1.0 6.1.0 6.5.3.0 24.0 17
      6.1.0 6.0.x 6.2.3 23.0 14
      6.0.x 4.0.0 through 5.0.2 5.4.1 N/A N/A
      5.2.x 3.0.5 and earlier 2.3.3 N/A N/A
      For more information about the Monitoring component versions included in each EEP, see Component Versions for Released EEPs.
    2. Create a snapshot of the Kibana index to capture index information before the upgrade. This information will be restored after the upgrade. For snapshot information, see https://www.elastic.co/guide/en/elasticsearch/reference/5.6/modules-snapshots.html.
  2. Before you upgrade metric monitoring components, create a backup of the configuration files to a location outside your installation directory. The following configuration file-lists include files that are commonly used for configuration and may not include every file that you may have customized.
    • Collectd configuration files:
      • /opt/mapr/conf/conf.d/warden.collectd.conf
      • /opt/mapr/collectd/collectd-<version>/etc/collectd.conf
      • /etc/logrotate.d/collectd
    • Grafana configuration files:
      • /opt/mapr/conf/conf.d/warden.grafana.conf
      • /opt/mapr/grafana/grafana-<version>/etc/grafana/grafana.ini
      • /opt/mapr/grafana/grafana- <version>/etc/grafana/ldap.toml
    • OpenTSDB configuration files:
      • /opt/mapr/conf/conf.d/warden.opentsdb.conf
      • /opt/mapr/opentsdb/opentsdb-<version>/etc/opentsdb/opentsdb.conf
      • /opt/mapr/opentsdb/opentsdb-<version>/etc/opentsdb/logback.xml
      • opt/mapr/opentsdb/opentsdb-<version>/bin/tsdb_cluster_mgmt.sh (This file is not automatically backed up.)
  3. Before you upgrade log monitoring components, create a backup of the following files to a location outside your installation directory. The following configuration file lists include files that are commonly used for configuration and may not include every file that you may have customized.
    • Kibana configuration files:
      • /opt/mapr/conf/conf.d/warden.kibana.conf
      • /opt/mapr/kibana/kibana-<version>/etc/conf/kibana.js
    • fluentd configuration files:
      • /opt/mapr/conf/conf.d/warden.fluentd.conf
      • /opt/mapr/fluentd/fluentd-<version>/etc/fluentd/fluentd.conf
      • /opt/mapr/fluentd/fluentd-<version>/etc/fluentd/es_config.conf
      • /opt/mapr/fluentd/fluentd-<version>/etc/fluentd/maprfs_config.conf
      • /opt/mapr/fluentd/fluentd-<version>/etc/fluentd/grok-patterns
      • /etc/logrotate/fluentd
    • Elasticsearch configuration files:
      • /opt/mapr/conf/conf.d/warden.elasticsearch.conf
      • /opt/mapr/elasticsearch/elasticsearch-<version>/etc/elasticsearch/elasticsearch.yml
      • /opt/mapr/elasticsearch/elasticsearch-<version>/etc/elasticsearch/logging.yml
      • /opt/mapr/elasticsearch/elasticsearch-<version>/etc/elasticsearch/curator.yml
      • /opt/mapr/elasticsearch/elasticsearch-<version>/etc/elasticsearch/curator_actions/delete_indices.yml (This file is not automatically backed up.)
      • /opt/mapr/elasticsearch/elasticsearch-<version>/var/lib/MaprMonitoring/ (This directory is the default location for Elasticsearch index data. You must back up this directory unless you specified a non-default location using the -ESDB parameter with configure.sh during installation.)
  4. Stop all monitoring services on the cluster.
    1. To stop collectd, run the following command: 
      maprcli node services -name collectd -nodes <space separated list of hostname/IPaddresses> -action stop
    2. To stop Grafana, run the following command: 
      maprcli node services -name grafana -nodes <space separated list of hostname/IPaddresses> -action stop
    3. To stop OpenTSDB, run the following command: 
      maprcli node services -name opentsdb -nodes <space separated list of hostname/IPaddresses> -action stop
    4. To stop Kibana, run the following command: 
      maprcli node services -name kibana -nodes <space separated list of hostname/IPaddresses> -action stop
    5. To stop fluentd, run the following command: 
      maprcli node services -name fluentd -nodes <space separated list of hostname/IPaddresses> -action stop
    6. To stop Elasticsearch, run the following command: 
      maprcli node services -name elasticsearch -nodes <space separated list of hostname/IPaddresses> -action stop