HPE Ezmeral Data Fabric 6.1.x is In Maintenance and transitions to "End of Maintenance" in June 2024. Please see the latest documentation.

About MapR 6.1
This site contains the main documentation for Version 6.1 of the MapR Converged Data Platform, including installation, configuration, administration, and reference information.
6.1 Installation
This section contains information about installing and upgrading MapR software. It also contains information about how to migrate data and applications from an Apache Hadoop cluster to a MapR cluster.
6.1 MapR Data Platform
MapR Data Platform is the industry-leading data platform for AI and analytics that solves enterprise business needs.
6.1 Administration
This section describes how to manage the nodes and services that make up a cluster.
- Administering Users and Clusters
  Lists topics that help manage a MapR cluster.
- Administering Nodes
  Provides a synopsis of managing nodes in a cluster.
- Administering Volumes
  This section provide information about how to organize and manage data using volumes, a unique feature of MapR clusters.
- Administering Files and Directories
- Administering Tables
  Administration of the MapR Database is done primarily via the command line (maprcli) or with the Managed Control System (MCS). Regardless of whether the MapR Database table is used for binary files or JSON documents, the same types of commands are used with slightly different parameter options. MapR Database administration is associated with tables, columns and column families, and table regions.
- Administering Streams
- Administering MapR Gateways
  A MapR gateway mediates one-way communication between a source MapR cluster and a destination cluster. You can replicate MapR Database tables (binary and JSON) and MapR Event Store For Apache Kafka streams. MapR gateways also apply updates from JSON tables to their secondary indexes and propagate Change Data Capture (CDC) logs.
- Administering Services
- Monitoring the Cluster
  This section describes how to monitor the health and performance of a MapR cluster.
  - Monitoring Using the Control System and the CLI
    Describes the Overview page in the Control System, which displays information about the cluster.
  - Using MapR Data Platform Monitoring (Spyglass Initiative)
    MapR Data Platform Monitoring (part of the Spyglass initiative) provides the ability to collect, store, and view metrics and logs for nodes, services, and jobs/applications.
    - MapR Monitoring Architecture
      MapR Monitoring integrates with open-source components to collect, aggregate, store, and visualize metrics and logs.
    - Metric Collection
      Metrics are collected from each node in the cluster so that administrators can use the data to monitor the cluster. In general, the collectd service collects metrics every 10 seconds. The exception is volume metrics which are collected every 10 minutes.
      - Configure Metric Retention
        By default, OpenTSDB stores two weeks of metrics. Based on your requirements, you can change metric retention period.
      - Configure Queue Filters for mapr.rm.<value> Metrics
        The YARN application metrics that are collected by JMX have the metric name syntax mapr.rm.<metric_name> and the metric values are aggregated among all the queues in the default queue. However, you can configure collectd to create a filter for each queue. As an alternative, you can use the REST API queue metrics (mapr.rm_queue.<metric_name>) which are by default set up for filtering by queue.
      - Configure the Collectd Service Heap Size
        The collectd service uses an embedded JVM when it gathers metrics from the CLDB, Node Manager, Resource Manager, and Drill. You can edit the Plugin Java section of collectd.conf to configure limits to the collectd virtual memory footprint.
      - CPU Metrics
        Every 10 seconds, the collectd service uses the cpu plugin to gather the following CPU metrics on each node in the cluster.
      - Disk Free Metrics
        Every 10 seconds, the collectd service uses the df plugin to gather the following disk free metrics on each node in the cluster.
      - Disk Metrics
        Every 10 seconds, the collectd service uses the disk plugin to gather the following disk metrics on each node in the cluster.
      - Drill Metrics
        Every 10 seconds, the collectd service uses the plugin to gather the following Drill metrics on each node in the cluster.
      - Hive JMX Metrics
        Every 10 seconds, the collectd service uses the Hive plug-in to gather the following Hive JMX metrics on each node in the cluster. Descriptions for the Hive metrics are not currently available.
      - Load Metrics
        Every 10 seconds, the collectd service uses the load plugin to gather the following load metrics on each node in the cluster.
      - MapR Alarm Metrics
        Every 10 seconds, the collectd service uses a plugin to gather the cluster alarms.
      - MapR Cache Metrics
        Every 10 seconds, the collectd service uses a plugin to gather the following data-fabric file system cache metrics on each node in the cluster.
      - MapR CLDB Metrics
        Every 10 seconds, the collectd service uses a MapR plugin to gather the following CLDB metrics on the primary CLDB node in the cluster.
      - MapR Database Metrics
        Every 10 seconds, the collectd service uses a plugin to gather MapR Database metrics on each node in the cluster. MapR Database provides both node and table metrics.
      - MapR Event Store For Apache Kafka Metrics
        Every 10 seconds, the collectd service uses a plugin to gather the following Streams metrics on each node in the cluster.
      - MapR File System Metrics
        Every 10 seconds, the collectd service uses a MapR plugin to gather the following MapR File System metrics on each node in the cluster.
      - MapR Process Metrics
        Every 10 seconds, the collectd service uses a plugin to gather the following MapR process metrics on each node in the cluster.
      - MapR I/O Metrics
        Every 10 seconds, the collectd service uses a plugin to gather the following MapR I/O metrics on each node in the cluster.
      - MapR RPC Metric
        Every 10 seconds, the collectd service uses a plugin to gather the following MapR RPC metrics on each node in the cluster.
      - Spark JMX Metrics
        Every 10 seconds, the collectd service gathers the following Spark JMX metrics on each node in the cluster.
      - MapR Topology Metrics
        Every 60 seconds, the collectd service uses a plugin to gather the following topology metrics on each node in the cluster. Use these metrics to understand disk utilization across a topology or rack. By default, these metrics include all racks and topologies associated with the cluster. However, you can use tags to specify which rack(s) or topologies(s) to include. Note: Racks and topologies can span multiple nodes and one rack can be associated with multiple topologies.
      - MapR Volume Metrics
        Every 10 seconds, the collectd service uses a plugin to gather the following MapR volume metrics on each CLDB node in the cluster.
      - Virtual Memory Metrics
        Every 10 seconds, the collectd service uses the vmem plugin to gather the following memory metrics on each node in the cluster.
      - Memory Metrics
        Every 10 seconds, the collectd service uses the memory and swap plugins to gather the following memory metrics on each node in the cluster.
      - Network Metrics
        Every 10 seconds, the collectd service uses the interface plugin to gather network metrics on each node in the cluster.
      - Node Manager Metrics
        Every 10 seconds, the collectd service uses a plugin to gather the following Node Manager metrics on each node in the cluster.
      - Resource Manager Metrics
        Every 10 seconds, the collectd service uses a MapR plugin to gather Resource Manager metrics on the active Resource Manager. Collectd gathers metrics on the Resource Manager JVM process, YARN applications, and nodes that are managed by the Resource Manager. The method used to gather the metrics differs based on the metric type.
    - Configure the OpenTSDB Service Heap Size
      By default, the OpenTSDB service is configured to use a default heap size of 6 gigabytes. The default heap size can be adjusted by modifying certain configuration files.
    - Metric Visualization
      Use dashboards to visualize metrics across multiple nodes and clusters.
    - Log Collection
      Fluentd collects log events from each node in the cluster and stores them in a centralized location so that administrators can search the logs when troubleshooting issues in the cluster. The process that fluentd uses to parse and send log events to Elasticsearch differs based on the formatting of log events in each log file.
    - Log Aggregation and Storage
      Fluentd uses a round-robin approach when writing logs to Elasticsearch nodes. If an Elasticsearch node in unavailable, Fluentd can fail over log storage to another Elasticsearch node.
    - Log Visualization
      Use dashboards to visualize the logs across multiple nodes and clusters.
    - MapR Monitoring Tips and Troubleshooting
      Lists the nuances of monitoring clusters.
    - Reconfiguring MapR Monitoring
      Changes to an existing cluster, such as the addition of services, may require additional steps to enable the collection of metrics and logs.
- Configuring Security
  Describes how to configure security and manage secure clusters.
- Managing Secure Clusters
  Provides procedures that will enable you to use MapR clusters securely.
- Administering the MapR Data Access Gateway
  The MapR Data Access Gateway is a service that acts as a proxy and gateway for translating requests between lightweight client applications and the MapR cluster. This section describes considerations when upgrading the service, how to modify configuration settings, and how to administer and manage the service.
- Planning for High Availability
- Administrator's Reference
  This section contains in-depth reference information for the administrator.
- Troubleshooting Cluster Administration
  Lists the common errors and their solutions.
- Best Practices for Backing Up MapR Information
  Lists the best practices and performance considerations to follow when backing up MapR information.
6.1 Development
This section contains information related to application development for Ezmeral ecosystem components and MapR Data Platform products, including the file system, Database (Key-Value and JSON), and Event Streams.
Other Docs
This section contains release-independent information, including: MapR Installer documentation, Ecosystem release notes, interoperability matrices, security vulnerabilities, and links to other MapR version documentation.
Glossary
Definitions for commonly used terms in MapR Converged Data Platform environments.

Configure Metric Retention

By default, OpenTSDB stores two weeks of metrics. Based on your requirements, you can change metric retention period.

The following cron job runs each day to purge metrics based on the retention period.

$min $hour * * * $OTSDB_HOME/bin/tsdb_cluster_mgmt.sh -purgeData >> $OTSDB_HOME/var/log/opentsdb/purgeData.log 2>&1

Complete the following steps to edit the metric retention period:

Open the /opt/mapr/opentsdb/opentsdb-<version>/bin/tsdb_cluster_mgmt.sh file.
In the following line, update the value of '2 weeks ago' to the new retention period.
```
$OT_HOME/bin/tsdb scan --delete 2000/01/01 $(date --date='2 weeks ago' +'%Y/%m/%d') sum $metric
```
For example, to delete metrics from 1/1/2000 to [current date - 2 days]:
```
$OT_HOME/bin/tsdb scan --delete 2000/01/01 $(date --date='2 days ago' +'%Y/%m/%d') sum $metric
```

WARNING MapR monitoring uses 2 MB disk space per minute per node when MapR Event Store For Apache Kafka metrics is enabled. This is approximately 3 GB per day on a single node or 7 GB per node per day with a 3X replication. This stream metrics data is automatically deleted every 30 days.

NOTE For more information, see the OpenTSDB scan command documentation.