Resource Manager Metrics

Every 10 seconds, the collectd service uses a HPE Ezmeral Data Fabric plugin to gather Resource Manager metrics on the active Resource Manager. Collectd gathers metrics on the Resource Manager JVM process, YARN applications, and nodes that are managed by the Resource Manager. The method used to gather the metrics differs based on the metric type.

YARN Application Metrics

Collectd gathers YARN application metrics via JMX and REST API. The application metrics that are collected by JMX have the metric name mapr.rm.<metric_name>. Application metrics collected via REST API have the metric name mapr.rm_queue.<metric_name>.

Metrics Collected Using JMX

The following metrics are collected using JMX. To filter these metrics by queue using the rm_queue tag, see Configure Queue Filters for mapr.rm.<value> Metrics.

mapr.rm.active_applications: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of active applications.
mapr.rm.active_users: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of users with active applications.
mapr.rm.aggregate_containers_allocated: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of allocated containers.
mapr.rm.aggregate_containers_released: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of released containers.

mapr.rm.allocated_MB: Additional Tags: rm_queue: Display values for a specified queue.; Description: The amount of memory allocated to the Resource Manager in MB.

mapr.rm.allocated_vcores: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of CPUs allocated to the Resource Manager.

mapr.rm.apps_completed: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of completed applications.

mapr.rm.apps_failed: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of failed applications.

mapr.rm.apps_killed: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of killed applications.

mapr.rm.apps_pending: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of pending applications.

mapr.rm.apps_running: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of running applications.

mapr.rm.apps_submitted: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of submitted applications.

mapr.rm.available_MB: Additional Tags: rm_queue: Display values for a specified queue.; Description: The amount of memory available to the Resource Manager in MB.

mapr.rm.available_disks: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of disks available to the Resource Manager.

mapr.rm.available_vcores: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of CPUs available to the Resource Manager.

mapr.rm.pending_MB: Additional Tags: rm_queue: Display values for a specified queue.; Description: The amount of memory, in MB, waiting to be allocated by the Resource Manager.

mapr.rm.pending_containers: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of containers waiting to be allocated by the Resource Manager.

mapr.rm.pending_disks: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of disks waiting to be allocated by the Resource Manager.

mapr.rm.pending_vcores: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of CPUs waiting to be allocated by the Resource Manager.

mapr.rm.reserved_MB: Additional Tags: rm_queue: Display values for a specified queue.; Description: The amount of memory reserved by the Resource Manager in MB.

mapr.rm.reserved_containers: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of containers reserved by the Resource Manager.

mapr.rm.reserved_disks: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of disks reserved by the Resource Manager.

mapr.rm.reserved_vcores: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of CPUs reserved by the Resource Manager.

Metrics Collected Using REST API

The following YARN application metrics are collected using REST API.

mapr.rm_queue.aggregate_containers_allocated: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of containers allocated for applications in the default and custom queues.
mapr.rm_queue.appmaster_used_disks: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of disks used by the Application Master for applications in the default and custom queues.

mapr.rm_queue.appmaster_used_memory: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the amount of memory, in MB, used by the Application Master for applications in the default and custom queues.

mapr.rm_queue.appmaster_used_vcores: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of CPUs used by the Application Master for applications in the default and custom queues.

mapr.rm_queue.apps_pending: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of pending applications in the default and custom queues.

mapr.rm_queue.apps_running: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of applications running in the default and custom queues.

mapr.rm_queue.fairshare_disks: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Fair Scheduler, this parameter is the number of disks allocated to default and custom queues.

mapr.rm_queue.fairshare_memory: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Fair Scheduler, this parameter denotes the amount of memory, in MB, allocated to default and custom queues.

mapr.rm_queue.fairshare_vcores: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Fair Scheduler, this parameter denotes the number of CPUs used by applications in the default and custom queues.

mapr.rm_queue.used_disks: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of disks used by applications in the default and custom queues.

mapr.rm_queue.used_memory: Additional Tags: rm_queue: Display values for a specified queue.; Description: The amount of memory, in MB, used by applications in the default and custom queues.

mapr.rm_queue.used_vcores: Additional Tags: rm_queue: Display values for a specified queue.; Description: The number of CPUs used by applications in the default and custom queues.

mapr.rm_queue.max_disks: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Fair Scheduler, this parameter denotes the maximum number of disks available to default and custom queues.

mapr.rm_queue.max_memory: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Fair Scheduler, this parameter denotes the maximum amount of memory, in MB, available to default and custom queues.

mapr.rm_queue.max_vcores: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Fair Scheduler, this parameter denotes the maximum number of CPUs available to default and custom queues.

mapr.rm_queue.user_allocated_disks

Additional Tags:

rm_queue: Display values for a specified queue.
rm_user: Display values for a specified user.

Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of disks allocated to the queues.

mapr.rm_queue.user_allocated_memory

Additional Tags:

rm_queue: Display values for a specified queue.
rm_user: Display values for a specified user.

Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the amount of memory, in MB, allocated to the queues.

mapr.rm_queue.user_allocated_vcores

Additional Tags:

rm_queue: Display values for a specified queue.
rm_user: Display values for a specified user.

Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of CPUs allocated to queues.

mapr.rm_queue.user_appmaster_used_disks: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of disks used by the queues.

mapr.rm_queue.appmaster_used_memory: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the amount of memory used by the queues.

mapr.rm_queue.appmaster_used_vcores: Additional Tags: rm_queue: Display values for a specified queue.; Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of CPUs used by the queues.

mapr.rm_queue.user_apps_pending

Additional Tags:

rm_queue: Display values for a specified queue.
rm_user: Display values for a specified user.

Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of applications pending in the queues.

mapr.rm_queue.user_apps_running

Additional Tags:

rm_queue: Display values for a specified queue.
rm_user: Display values for a specified user.

Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of applications running in the queues.

mapr.rm_queue.user_used_disks

Additional Tags:

rm_queue: Display values for a specified queue.
rm_user: Display values for a specified user.

Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of number of disks used by the queues.

mapr.rm_queue.user_used_memory

Additional Tags:

rm_queue: Display values for a specified queue.
rm_user: Display values for a specified user.

Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the amount of memory, in MB, used by the queues.

mapr.rm_queue.user_used_vcores

Additional Tags:

rm_queue: Display values for a specified queue.
rm_user: Display values for a specified user.

Description: When queue resources are managed by the Capacity Scheduler, this parameter denotes the number of CPUs used by the queues.

Resource Manager Node Metrics

The following are the Node metrics:

mapr.rm_cluster.active_nodes: The number of nodes in the cluster where containers are running.

mapr.rm_cluster.total_nodes: The number of nodes in the cluster.

mapr.rm_cluster.unhealthy_nodes: The number of nodes in the cluster that are unable to accept applications.

Resource Manager JVM Metrics

The following Resource Manager metrics are collected using JMX:

mapr.rm.jvm.gc_count: The number of garbage collections.

mapr.rm.jvm.gc_count_ps_mark_sweep: The number of parallel scavenge mark sweep collections.

mapr.rm.jvm.gc_count_ps_scavenge: The number of parallel scavenge collections.

mapr.rm.jvm.gc_time_millis: The amount of time, in milliseconds, spent on garbage collection.

mapr.rm.jvm.gc_time_millis_ps_mark_sweep: The amount of time, in milliseconds, spent on parallel scavenge mark sweep collection.

mapr.rm.jvm.gc_time_millis_ps_scavenge: The amount of time, in milliseconds, spent on parallel scavenge collection.

mapr.rm.jvm.log_error: The total number of ERROR logs.

mapr.rm.jvm.log_fatal: The total number of FATAL logs.

mapr.rm.jvm.log_info: The total number of INFO logs.

mapr.rm.jvm.log_warn: The total number of WARN logs.

mapr.rm.jvm.mem_heap_committed_m: The amount of heap memory, in megabytes, committed to the Resource Manager.

mapr.rm.jvm.mem_heap_max_m: The maximum amount of heap memory, in megabytes, that can be committed to the Resource Manager.

mapr.rm.jvm.mem_heap_used_m: The amount of heap memory, in megabytes, used by the Resource Manager.

mapr.rm.jvm.mem_max_m: The maximum amount of memory, in megabytes, that can be committed to the Resource Manager.

mapr.rm.jvm.mem_non_heap_committed_m: The amount of non-heap memory, in megabytes, committed to the Resource Manager.

mapr.rm.jvm.mem_non_heap_max_m: The maximum amount of non-heap memory, in megabytes, that can be committed to the Resource Manager.

mapr.rm.jvm.mem_non_heap_used_m: The maximum amount of non-heap memory, in megabytes, that can be used by the Resource Manager.

mapr.rm.jvm.threads_blocked: The number of Resource Manager threads in BLOCKED state.

mapr.rm.jvm.threads_new: The number of Resource Manager threads in NEW state.

mapr.rm.jvm.threads_runnable: The number of Resource Manager threads in RUNNABLE state.

mapr.rm.jvm.threads_terminated: The number of Resource Manager threads in TERMINATED state.

mapr.rm.jvm.threads_time_waiting: The number of Resource Manager threads in TIMED_WAITING state.

mapr.rm.jvm.threads_waiting: The number of Resource Manager threads in WAITING state.