YARN Container Resources

Provides an overview of YARN.

A YARN application can be a MapReduce version 2 (MRv2) application or a non-MapReduce application. The Warden on each node calculates the resources that can be allocated to process YARN applications. Each application has an ApplicationMaster that negotiates YARN container resources. For MapReduce applications, YARN processes each map or reduce task in a container. The ApplicationMaster requests resources from the ResourceManager based on memory, CPU, and disk requirements for the YARN containers. For YARN containers that process MRv2 tasks, there are additional considerations. See YARN Container Resources for MapReduce Version 2 Applications for details.

The ApplicationMaster requests YARN container resources based on the values of the following parameters:
yarn.scheduler.minimum-allocation-mb
Default: 1024
Description: Defines the minimum memory allocation available for a container in MB.

To change the value, edit the yarn-site.xml file for the node that runs the ResourceManager. Assign the new value to this property, then restart the ResourceManager.

yarn.scheduler.maximum-allocation-mb
Default: 8192
Description: Defines the maximum memory allocation available for a container in MB.

To change the value, edit the yarn-site.xml file for the node that runs the ResourceManager. Assign the new value to this property, then restart the ResourceManager.

yarn.nodemanager.resource.memory-mb
Default: Variable. This value is calculated by Warden.
Description: Defines the memory available to processing Yarn containers on the node in MB.

Warden uses the following formula to calculate this value: [total physical memory on node] - [memory required by the operating system, file system, and HPE Ezmeral Data Fabric services installed on the node].

To determine the value, go to the ResourceManager UI and view the memory available for that node.

yarn.nodemanager.resource.cpu-vcores
Default: Variable. This value is calculated by Warden.
Description: Defines the number of CPUs available to process YARN containers on this node.

Warden uses the following formula to calculate this value: [Number of CPU cores on node] - [Number of CPU cores assigned to file system].

To determine the value, go to the ResourceManager UI or the YARN pane on the Control System and view the number of CPUs available for that node.

To change the value, edit the yarn-site.xml file for the node, assign the new value to this property, then restart the NodeManager.

yarn.nodemanager.resource.io-spindles
Default: Variable. This value is calculated by Warden.
Description: Defines the number of disks available to process YARN containers. Warden uses the following formula to calculate this value: [Number of of disks on the node].

To determine the value, go to the ResourceManager UI or the YARN pane on the Control System and view the disk information for this node.