Configuring Multiple YARN Clusters
This topic describes how to set up multiple Myriad frameworks and multiple YARN clusters in the Mesos environment.
About this task
Multiple YARN clusters are created in the Mesos environment when multiple Myriad
frameworks are each running a Resource Manager. The Resource Manager spawns Node
Managers and Job History Service processes which form a cluster. To create multiple
YARN clusters, a Resource Manager is run under the Myriad framework with the cluster
name specified. The cluster name differentiates between staging, system, and local
volume MapReduce directories. When configuring for multiple YARN clusters, run
configure.sh
on the nodes where you want the 1st Myriad
framework and YARN cluster, then run configure.sh
with different
parameters on the nodes where you want the 2nd Myriad framework and YARN cluster,
and so on.
Procedure
-
Run
configure.sh
on the slave nodes for the YARN cluster. Specify a Resource Manager hostname, Job History name, Myriad framework, and cluster prefix name.NOTE It is recommended that you use the same value for theFor example, if you set the parameter value to framework1 for each of the parameters, the-RM, -HS, -MF, and -MCL
parameters.configure.sh
parameters could be similar to the following:/opt/mapr/server/configure.sh \ -C 10.10.100.16 \ -Z 10.10.100.16 -u mapr -g mapr \ -N myCluster \ -F /root/disk.list \ -RM framework1.marathon.mesos \ -HS jobhistory.framework1.mesos \ -MF framework1 \ -MCL framework1
-
Run
configure.sh
on the slave nodes for each new YARN cluster. Specify new values for Resource Manager hostname, Job History name, Myriad framework, and cluster prefix name.For example, on the subsequent YARN cluster, if you set the parameter value to framework2 for each of the parameters, theconfigure.sh
parameters could be similar to the following:/opt/mapr/server/configure.sh \ -C 10.10.100.16 \ -Z 10.10.100.16 -u mapr -g mapr \ -N myCluster \ -F /root/disk.list \ -RM framework2.marathon.mesos \ -HS jobhistory.framework2.mesos \ -MF framework2 \ -MCL framework2
Example
For example, the following creates three Myriad frameworks and the associated YARN cluster name prefix (framework1, framework2, and framework3). In the following examples, -N is set to myCluster, the primary IP address is 10.10.100.16, and the values for the -RM, -HS, -MF, and -MCL parameters change for each YARN cluster.
// On slave nodes 1 - 3, run:
/opt/mapr/server/configure.sh \
-C 10.10.100.16 \
-Z 10.10.100.16 -u mapr -g mapr \
-N myCluster \
-F /root/disk.list \
-RM framework1.marathon.mesos \
-HS jobhistory.framework1.mesos \
-MF framework1 \
-MCL framework1
Create the 2nd cluster for framework2:
// On slave nodes 4 - 6, run:
/opt/mapr/server/configure.sh \
-C 10.10.100.16 \
-Z 10.10.100.16 -u mapr -g mapr \
-N myCluster \
-F /root/disk.list \
-RM framework2.marathon.mesos \
-HS jobhistory.framework2.mesos \
-MF framework2 \
-MCL framework2
Create the 3rd cluster for framework3:
// On slave nodes 7 - 9, run:
/opt/mapr/server/configure.sh \
-C 10.10.100.16 \
-Z 10.10.100.16 -u mapr -g mapr \
-N myCluster \
-F /root/disk.list \
-RM framework3.marathon.mesos \
-HS jobhistory.framework3.mesos \
-MF framework3 \
-MCL framework3