Configure Spark JAR Location
About this task
Procedure
-
Create a zip archive containing all the JARs from the
SPARK_HOME/jars
directory. For example:cd /opt/mapr/spark/spark-<version>/jars/ zip /opt/mapr/spark/spark-<version>/spark-jars.zip ./*
-
Copy the zip file from the local file system to a world-readable location on
MapR-FS. You can upload it to the home of the
current user:
hadoop fs -put /opt/mapr/spark/spark-<version>/spark-jars.zip
For example:
hadoop fs -put /opt/mapr/spark/spark-2.0.1/spark-jars.zip /user/mapr/
-
Set the spark.yarn.archive property in the spark-defaults.conf file to point to
the world-readable location where you added the zip file. Apply this setting on
the node where you will be submitting your Spark jobs.
spark.yarn.archive maprfs:///<path to zip>
For example:
spark.yarn.archive maprfs:///user/mapr/spark-jars.zip