Configure Authentication for Spark on YARN
When authentication is enabled, authentication keys are randomly generated for each job.
Prerequisites
NOTE Starting in EEP 4.0, for secure clusters, you can skip this step. For new
installs done through the 6.0 MapR Installer, the installer enables this
configuration. For manual installs and upgrades, running configure.sh
-R, as the final step in the configuration process, enables these
settings.
About this task
Complete the following step to manually configure authentication on a non-secure cluster or in versions earlier than EEP 4.0:
Procedure
-
Configure the following property in the
spark-defaults.conf
file on each Spark node:spark.authenticate true
The
spark-defaults.conf
file is in the following location:/opt/mapr/spark/spark-<version>/conf/
-
If you have configured an external shuffle service, you must also add the
following property to
yarn-site.xml
on each Spark node:<property> <name>spark.authenticate</name> <value>true</value> </property>