Understanding Zeppelin Docker Parameters

Connection Port

docker run -it \
    ... 
    -e ZEPPELIN_SSL_PORT=<port-number> \
    -p <port-number>:<port-number> \ 
    ...

Bridge Networking

By default, Docker uses bridge networking. In general, bridge networking provides better isolation from the host machine and other containers.

docker run -it \
    ... 
    -e HOST_IP=<docker-host-ip> \
    -p 10000-10010:10000-10010 \ 
    -p 11000-11010:11000-11010 \ 
    ...

The <docker-host-ip> must be an actual IP address. If you are running the container on your laptop, you cannot specify localhost as the IP address.

Specifying the 10000-10010 port range reserves the range for the Livy launcher. If you are already using these ports for other reasons, use the LIVY_RSC_PORT_RANGE environment variable to specify a different range.

If you plan to use Spark interpreter, you must reserve the 11000-11010 port range for Spark. To reserve a different port range, use the SPARK_PORT_RANGE environment variable.

docker run -it \
    ... 
    -e HOST_IP=<docker-host-ip> \
    -p 10011-10021:10011-10021 \
    -e LIVY_RSC_PORT_RANGE="10011~10021" \
    -p 13011-13021:13011-13021 \
    -e SPARK_PORT_RANGE="13011~13021" \ 
    ...

docker run -it \
    ... 
    --network=host \
    -e HOST_IP=<docker-host-ip> \
    ...

See https://docs.docker.com/engine/userguide/networking/ for more details about Docker networking.

MapR Clusters

docker run -it \
    ... 
    -e MAPR_CLUSTER=my.cluster.com \
    -e MAPR_CLDB_HOSTS=172.24.11.84,172.24.8.72,172.24.9.248 \
    ...

MapR Ticketing

If your MapR cluster is secure, you need a copy of the MapR ticket on your local host so you can specify a mount point in your docker run command. This makes the ticket visible to the Zeppelin container. The sample command shown earlier uses MapR tickets.

To determine whether your cluster is secure, view the contents of the file /opt/mapr/conf/mapr-clusters.conf on your MapR cluster. For example, the following shows a secure cluster:

my.cluster.com secure=true ip-172-24-11-84

If your cluster is secure, follow these steps to make the ticket visible to the Zeppelin container:

1. Generate a service ticket for the container user by following the instructions at Generating a Service Ticket.
2. Copy the generated ticket file to your local host machine. This is your source ticket file.
3. Change the owner and group on your source ticket so it matches the UID and GID in the ticket file.
4. Specify the source ticket path in the Docker mount point, as described in the table below.

The table lists the parameters related to MapR tickets and their values in the sample command:

See Security Considerations for the MapR PACC for further information.

FUSE-Based POSIX Client

With the FUSE POSIX Client for File-Based Applications, you can access the MapR File System using POSIX shell commands instead of Hadoop commands. To do so, you must specify the MapR File System mount point environment variable (MAPR_MOUNT_PATH) and other FUSE parameters in your docker run command. In the sample command shown earlier, the following are the relevant parameters and their settings:

All of these parameters are required except security-opt. You must specify security-opt if you are running on an Ubuntu host.

Pig, Livy, and Spark Interpreters

docker run -it ... -e MAPR_HS_HOST=172.24.9.248 ...

This enhances the performance of those interpreters. If your MapR cluster does not have a HistoryServer, your Pig and Spark jobs will run, but they may perform poorly.

Notebook Storage

The environment variable ZEPPELIN_NOTEBOOK_DIR specifies where to store your notebooks. If you do not specify ZEPPELIN_NOTEBOOK_DIR, Zeppelin stores your notebooks in the directory /opt/mapr/zeppelin/zeppelin-0.8.0/notebook.

You can store your notebooks either in a MapR File System or your container's local filesystem:

MapR File System using the FUSE-based POSIX client

In the sample command shown earlier, MapR Data Science Refinery stores your notebooks in a directory named /user/mapruser1/notebook in the MapR File System using the FUSE-Based POSIX Client.

The example assumes your MapR filesystem mount point is /mapr and your cluster name is my.cluster.com:

docker run -it ... \
   -e ZEPPELIN_NOTEBOOK_DIR=/mapr/my.cluster.com/user/mapruser1/notebook ...

IMPORTANT You must specify the parameters used by the FUSE-Based POSIX Client. If Docker is unable to start the MapR FUSE-Based POSIX Client, you cannot open Zeppelin in your browser. Your browser will return the following error:

HTTP ERROR: 503

MapR File System without the FUSE-based POSIX client

Starting in MapR Data Science Refinery 1.3, you can store your notebooks in the MapR File System without using the MapR FUSE-Based POSIX Client. To use this option, specify the full MapR File System path in the ZEPPLIN_NOTEBOOK_DIR variable.

The following example is equivalent to the previous:

docker run -it ... \
   -e ZEPPELIN_NOTEBOOK_DIR=maprfs:///user/mapruser1/notebook ...

Local filesystem

To store your notebooks in your local filesystem, specify the container's local path in ZEPPELIN_NOTEBOOK_DIR:

docker run -it ... \
   -e ZEPPELIN_NOTEBOOK_DIR=/opt/mapr/notebook ...

Python Version

By default, when you use Python with either the Livy or Spark interpreters, the interpreters use Python 2. Although you can run only one version of Python with MapR Data Science Refinery, you can switch to Python 3, by setting the PYSPARK_PYTHON environment variable to the absolute path of the Python 3 executable on your MapR cluster:

docker run -it ... -e PYSPARK_PYTHON=/usr/local/bin/python3.6 ...

You can also install custom Python packages. See Installing Custom Packages for PySpark

Idle Interpreter Timeout Threshold

Starting with the 1.3 release, by default, MapR Data Science Refinery terminates interpreters that have been idle for one hour. To modify this idle timeout threshold, specify the ZEPPELIN_INTERPRETER_LIFECYCLE_MANAGER_TIMEOUT_THRESHOLD environment variable. The parameter value is in milliseconds.

The following example sets the idle timeout to 10 minutes:

docker run -it ... \
   -e ZEPPELIN_INTERPRETER_LIFECYCLE_MANAGER_TIMEOUT_THRESHOLD=600000 ...

If a Spark job terminates due to the Spark interpreter reaching the timeout threshold, the job shows a status of FAILED in the YARN UI.

Configuration Storage

By default, the following Zeppelin configuration files are stored in /opt/mapr/zeppelin/zeppelin-0.8.0/conf/:

• interpreter.json
• notebook-authorization.json

Starting with MapR Data Science Refinery 1.3, you can store these files in the MapR File System by specifying the ZEPPELIN_CONFIG_FS_DIR environment variable. You can also specify a local system for this variable.

The following shows sample commands for the three available options:

MapR File System using the MapR FUSE-Based POSIX Client

This example assumes your MapR File System mount point is /mapr and your cluster name is my.cluster.com:

docker run -it ... \
   -e ZEPPELIN_CONFIG_FS_DIR=/mapr/my.cluster.com/user/mapruser1/dsrconf ...

IMPORTANT You must specify the parameters used by the FUSE-Based POSIX Client to use this option.

MapR File System without the MapR FUSE-Based POSIX Client

The following example is equivalent to the previous:

docker run -it ... \
   -e ZEPPELIN_CONFIG_FS_DIR=maprfs:///user/mapruser1/dsrconf ...

Local filesystem

The following specifies a local filesystem path:

docker run -it ... \
   -e ZEPPELIN_CONFIG_FS_DIR=/opt/mapr/dsrconf ...

If all of the following apply, then you must restart all containers to enable the new configuration settings:

• ZEPPELIN_CONFIG_FS_DIR is set to a MapR File System path
• Multiple containers share the two configuration files
• You make a change in either of the configuration files that requires a container restart

Default Drill JDBC Connection URL

The default Drill JDBC connection URL is jdbc:drill:drillbit=drillbitnode:31010. Starting with MapR Data Science Refinery 1.3, you can configure this default URL using one of the following two environment variables:

MAPR_DRILLBITS_HOST

Description

Comma separated list of Drillbit servers and optional port numbers.

Use if you want to connect to Drill through a Drillbit server. If you do not specify port numbers, they default to 31010.

Sample commands

docker run -it ... \
  -e MAPR_DRILLBITS_HOSTS=node1.cluster.com,node2.cluster.com ...

docker run -it ... \
  -e MAPR_DRILLBITS_HOSTS=node1.cluster.com:31010,node2.cluster.com:31010 ...

Resulting URL

jdbc:drill:drillbit=node1.cluster.com:31010,node2.cluster.com:31010 

MAPR_ZK_QUORUM

Description

Comma separated list of servers and optional port numbers in your Zookeeper cluster.

Use if you want to connect to Drill through a Zookeeper cluster. If you do not specify port numbers, they default to 5181.

Sample commands

docker run -it ... \
  -e MAPR_ZK_QUORUM=node1.cluster.com,node2.cluster.com,node3.cluster.com \
  ...

docker run -it ... \
  -e MAPR_ZK_QUORUM=node1.cluster.com:5181,node2.cluster.com:5181,node3.cluster.com:5181 \
  ...

Resulting URL

jdbc:drill:zk=node1.cluster.com:5181,node2.cluster.com:5181,node3.cluster.com:5181/drill/my.cluster.com-drillbits

You should specify only one of the two environment variables. If you mistakenly specify both, MAPR_DRILLBITS_HOST takes precedence.

See Start the Drill Shell (SQLLine) for more information about the Drill JDBC connection URL. See Drill JDBC for more information about setting other options in your Drill JDBC connection string for MapR Data Science Refinery.

MapR-Specific Environment Variables

The following table lists all MapR-specific environment variables you can use in your docker run command. The second column contains a short description of each variable. The third column provides links to detailed descriptions, including situations where you need to use each variable.