Ports Used by Spark
To run a Spark job from a client node, ephemeral ports should be opened in the cluster for the client from which you are running the Spark job.
If you do not want to open all the ephemeral ports, you can use the configuration parameter to specify the range of ports.
To set ports to special values, use the spark.driver.port
, spark.blockManager.port
, and spark.port.maxRetries
properties.
The spark.port.maxRetries
property is 16 by default.
For example, if you need to open port 200 for spark.blockManager.port
from 40000, set spark.blockManager.port = 40000
and spark.port.maxRetries = 200
.
For a list of Web UIs ports dynamically used when starting spark contexts, see the open source documentation.
Service | Port Number |
---|---|
Spark Standalone Master (RPC) | 7077 |
Spark Standalone Master (Web UI) | 8580, 8980* |
Spark Standalone Worker | 8581, 8981* |
Spark Thrift Server | 2304 |
Spark History Server | 18080,18480* |
Spark External Shuffle Service (if yarn shuffle service is enabled) | 7337 |
CLDB | 7222 |
ZooKeeper | 5181 |
Nodes running ResourceManager | 8032 |
MapR Filesystem Server | 5660, 5692 |