Spark - Worker Node

Spark Cluster

About

A worker node in a cluster (machine)

Worker can also mean executor

Spark Cluster

Management

Get Hostname

import socket
def getHostname(x):
    return socket.gethostname()
num_worker=3
# A rdd with three partitions
rdd = sc.parallelize(range(1,num_worker+1),num_worker)
rdd.map(getHostname).distinct().sortBy(lambda x: x).collect()
['wn0-hddev2', 'wn1-hddev2', 'wn4-hddev2']





Discover More
Spark Cluster
Spark - Cluster

A cluster in Spark has the following component: A spark application composed of a driver program which include the SparkContext (for RDD) or the Spark Session for a data frame which connect to a cluster...



Share this page:
Follow us:
Task Runner