Spark - Spark-submit

> Database > Spark

1 - About

The spark submit application to submit application.

The spark-submit script is used to launch applications on a cluster.

Spark jobs are generally submitted from an edge node

Advertising

3 - Syntax

./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --jars <additional1.jar,additional2.jar> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

where:

4 - Example

4.1 - Run application locally on 8 cores

./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master local[8] \
  /path/to/examples.jar \
  100

More ?. See Java submit

5 - Documentation / Reference

Advertising
db/spark/spark-submit.txt · Last modified: 2018/07/09 13:29 by gerardnico